=> Please click "Call Settings - Fix Recording Issues - Ask New Parameters" in Boldbeast Recorder - That's it.
Also, could you update your forum thread on settings for Oreo 8.1.x for Nexus 6P? Apart from the Alsa Device, are my settings correct?
=> By default Alsa Device is set to "Auto", Boldbeast Recorder will automatically detect which alsa device to use and it works well. However in some third-party ROMs the "Auto" option may not work, in this case please set Alsa Device to "Device0" directly then it works at once.
I just tried clean flashing an updated Oreo ROM for July (The ABC Rom) as well as tried the Google Phone dialer instead of the AOSP, and the same thing happens. When pressing the record button, it won't let me hang up, even if the call has ended. It just hangs, and the only fix is to hard reboot.
=> Please disable the "Call Settings - Fix Recording Issues - Start Input Stream" option and test again.