I've had the same problem after installing LineageOS.
My phone would for some reason mirror the expected behavior after plugging in the adapter at times.
It would output audio through the speakers after I plugged in my adapter. And after unplugging the adapter it would somehow try to output audio through a now non existing audio connection. It would be stuk not using the speakers until I reconnected the adapter or rebooted the phone. Rebooting the phone with the adapter connected would fix the problem when it happened.
I tried to get a log when it happened so that I could file a bug report.
But before I could it stopped happening.
No idea what fixed it for me since I didn't do anything to try and fix it.
Note: I use a passive 3.5mm adapter (without a DAC).
Instead of one with a DAC like the CX31993.
Device: KB2003 (EU)
Firmware version: KB2003 C.33
Experienced the problem in the following versions:
20220825 and 20220901
It disappeared while using 20220901.
Edit: added device info.