I did some searching and see others had a similar issue when they upgraded to 4.2.2, but that was a long time ago. Surely these issues would be fixed by now?
A few things to note:
I'm also running CM11 on my N4 and bluetooth works fine.
All of my music is "pinned" so it's cached on the device - no streaming.
Wifi is turned off