Default Sony Xperia ZL C6502

I get this speaker bug sometimes where the sound comes from the front rather than the speaker from the back. Anyone else have this issue? When I have a headset plugged in and I unplug it the music starts coming from speaker rather than the back speaker. Then the back speaker doesn't work until I manually reboot the phone.