Default [Q] Sound on speakers with headphones plugged in

I'm using a pair of apple wired headphones no volume buttons just the ear buds.

When I plug it in and play pandora or songza it seems the music will play through speakers even though headphone is connected. I have to unplug and replug the headphones while the song is playing for the music to switch.

It's not totally consistent but happens frequently. It may be worse when im switching between multiple apps, pandora -> songza. Or if there is a delay between starting songs.

This happens on stock sony and CM11.