The speaker is okay, nothing to write home about but not terrible. I'd say it's better than the Nexus 4 but certainly not as good as the HTC One M7/M8. The headphone jack is awful. I'm not quite an "audiophile" but the sound is not good. It's still listenable, but it's a significant step down from the HTC One M7 or Galaxy S3 I've used in the past. It's actually to the point where I would rather listen to music over bluetooth than use my headphones because the quality drop over bluetooth isn't as significant as the difference in DAC quality between the N5 and my bluetooth receiver.
All in all, audio quality is probably the one thing that makes me regret the phone. If you plan on using headphones with the phone a lot, and don't want to deal with a USB DAC (including the issues getting it working on the Nexus 5) I would avoid the phone. I wish I had waited and picked up an HTC One M8 honestly, and that would be my recommendation. Now that they've dropped the "Beats" branding the audio quality is actually very good without the overpowering bass of the original M7.