As an early nexus devices adopter, I do agree ! I didn't try to return my nexus 5 and its bad speaker, as it wasn't a deal breaker for me. But for the N9, considering that google sold it to us as a premium device (489€, ouch !) I expect a premium build quality.
I'll send them an email this...