I'm thinking it might have something to do with the case I'm using (official Google nexus case) which seems awfully tight and might pull on the case when I'm pressing the buttons.
This doesn't bode well for the quality of the aluminum used.
Update 11/6: I contacted Google customer support. They were very nice and I got the RMA for the phone and the case. The woman seemed to think it was a manufacturing defect for this phone and/or this case only.
I am going to hang back a bit and see how the whole thing shakes out. If this doesn't turn out to be a chronic issue, I will definitely re-purchase a Nexus 6p.