So I narrowed down the issue a bit further. It has to do with whether or not I'm on LTE. I still think Google voice might be interfering. But something is different with how the call is delivered when i'm on LTE. When I set my phone to CDMA only, the call rings, i answer it and everything is fine. When i turn back on LTE. The call rings, i answer it and it's dead air. No voice in or out. Really weird.
Other things I've tried. Wifi Off/On, no difference. Signed out of my google/gvoice accounts. I also disabled the feature to deliver calls to Google talk/hangouts. Maybe it takes a while for the settings to take effect.