Default Calling out or receiving calls intermittently in speaker mode.

I have this weird problem while taking a call or calling out that my phone tends to switch to speaker mode itself. Anyone having this issue as well? And how to tackle it to make it stick onto earpiece speaker only..

Sent from my Nexus 5 via XDA Premium HD
=======================================
Nexus 5: 312K: Running stock deodexed 4.4.4 with ElementalX kernel

=======================================
Transformer TF101(B60): Kitkat-Katkiss 4.4.4
System Speed: Overclock Dual Core 1400Ghz


=======================================