Mine is like that only when I'm talking to my son on his S4. He said the same thing that when he's talking to me the voice quality is super clear. He's on Bell, I'm on Telus.
Phone: Nexus 5
ROM: Stock 4.4.2 (Rooted)
Kernel: franco r32
Headpones: Beyerdynamic Tesla DT-1350, Sennheiser CX 880i