I can hear the other party just fine, but they tell me that from my end the sound is very muffled. I finally figured out that I can solve this by turning the volume down 2 or 3 notches, but then I can only make the call in a quiet room or else I can't hear them properly!
I reckon what must be happening is that the phone is trying to compensate for the speaker and mic being so close together (stopping feedback) that at loud volume it just aggressively cancels too much out which then includes my own voice.
So now I think I also know why normal speakerphone/handsfree phone calls are so soft in volume - Sony have deliberately knobbled it as a workaround rather than fixing the core issue. But other apps are allowed to use a higher speaker volume and so run into this issue if using the microphone as well.