I don't know exactly what fixed it, but its working beautifully now. I have been using it for the last 2 weeks. I faced stuttering only twice for less than 2 seconds each time, that too during the initial connection attempt.
The things that I suspect that could have fixed it include
1. I updated to the latest PTV3000 firmware.
2. I had to sent my Nexus 7 for repair due to speaker issues. They might have replaced the motherboard which in turn could have fixed something.
3. Weather changed from 25 degree celsius to less than 5 degrees.
4. My up stair neighbour moved out.
As long as it works the way it is now, I will happily dismiss this as one of another life's mysteries.
It works properly even with a case.