There are a few things that you need to pay attention to.
1. Neon-VFPv3 is for Cortex-a8 and not for a9. You may want to flag it as neon-fp16..
2. I believed that your toolchain is targetting linux kernel version 3.4.x or something but not for version 3.1.10.
3. I have the same issues with my owm builds gcc-4.8/4.9 without any solution.
4. Try some of -Ofast flag to see the improvement on v3.1.10
I took the plunge and tried a new toolchain. Ended up trying a 4.9 linaro one for the Grimlock Kernel. Works like a champ on my TF300t. HOWEVER for some reason it will not even boot on a TF700. I'm told it vibrates and the screen goes all white or something. So here is the question:
Why would new toolchains work fine on a TF300 but not on a TF700? One of the transformers' great mysteries