Update: I am back on this 3.2 WiFi ROM and don't have any issues. I did two things different this time. First, I wiped data (factory reset). Second, I installed my new updated kernel v2b. While the latter may give some smoothness improvements, it is the former that is likely making the difference. When I did this the first time, I was lazy and didn't wipe data. The ROM install itself wipes cache and dalvik cache, so this was the same for everybody, but the data wipe may have been a wildcard and may have been worse for some due to the settings they had etc.
If you do decide to do this (wipe data and upgrade), you won't lose all your SDCard install zip files, but it will erase system settings (WiFi networks/passwords), email accounts, browser bookmarks, installed apps and their settings. You will boot into a new tab this way and start from scatch. You can use Titanium Backup so you can backup on 3.1 and restore apps quickly. Also, make full CWM backup and you can just go back to 3.1 as it was if you like.
I'm definitely happy and staying on 3.2 now. It sounds like the Overcome ROM is also flying along on 3.2 as well after his last update. Moving onward and upward!
Nexus 4 (my custom kernel for the 4 click here
Nexus 7 (my custom kernel for the 7 click here
) -Thx to fordwolden, the most generous guy on XDA!
Transformer Prime (my custom kernel for the Prime click here
Always have your stuff when you need it with Dropbox. 2GB account is free! Use this LINK and you and I both get 500MB extra free