Not necessarily, the aokp and miui beta 3 roms by stelios97, which are based on ics, but run on the nAa-11 kernel show a fair amount of promise in speed and stability. I'm very much looking forward to improvements in the miui-rom especially.
As far as the ICS kernel and MiniCM9 by nAa are concerned, they're still very early works with lots of bugs to be ironed out. I am sure in future iterations you will see lots of improvement in the overall user experience.
All that said though, my phone worked perfectly fine on eclair too, so i guess it's all about what you want from your phone, nothing is perfect, and different people have different needs or expectations.
I just love to see that our little devices are capable of so much more than people think or expect, so for what it's worth... there will come a time when it will be impossible for us to run the latest android, but i don't think that time is here yet.