And, I really don't know many people who sit down with two tablets and work on both at the same time. These sorts of differences (some of which are intentional, such as the pauses before gestures register, switching orientation, etc., and I tend not to prefer Apple's frantic settings) are only obvious when you have the two next to each other.
That's not to say that Apple hasn't optimized the hell out of iOS on the iPad 2 to make sure everything's smooth--that's their shtick. Keep it simple and smooth (their very own KISS principle). But, by itself, I've found Honeycomb (on the Xoom) to be plenty fast and smooth for actual, live, everyday use. Sure, I could sit there and nitpick about a tiny bit of lagging every now and then, but it doesn't actually get in the way of getting things done.
It'll all be good over time. We're in that initial place that Android was on phones, and I think we all know how that's turned out so far.