I've noticed the difference between my old Desire and my wife's Xperia Ray which are both gingerbread. When pulling the notification bar down theExactly. It is the time difference between when you move your finger and the screen is done rendering. A lot of code is executed during that time.
I think that a good part of it is related to smoothing algorithms and resource management in Android 4+. If you keep changing timespans in which a certain peace of grahics moves from one place to another in seperate thread, you will get smooth but slow scrooling because the panel will keep scrolling while the next target and time is calculated. It would have to be very poorly coded to actually get freezes even with performance issues.
graphics are far less smooth but the input lag is a lot lot less so I wouldn't be surprised if that is at least some of the reason.
Would be interesting to compare my Desire running ICS or JB to see if it's different. Can I be bothered though? probably not.