I'm in a similar situation. I picked up the S4 Thursday, played with it for a day, and hated it. The camera was laggy (0.5-1 second shutter lag), the UI and apps themselves were laggy, it felt like I downgraded because of the smaller screen and the battery life was a lot worse. I went back to my Note 2 for the weekend and thought about returning the S4.
Then last night I decided to give the S4 another shot. I did a factory reset, installed my apps again (except I only installed what I REALLY used, which is about 60 apps), rooted, and froze/uninstalled a TON of apps and processes using Titanium Backup. After customizing it even more it's been much better in terms of performance and battery life. The screen is amazing on the S4, it's much brighter, whiter, no jagged edges on text (MBP > MBP Retina), easily viewable in sunlight, etc.
I thought back to when I first got my Note 2 and realized that the S4 is still brand new, when the Note 2 was new it was also laggy and full of bloatware. The S4 is only going to get better as OTA updates come out to fix these few minor issues, and once devs start to build solid ROM's it's going to be just as reliable as my Note 2 is on Beans 17.
I enjoy having a phone that I can actually fully use in one hand, fit in my pocket without being uncomfortable while sitting, and more importantly actually be able to fit it in cupholders!
