s4 is totally smooth for me. maybe you have some apps running that are constantly eating up system resources?
also bare in mind that your nexus does have a faster cpu and gpu i believe?
but honestly i don't know how people have problems with the s4, it's already like butter out of the box!
if you really need to, you can root and look into overclocking or use a custom kernel to further improve performance....but tbh, with the spec of the s4, it's really not necessary for performance in my opinion.