Hey! I had a N5 prior to the Z1.
I love the N5, in some ways it is the perfect XDA phone. I swapped simply because I wanted to try something with better build quility. As you're aware, the camera on the N5 isn't great, one of the cut backs to get it so cheap. The Z1 camera is fantastic (or else it is compared to the N5.) Also, the finish of the phone is classy compared to the N5 (another cut back.)
As for speed: out of the box the N5 is faster. They are both running the same processor, though the N5 is clocked slightly faster. Obviously the N5 is running vanilla android, though in all fairness the sony skin isn't particularly intrusive. Not that the Z1 is slow. I'm running Kitslim rom with doomkernel overclocked at 2.4, with thermal throttle off. It's very quick, just not as quick as a N5 say with trinity and rasta. Although to counter this, even overclocked I get better battery from the Z1.
They are both very good phones. one thing I really miss on the N5 is the huge amount of roms to try. I'm a flash addict, and there's some good roms for the Z1, but so so many more for the N5. Worth thinking about if you flash a lot.
I'm not sure if this helps. I love the Z1. It does everything I need it to really well. It's a beautiful phone, and quick. Every so often though I miss the freedom to flash that you get from the N5 community.