I agree, if you love the Note 3 you're likely not going to like the S5. It's quite a bit smaller screen but the phone is almost the same size. No Stylus (if you use). You just aren't comparing apples to apples and i also believe it's a step down from the Note 3.
On a side note, I love having the latest and greatest but this is the first phone i had that i never am finding myself looking for an upgrade. The Note 3 is amazing and there isn't really anything out there better to the point you would even notice a difference. i mean the phone is barely 6-7 months old and the spec war is at a stand still. All the phones are about the same now spec wise. So i say just relax and enjoy your phone for awhile.