I switched last week. Best technology-related decision I've made this decade. The Note was fine, but since I couldn't stand TouchWiz, I was running CM9 (which wasn't quite full-featured, as the pen wasn't fully supported). The GNex feels so much better in the hand -- and pocket.
I hardly notice a difference with screen quality or size by looking at it, though the slight shrink in keyboard size took some getting used to.
Oh, plus I was able to stuff some extra cash in my pocket in the trade, even though the Note and Nexus were the same price when I bought the Note.
HTC 8X (625a)