I've had issues with the microusb port on multiple phones in the past, and have definitely noticed the N4's socket getting looser.
On an old HTC sensation, I had to resolder the port on. Can you reproduce the behavior by applying slight pressure to the plug in various directions?
Does your connection also intermittently drop out when connected to a computer?
A workaround for the time being could be to use a kernel that allows you to force fast charging regardless of the detected charger. But at that rate, it's quite possible you'll lose your usb port altogether sometime in the future..
You could also enable adb over wireless, and check the charger type:
You should be seeing DCP (has the data pins connected, possibly with some resistance).
Or proprietary if you're using an Apple-compatible charger (has aspecific voltage on the data pins).
If you see SDP, it means it's in "USB" mode.
If it's staying DCP/proprietary but the charge rate is dropping to "USB," it means the phone is throttling down the current because it's detecting the charger can't deliver (voltage dropping below threshold).