This started happening to me last weekend. I figured it may have something to do with being in Japan, on 50Hz power or something mundane, but having returned to 60Hz land the problem remains.
I had used my wife's Samsung Nexus S charger for one night instead of my GN unit, and that may have tweaked the center tongue of the USB port downwards. I've used a pushpin to pry up slightly on the connector with hopes that it's so simple.
Note, this is on a completely unmodified GSM Galaxy Nexus running 4.0.4 IMM76D with the 2000MAh Samsung battery.
Google Galaxy Nexus GSM Black 2000MAh
HTC One V / "Primo" GSM - Ninja Tel DEF CON XX Exclusive
Google Nexus S / GT-I9020T
"Day One" Google Nexus One / "Passion"
MotoFone F3 / "Zombie Apocalypse Phone"
Apple "new" iPad 64gb WiFi
HTC Wing / "Herald"
(formerly owned) - Apple iPad 2 64GB WiFi Viewsonic G Tablet / "Harmony", LG-P999 / G2X, Apple iPad 16GB 3G, TMOUS1024 HD2/Leo 1024, 4xHTC G1/Dream, HTC MDA/Wizard, HTC SDA/Tornado, Nokia N900, Nokia 6600, Danger Hiptop, Hiptop Color.