I think this is a bug in download manager. I don't know of a fix. But I've noticed on my Nexus 4 that it only keeps displaying notifications again if you don't dismiss the download notification before shutting down or rebooting if that helps at all. When it does happen, just like you said you can go and clear the cache of download manager.
Sean Behan

My Devices:

Nexus 4 -- ROM: Paranoid Android Kernel: Hells Doctor Kernel

Nexus 7 (WiFi 2012) -- ROM: Paranoid Android Kernel: Franco Kernel

Xperia Mini Pro BRICKED