I think this is a bug in download manager. I don't know of a fix. But I've noticed on my Nexus 4 that it only keeps displaying notifications again if you don't dismiss the download notification before shutting down or rebooting if that helps at all. When it does happen, just like you said you can go and clear the cache of download manager.

XDA Developers was founded by developers, for developers. It is now a valuable resource for people who want to make the most of their mobile devices, from customizing the look and feel to adding new functionality.Are you a developer?