I got tired of this issue too. Never had the icon hang there forever, but wouldn't disappear immediately sometimes even after listening to all my voicemail. It would stick around for about 5-10 minutes after. In the end I just setup Google Voice using the voicemail only option called Google Voice Lite. Switched it under voicemail options on my phone and followed the provisioning instructions to ring Google Voice's voicemail box instead. Now Google Voice acts like Visual Voicemail on my Nexus 5. You can try that out, but I think its a US only feature.
I had the very same problem it was annoying but i just ignored it untill the other day, i noticed on android authority there was a guide to instal the same cyanogen 4.4.4 software the oneplus one has so i flashed it and hey presto voicemail icon gone i suppose that goes for reinstalling stock roms too, or at the very worst i think they get deleted after 30 days.......
It could be an issue on carrier end....they are still pushing the notification to you maybe.
---------- Post added at 11:46 AM ---------- Previous post was at 11:45 AM ----------
If you can get someone else's sim card and try it....if notification goes away for that sim, then back for yours....its the carrier.
Just an idea.