Definitely not if you're already on the 4.4.x bootloader. I'm switching back and forth between CM11 and TouchWiz stock all the time. All it takes is a quick restore in TWRP.
The worst case scenario is that you might encounter some Force Closes, especially from the Samsung stock apps. In which case, just re-flash the ROM and all the problems will go away .
CyanogenMod 11 is a good start: M8 is quite stable and capable of being a daily driver. The battery life is insanely good on CM11 stock kernel.
Slim AOSP is another stable one with active development.
Keep in mind that you're giving up quite a bit using AOSP ROMs: poor S-Pen support, terrible camera app (no slow-mo or 4K recording), no Wi-Fi & VoLTE calling, no aptX codec (higher sound quality for Bluetooth speakers) but in return, you'll see that your phone feels much, much faster and smoother (like the true beast that the hardware inside is). Also much better battery life.
So yeah, pick your poison