I've been able to go from stock 4.1.1 to Kit Kat based CM11 and back to stock 4.1.1 without any hitches at all. This would be because CM (like most custom ROMs) doesn't mess with your bootloader. If you had the 4.3 bootloader, however, you can only go back to stock 4.3. You should be able to flash the 4.3 stock restore from custom recovery without any hitches. This option has always worked for me.