Hmm, my phone is in this situation now. Trying to go from CM11 back to stock when I made the backup.
Never used a backup before so hoping it works, but it is just sitting at Restoring Modem for a while now..
Not sure if it restart and try again or wait.. but for how long.
EDIT: Restarted and tried again and it went through the restoring modem bit like instantly.