Default [Q] Restoring a backup of an older version of Android?

I'm currently running a ROM that is based on Android 4.3. I also have a backup of a ROM I kind of like more, but it is from an older build of Android (not 100% sure the version number). I vaguely recall reading somewhere that it could brick your phone if you try to go to an older version without following special steps. Am I right in taking caution by not just restoring the backup, or is it perfectly safe? Thanks!