The biggest advantage of exFAT over FAT32 is that you don't have to worry about a 4GB file size limit. Max single file size on FAT32 is about 4GB; exFAT is about 16ZB (zetabytes). FAT32 only supports disk sizes of up to about 2TB, whereas exFAT goes up to nearly 64ZB. The location of the boot sector is different between file systems as well as some speed differences. Most of the other changes don't matter enough to list.
Essentially, it boils down to the fact that you don't have to split the backups into separate files if you are using exFAT. That's the number one advantage in reference to custom recovery.
On the flip side, if you don't care about split backups, or having larger single files than 4GB, then just stick with FAT32 for nearly universal compatibility.