Thanks for your suggestion to use exFAT instead of fat32.My guess is you used to backup 'system' which is smaller in size than 'system.img' and your external card is formatted in fat32 which doesn't allow to store larger files than 4gb. If I'm right, formatting it to exFAT would fix your issue.
I will backup first the data and then format to exFAT -> restore the data -> (re)install latest twrp 3.5 -> try again if the system image backup will be succeed.
UPDATE: exFAT solved my issue. Thanks @AndrzejDwo
UPDATE2: exFAT is much slower than FAT32 or do I have to set some parameters?