Hmm... Does it "flicker" before rebooting?
My best guess is that TWRP simply crashes and forces a reboot. From the top of my head the reason for that kind of behaviour can only be the encryption or trustkernel. I started my work with a non-functional encryption. After that I already had a working system so I probably never really tested the stock rom scenario with encryption activated.
To finally get encryption to work on recovery and not messing up the system I needed to do a copy of the encryption keys to a temp location. Trustkernel under TWRP uses this temp location for booting up. Maybe this isn't the right thing to do while there are no keys present to copy OR the data partition is already encrypted, trustkernel detects that the keys are invalid and therefore refuses to work as a security measure.
If it is the later you can try to flash "userdata" either the same way as you did with recovery or using SPFlash (only select the "userdata" partition). As Image you could use the one provided in the stock rom.
If it is the former you should be able to boot with a TWRP release where the encryption isn't setup the way it is now (for the L TEE that means v20210415). After you have installed LOS the encryption should be properly setup and you should be able to flash the latest TWRP release from within TWRP.
Please try the "userdata" flash first and if that's not working try the older release. If it's just an encryption problem I can solve this quite easily by extending the installation section. If it's the other thing I need to do a deeper investigation to find the cause (meaning I once more need to start testing installation procedure from a genuine stock rom )