So as you are suggesting, there are two possible encryption passcode promts. 1. in twrp recovery 2. at system boot
I still get the encryption passcode when booting into twrp as well. I have magisk flashed already. I wonder why it is still there.... Did I need to reboot into twrp after fomatting the data partition/before flashing the FullStock zip? Does uninstalling/re-flashing magisk fix it?
I was getting the encryption passcode prompt for the system boot at one point, but as I said, formatting the data partition before flashing the FullStock zip fixed that.
Ill explain the issue and steps for flashing and hopefully that will answer your question.
The 11g, 11h and 20a stock ROMs will try to encrypt the data partition on your phone the first time you boot to the OS. Sometimes it will show "Encrypting.." on the screen with a little android guy. Other times it might just say "Starting Android.." or something like that with a progress bar. Either way, it will always try to encrypt your data.
Now. Assuming you have been using your phone and now want to reflash a rom (Stock or otherwise) and flash magisk in the process:
You reboot to TWRP. If TWRP asks you for a passcode, that means your data is encrypted, and TWRP will not be able to mount the system, vendor or data partitions.
In order for anything to flash correctly, you will need to go to Wipe -> Format Data. This will format the entire data partition and remove encryption. But TWRP still has not mounted system, vendor or data at this point. So you need to either
A.) go back to the main TWRP menu and tap Reboot -> Reboot Recovery
or
B.) Manually mount System and Data.
Option A is generally safer because sometimes when I have manually mounted system it still wouldnt flash because the partition was somehow in use.
After rebooting back to recovery, all partitions will be mounted correctly, your data will not be encrypted, and you will not get a password prompt in TWRP. You will just get the standard excerpt about it being dangerous and asking if you want to keep the device read only blah blah.
Now at this point, you can flash whatever rom you want. After flashing any roms or bootloaders or anything that is going to replace a part of the OS, then you flash Magisk.
Watching the magisk flash logs on the screen, you could see one or more of the following:
"encrypted data detected. preserving force encryption"
something regarding it patching dm-verity
then at the end it will try and repack the patched boot image. At this point, whether your device is going to be force-encrypted or not, magisk has patched boot and any other areas needed so that you TWRP will be able to work without entering a passcode when you boot to recovery.
Now, when you initially boot to the OS,
key point is that you do not enable secure startup when setting up your phone. I think this is the one thing I missed when I have been explaining this to everybody and it's the one thing I never do when I set up my phone. I dont know why it never came to mind until now. That is the feature that asks you to enter a passcode or pattern on boot and ultimately encrypts the entire device.
You can add a pattern or passcode to the phone for your screen lock, but enabling secure startup may just hose all of the work you did with formatting and flashing magisk.
I hope this helps. I am going to be uploading pre-rooted SysBoot roms for 11g, 11h and 20a today with patched kernels that should have dm-verity and force-encryption disabled. I am not sure if the patched kernels are actually going to work, but the roms will at least have root integrated. Ill post an update when they are online.