After more than 1x times of trial and error, I've finally found out how to decrypt data on stock OOS with magisk.
Decrypt Data on stock OOS with magisk:
1. Go to “Wipe” section, tap “Format Data”, and type “yes”
2. Transfer no_verity_op5.zip to the device over MTP
3. Go to “Install” section, select no_verity_op5.zip, and swipe to confirm the flash
4. Go to “Install” section, select magisk.zip, and swipe to confirm the flash
PS. you should format data before flashing no_verity_op5.zip.
[Proof]
[How it work?]
OnePlus 5 uses FBE(File-Based Encryption) to encrypt data partition.
The latest no-verity-opt-encrypt zip (
https://build.nethunter.com/android-tools/no-verity-opt-encrypt/ ) only replace "forceencrypt" to "encryptable".
It didn't replace "fileencryption", so your data partition is still encrypted.
In order to decrypt data partition, I modified the no-verity-opt-encrypt.zip by myself.
The no_verity_op5.zip will replace "fileencryption" to "encryptable".
boot.img/ramdis/fstab.qcom