Are there 2 copies of /system on the phone?

[BillGoss]

My phone (3T) has an unlocked bootloader, is encrypted, not rooted, and running stock OOS 5.0.
I flashed TWRP and discovered that stock OOS restores the stock recovery in boot.
I saw the Oreo dm-verity thread by xenet, had a look at the zip file, noticed that it just modified fstab to prevent force encrypt, so I flashed it to see what happens.
And nothing happens. After the system had booted, fstab is unchanged from the original stock copy.
So I'm wondering whether this file is also restored when booting up on stock.
I get aggressive and go back to TWRP and delete /system/etc and /system/bin and modify build.prop.
Surely now the phone won't boot!
Wrong! It boots up and everything is back to normal in /system.
I go back to TWRP and have a look at /system and it shows me one without the etc and bin folders and has the modified build.prop.

What's going on? How can I see one version of /system in TWRP but a different version (ie, stock) when the phone has booted?

By the way I've been an Android user for many years and have rooted and flashed custom ROMs on a variety of phones and I've never seen anything like what's happening on my 3T. I'm sure that dm-verity is somehow involved in this.

 

[nicknacknuke]

Happened to me on my earlier OOS 5.0 attempts...
But i suspected Magisk is involved in my case.

I downloaded Magisk Module "System Terminal Debloater,"
remove some apps like Duo, Chrome, and Google Play Movies.

Some restarts, they magically re-appear again on Apps Drawer...

Haven't touch them yet again after....

 

[BillGoss]

Happened to me on my earlier OOS 5.0 attempts...
But i suspected Magisk is involved in my case.

I downloaded Magisk Module "System Terminal Debloater,"
remove some apps like Duo, Chrome, and Google Play Movies.

Some restarts, they magically re-appear again on Apps Drawer...

Haven't touch them yet again after....

Thanks.
I should have mentioned that I'm also not rooted. So stock OOS 5.0.

Sent from my OnePlus 3T using XDA Labs

 

[Jackhass]

When you boot TWRP for the first time, it should ask you if you want to put the /system in read/write mode or if you want to leave it unchanged, did you choose the right option?

 

[BillGoss]

When you boot TWRP for the first time, it should ask you if you want to put the /system in read/write mode or if you want to leave it unchanged, did you choose the right option?

No, I don't get that message because my phone is encrypted with a password. So the first thing I see in TWRP is the request for the password and then I'm presented with the menus.
However, in the Mounted menu, system isn't mounted and I have the option of mounting it in read-only mode.

Sent from my OnePlus 3T using XDA Labs

 

[Sam Nakamura]

No, I don't get that message because my phone is encrypted with a password. So the first thing I see in TWRP is the request for the password and then I'm presented with the menus.
However, in the Mounted menu, system isn't mounted and I have the option of mounting it in read-only mode.

After first time flashing TWRP a folder gets created on your internal storage, with a hidden file called .twrps, go delete it and reboot recovery to trigger the message "allowing system modifications" on TWRP's first boot...
It's not about encryption, it's just that TWRP remember the decision you made due to the file I pointed out...

Sent from my OnePlus 3T using XDA Labs

 

[Sam Nakamura]

After first time flashing TWRP a folder gets created on your internal storage, with a hidden file called .twrps, go delete it and reboot recovery to trigger the message "allowing system modifications" on TWRP's first boot...
It's not about encryption, it's just that TWRP remember the decision you made due to the file I pointed out...

Somehow the attachment strikes on previous post

Edit: still not working, check your TWRP Folder on storage to find the file


Sent from my OnePlus 3T using XDA Labs

 

[BillGoss]

Somehow the attachment strikes on previous post

Edit: still not working, check your TWRP Folder on storage to find the file

Thanks, you are correct. I'd forgotten that that TWRP remembers. Deleting .twrps does bring up the RO prompt after decrypting storage.

When you boot TWRP for the first time, it should ask you if you want to put the /system in read/write mode or if you want to leave it unchanged, did you choose the right option?

I had allowed changes to the system otherwise I couldn't have made changes to it, which includes the ability to restore the system partition.

But I'm still unclear why if I make changes to the system partition and boot with the stock kernel, then after the boot none of the changes are present in the system partition, but if I boot back into TWRP then the changes are all there.

 

[nhshah7]

I recall someone in another OOS 5 thread saying that the stock kernal replaces TWRP with stock recovery if you don't flash root (magisk/superSU). Is it possible that the kernel re-flashes system on boot? Another possibility is that TWRP thinks it's making changes to system but it's not actually? Not quite sure, I've never heard of anything like this before either, just throwing other ideas out there.

I've never read anything about the OP3T or any oneplus phones for that matter having A/B system partitions like the pixels. *shrug*

 

[BillGoss]

@nhshah7, something's like what you suggest must be going on to account for what I'm seeing. I'm hoping that someone can confirm my observations and provide a definite answer.

 

[Xennet]

@BillGoss

My thread has been updated relating to all your queries...
Thank you...

https://forum.xda-developers.com/oneplus-3t/how-to/disable-dm-verity-force-encryption-op3t-t3688748

 

[BillGoss]

@BillGoss

My thread has been updated relating to all your queries...
Thank you...

https://forum.xda-developers.com/oneplus-3t/how-to/disable-dm-verity-force-encryption-op3t-t3688748

Actually it doesn't explain how TWRP can make changes to system yet the phone boots up on an unmodified system if using the stock kernel. And then, when you boot back into TWRP and look at system, the changes are still there.
Where does the unmodified system come from?
Where does the modified system live?
Why doesn't modifying system result in a failed boot due to dm-verity, while restoring a backup of system does result in a failed boot?
So many questions with no answers.

 

[rk2612]

....So many questions with no answers.

Not sure if this is applicable in your case but the following possibilities may be worth considering for you:
1. Are you sure that the system image is actually getting modified? If the system partition is not mounted before flashing the zip and the zip being flashed does not mount the system partition in read / write, then no changes to system partitions will actually be written.
2. If dm-verity is enabled, then restoring system could result in an error as this is different from restoring a system-image (nandroid copy of the whole partition and not just the files in the system partition). DM-verity can be triggered if the files are all the same but the dm-verity signature computed by hashing the system partition has changed.
3. For boot partitions, strange behaviour can occur if remnants of the previous boot.img are still in the partition (...e.g. if the previous boot.img was of larger size and a new boot.img of a smaller is flashed, then there will be some bytes after the new boot.img that are from the previous boot.img). To verify this, format the boot partition from fastboot and see if you notice anything different with the new boot.img.
4. In Oreo / 8.0, dm-verity flags are stored in dtb (device tree blobs) inside the kernel and not in the fstab file. Only data encryption can be changed from the fstab file and dm-verity needs to be changed from changing the dtb (...Magisk beta v1456 and SuperSu 2.82 SR4 do this, I think).

 

[BillGoss]

Not sure if this is applicable in your case but the following possibilities may be worth considering for you:
1. Are you sure that the system image is actually getting modified? If the system partition is not mounted before flashing the zip and the zip being flashed does not mount the system partition in read / write, then no changes to system partitions will actually be written.
2. If dm-verity is enabled, then restoring system could result in an error as this is different from restoring a system-image (nandroid copy of the whole partition and not just the files in the system partition). DM-verity can be triggered if the files are all the same but the dm-verity signature computed by hashing the system partition has changed.
3. For boot partitions, strange behaviour can occur if remnants of the previous boot.img are still in the partition (...e.g. if the previous boot.img was of larger size and a new boot.img of a smaller is flashed, then there will be some bytes after the new boot.img that are from the previous boot.img). To verify this, format the boot partition from fastboot and see if you notice anything different with the new boot.img.
4. In Oreo / 8.0, dm-verity flags are stored in dtb (device tree blobs) inside the kernel and not in the fstab file. Only data encryption can be changed from the fstab file and dm-verity needs to be changed from changing the dtb (...Magisk beta v1456 and SuperSu 2.82 SR4 do this, I think).

I'll come back to 1.

2. That makes sense and accounts for why a restore of the system partition with the stock boot image causes me to get dumped back in fastboot mode. If I flash the stock system zip file then the system boots properly.
3. I've not had any issues with strange boot behaviour. I'm always starting with stock or flashing kernels that modify the stock boot image, like Blu Spark.
4. I gathered this from my reading of various threads. If I want to make changes to the system partition and get them to stick and not fail dm-verity then I have to flash a custom kernel. I've proven this in my testing. (A rooting solution would also work, but I've not done this).

Back to 1:
Here's what I've done:
Starting with pure stock image (flash OOS 5.0).
Boot into fastboot and flash TWRP.
Boot into recovery.
Mount system as rw. (In ro mode the next step fails)
Delete the bin, etc, and lib folders in system using the TWRP file manager. (Screenshot a)
Reboot system.
... First interesting fact ...
System boots ok, deleted folders are present in file manager. (Screenshot b)
Boot into fastboot and flash TWRP. (Booting with stock restores stock recovery)
Mount system.
... Second interesting fact ...
TWRP file manager shows that deleted folders are missing. (Screenshot c)
Flash custom kernel or patched boot image
Reboot system
... Third interesting fact ...
System fails to boot. Hangs on splash screen.

So TWRP made the changes (otherwise how could they be visible between reboots, including a replacement of recovery) and I only did them once.
Yet they don't actually take effect until I replace the stock boot image.
So, where are the changes hiding? What did TWRP actually change?

Screenshots (note that TWRP has the wrong timezone set so the time shown is wrong):

 

[rk2612]

....
Back to 1:
Here's what I've done:
Starting with pure stock image (flash OOS 5.0).
Boot into fastboot and flash TWRP.
Boot into recovery.
Mount system as rw. (In ro mode the next step fails)
Delete the bin, etc, and lib folders in system using the TWRP file manager. (Screenshot a)
Reboot system.
... First interesting fact ...
System boots ok, deleted folders are present in file manager. (Screenshot b)
Boot into fastboot and flash TWRP. (Booting with stock restores stock recovery)
Mount system.
... Second interesting fact ...
TWRP file manager shows that deleted folders are missing. (Screenshot c)
Flash custom kernel or patched boot image
Reboot system
... Third interesting fact ...
System fails to boot. Hangs on splash screen.

So TWRP made the changes (otherwise how could they be visible between reboots, including a replacement of recovery) and I only did them once.
Yet they don't actually take effect until I replace the stock boot image.
So, where are the changes hiding? What did TWRP actually change?

Screenshots (note that TWRP has the wrong timezone set so the time shown is wrong):

Some more thoughts for you to consider:
1. Have you tried this with the official TWRP recovery version 3.2.0-0?
2. Is there anything inside the folders that you see using the file manager after a regular boot? Folders of same name may exist in the boot ramdisk and these are merged with system folders after boot.
3. Try wiping cache between reboots and see if that changes any of your observations.

 

[BillGoss]

Some more thoughts for you to consider:
1. Have you tried this with the official TWRP recovery version 3.2.0-0?
2. Is there anything inside the folders that you see using the file manager after a regular boot? Folders of same name may exist in the boot ramdisk and these are merged with system folders after boot.
3. Try wiping cache between reboots and see if that changes any of your observations.

Good questions. They got me thinking more about how this could possibly work.
I had a look at the cache and there's definitely no copy of the system hiding there.
I also unpacked the ramdisk in the boot image and it had nothing in system. Furthermore, the boot position is only 64 MB, no where near enough to hold the system.
Then I installed Magisk so that I could browse around the phone's partitions and take copies.

I learnt two things from this:
1. If there's a second copy of the system there are only three partitions large enough to hold it (/proc/partitions shows the sizes in 1 kB blocks). The system is about 1 GB. There is space in the system partition (sde20) for 3 GB. There's also space in the data partition (sca15). And there's space in the major partition holding the modems (sdf).
I could eliminate the data partition by formatting it but restoring the internal storage (sdcard) is such a a pain.
So I'll just accept that there is space for a copy, but I'm unlikely to find out exactly where.
2. When I had Magisk installed installed and the system boot, I added a folder and file to /system/priv-app using a file manager (so not using TWRP). I then booted into recovery, flashed the stock boot image, and rebooted. I was expecting it to fail dm-verity (modified system) but it didn't. After booting up there's no evidence of the folder I added to priv-app.
And if I restore the Magisk boot image then the additions show up again.

I'm actually very impressed with how the stock system (kernel, recovery, system) protects itself from modification. Very cool!