[RECOVERY][ROOT]TWRP 3.2.3-1 Galaxy Tab S4 - T830/T835

[ianmacd]

Yep you're correct the patched boot was intended to disable dm-verity, however it won't work because of the dtb issue.
Unless the dtb can be patched no modifications are possible.

I was tired last night and so focused on disabling dm-verity that I didn't stop to think about the contents of the boot image I was flashing. Of course it could never work, because it still had the original DTB. I woke up this morning and immediately realised my mistake.

Upon checking the stock and repacked dtb the headers are indeed different which I guess is why it won't boot.

So I have taken a different tack and hex edited the dtb to preserve the header. Now I have done this two different ways as I don't know which will be successful, if any.
Have also recompiled TWRP with fixes for mounting the partitions with errors and fixed some errors in the forced encryption patch.

Just to note, the encryption patch modifies the vendor partition which means dm-verity will be triggered. This is fine as long as the patched dtb boots and does it's job and disables dm-verity!

Thanks for the new build of TWRP. Unfortunately, it still fails to mount odm and still complains that /system is busy. Whatever is being mounted as /system isn't really /system, though. It contains only 8 Mb of data.

df -h on the device reveals that /dev/block/sda20 is being used for /data, /sdcard, /system and /system_image.

Similarly, /cache and /efs both share sda23. /vendor is the only one that is uniquely assigned, getting sda21. That can't be right, can it?

I have spent the morning playing with the 2 new boot images. Using Odin, I performed a fresh stock firmware installation prior to each installation. Odin was also used to install the boot images Sadly, both result in Verification Failed when Android comes up, so I never get as far as testing the encryption patch.

After the verification failure of the second image, I decided to reboot to TWRP and walk through the various screens and their functions in order to produce a meaningful log for you. I untarred boot.img from the boot1 image and flashed that, following immediately wish a format of DATA. The screen froze (the progress animation at the bottom of the screen seized) and the system rebooted. This was the reboot phenomenon I spoke of yesterday.

Back in TWRP, I grabbed a last_kmsg for you, but I don't know if it contains any clues as to why the system rebooted. This is only the second time I have triggered a reboot by formatting DATA.

I tried exactly the same thing again: flashed boot.img and then immediately formatted DATA. This time, no reboot. It seems to occur only the first time DATA is formatted after receiving a verification error when booting into Android.

Next, I tried mounting all of the file systems.

Next, I did a back-up of boot and system. That seemed to work fine.

Next, I did a simple wipe of DATA. Again, this caused an immediate reboot, but this time the device's vibration came on and stayed on until the Samsung splash screen came up.

Back in TWRP, I produced a second last_kmsg for you.

I tried another simple wipe of DATA, this time with no reboot. I followed it with a format of DATA. Again, no reboot.

If those sda devices are genuinely wrongly assigned, then that would explain all of the weirdness, I guess.

I'm attaching a log this time. I meant to do this last night, but I fell asleep on the couch. The dmesg portion is too big for XDA, so you can download it here.

This device is starting to irritate me. It doesn't have a hard Home button, so rebooting into Download mode requires one to first go through Recovery. Even turning the machine off is impossible through TWRP. Shutdowns always reboot, no matter how they're attempted. I have to reinstall stock firmware just to be able to turn it off.

Anyway, thanks for your persistence. This device is a tricky one.

 

[ashyx]

I was tired last night and so focused on disabling dm-verity that I didn't stop to think about the contents of the boot image I was flashing. Of course it could never work, because it still had the original DTB. I woke up this morning and immediately realised my mistake.

Upon checking the stock and repacked dtb the headers are indeed different which I guess is why it won't boot.



Thanks for the new build of TWRP. Unfortunately, it still fails to mount odm and still complains that /system is busy. Whatever is being mounted as /system isn't really /system, though. It contains only 8 Mb of data.

df -h on the device reveals that /dev/block/sda20 is being used for /data, /sdcard, /system and /system_image.

Similarly, /cache and /efs both share sda23. /vendor is the only one that is uniquely assigned, getting sda21. That can't be right, can it?

I have spent the morning playing with the 2 new boot images. Using Odin, I performed a fresh stock firmware installation prior to each installation. Odin was also used to install the boot images Sadly, both result in Verification Failed when Android comes up, so I never get as far as testing the encryption patch.

After the verification failure of the second image, I decided to reboot to TWRP and walk through the various screens and their functions in order to produce a meaningful log for you. I untarred boot.img from the boot1 image and flashed that, following immediately wish a format of DATA. The screen froze (the progress animation at the bottom of the screen seized) and the system rebooted. This was the reboot phenomenon I spoke of yesterday.

Back in TWRP, I grabbed a last_kmsg for you, but I don't know if it contains any clues as to why the system rebooted. This is only the second time I have triggered a reboot by formatting DATA.

I tried exactly the same thing again: flashed boot.img and then immediately formatted DATA. This time, no reboot. It seems to occur only the first time DATA is formatted after receiving a verification error when booting into Android.

Next, I tried mounting all of the file systems.

Next, I did a back-up of boot and system. That seemed to work fine.

Next, I did a simple wipe of DATA. Again, this caused an immediate reboot, but this time the device's vibration came on and stayed on until the Samsung splash screen came up.

Back in TWRP, I produced a second last_kmsg for you.

I tried another simple wipe of DATA, this time with no reboot. I followed it with a format of DATA. Again, no reboot.

If those sda devices are genuinely wrongly assigned, then that would explain all of the weirdness, I guess.

I'm attaching a log this time. I meant to do this last night, but I fell asleep on the couch. The dmesg portion is too big for XDA, so you can download it here.

This device is starting to irritate me. It doesn't have a hard Home button, so rebooting into Download mode requires one to first go through Recovery. Even turning the machine off is impossible through TWRP. Shutdowns always reboot, no matter how they're attempted. I have to reinstall stock firmware just to be able to turn it off.

Anyway, thanks for your persistence. This device is a tricky one.

Thanks for all that info.
I think I've nailed the mount issues now.
Hopefully if that is fixed I'll move on to this damn dm-verity issue. TBH I can't understand why the patched dtb isn't working. I know it's patched correctly and there is certainly no sign of any dm-verity flags. I've unpacked and checked, so no idea where the system is getting it's dm-verity flag from from?

Can you humour me and try something out?
Re-install complete stock, no TWRP, no modifications then just flash T830XXU1ARH8_dmverity_patched_boot1 and see if the OS actually boots?

Hopefully this build of TWRP sorts the mount issues: https://androidfilehost.com/?fid=1322778262904010020

 

[ianmacd]

Thanks for all that info.
I think I've nailed the mount issues now.
Hopefully if that is fixed I'll move on to this damn dm-verity issue. TBH I can't understand why the patched dtb isn't working. I know it's patched correctly and there is certainly no sign of any dm-verity flags. I've unpacked and checked, so no idea where the system is getting it's dm-verity flag from from?

Can you humour me and try something out?
Re-install complete stock, no TWRP, no modifications then just flash T830XXU1ARH8_dmverity_patched_boot1 and see if the OS actually boots?

OK, we've made some more progress with this build.

I first reinstalled the stock firmware, booted once into system, and then flashed the boot1 boot image as you requested. That yielded the now familiar Verification Failed.

I reset the system as demanded by the device, only this time it still showed the open padlock system and the word Custom while booting, and took a long time to come up, but it did eventually come up. On previous attempts to reset the device after a Verification Failed, the device had got itself stuck in a bootloop, and I had had to reinstall the firmware from Odin. But now it had apparently booted with the patched boot image!

To make absolutely sure, I booted back to Download Mode and then flashed the boot1 boot image a second time. After a reboot to system, the device came up again! I'm really not sure what brought about this change in behaviour, but probably something to do with having fixed those mismapped block devices.

Anyway, this was good news. I booted back to Download Mode and flashed your new TWRP build. Everything mounted beautifully, including odm. df confirmed that each file-system was now properly mapped to a unique block device.

I flashed the force-encryption disabler and then attempted to format DATA. That failed. Unable to format to remove encryption.

I tried rebooting to TWRP and just wiping DATA this time. Again, it failed. Unable to wipe Data.

I decided to reboot to system at this point, but it failed with Device Storage Damaged. Something clearly went badly wrong when trying to format DATA.

So, the good news is that the mounting is fixed, along with the dm-verity anomaly apparently, even if I can't quite grasp why.

Now there's just the DATA formatting issue to contend with.

The recovery log is attached. dmesg can be found here.

 

[ashyx]

OK, we've made some more progress with this build.

I first reinstalled the stock firmware, booted once into system, and then flashed the boot1 boot image as you requested. That yielded the now familiar Verification Failed.

I reset the system as demanded by the device, only this time it still showed the open padlock system and the word Custom while booting, and took a long time to come up, but it did eventually come up. On previous attempts to reset the device after a Verification Failed, the device had got itself stuck in a bootloop, and I had had to reinstall the firmware from Odin. But now it had apparently booted with the patched boot image!

To make absolutely sure, I booted back to Download Mode and then flashed the boot1 boot image a second time. After a reboot to system, the device came up again! I'm really not sure what brought about this change in behaviour, but probably something to do with having fixed those mismapped block devices.

Anyway, this was good news. I booted back to Download Mode and flashed your new TWRP build. Everything mounted beautifully, including odm. df confirmed that each file-system was now properly mapped to a unique block device.

I flashed the force-encryption disabler and then attempted to format DATA. That failed. Unable to format to remove encryption.

I tried rebooting to TWRP and just wiping DATA this time. Again, it failed. Unable to wipe Data.

I decided to reboot to system at this point, but it failed with Device Storage Damaged. Something clearly went badly wrong when trying to format DATA.

So, the good news is that the mounting is fixed, along with the dm-verity anomaly apparently, even if I can't quite grasp why.

Now there's just the DATA formatting issue to contend with.

The recovery log is attached. dmesg can be found here.

That's good news that the mount issues are now sorted and dm-verity is fixed. :good:

Was a bit worried when you said you had issues with Formatting and mounting /DATA. At first I thought your EMMC had crashed, however it seems OK as all the other partitions on the EMMC are mounted.

According to the the recovery log /DATA doesn't have a block device associated with it.


I'm guessing at some point dev/block/sda27 has become corrupt possibly due to bad blocks. Especially considering how previously you remarked that TWRP would randomly reboot when attempting to Format /DATA.

You can try and fix this with E2FSCK.

First boot to recovery and copy the BADBLOCKS binary to /sbin
https://androidfilehost.com/?fid=1322778262904010583

Using terminal run:
chmod 0755 sbin/badblocks

Check block sda27 is actually still there:
cat /proc/partitions

Make sure /DATA is unmounted:
umount /dev/block/sda27
(don't worry if you get 'invalid argument' just means sda27 is already unmounted)

Run the following command:
e2fsck -cfv /dev/block/sda27

(just for info on the syntax -c is use the badblocks binary to scan for bad blocks, f is force a check, v is verbose mode)

Note if there are any bad blocks and if e2fsck fixes them.

Hopefully that's all that's needed.

By the way I have patched the kernel source to include the dm-verity changes. The kernel now builds with the patched dtb, so no unpacking/repacking or hex editing needed.

 

[ianmacd]

I'm guessing at some point dev/block/sda27 has become corrupt possibly due to bad blocks. Especially considering how previously you remarked that TWRP would randomly reboot when attempting to Format /DATA.

You can try and fix this with E2FSCK.

First boot to recovery and copy the BADBLOCKS binary to /sbin
https://androidfilehost.com/?fid=1322778262904010583

Using terminal run:
chmod 0755 sbin/badblocks

Check block sda27 is actually still there:
cat /proc/partitions

Make sure /DATA is unmounted:
umount /dev/block/sda27
(don't worry if you get 'invalid argument' just means sda27 is already unmounted)

Run the following command:
e2fsck -cfv /dev/block/sda27

(just for info on the syntax -c is use the badblocks binary to scan for bad blocks, f is force a check, v is verbose mode)

Note if there are any bad blocks and if e2fsck fixes them.

Hopefully that's all that's needed.

Bad magic number in superblock.
Superblock invalid, trying backup blocks...

And then it fails. What's going on here?

TWRP's fdisk doesn't show file-system code types, so I can't verify that this is supposed to be an ext4 fs. I can't see another tool that will tell me this, short of catting the raw sda device.

I can install stock firmware and it comes up at the installation screen with no error. I guess I'll run through the Android installation and satisfy myself that the file-system seems OK that way. If not, I guess I'll have to reinstall with the normal CSC file (not HOME_CSC) and force a full format of the device. I'm hoping there's some other explanation than file-system damage, however.

By the way I have patched the kernel source to include the dm-verity changes. The kernel now builds with the patched dtb, so no unpacking/repacking or hex editing needed.

That's great.

Android reinstalled without issue. There are no errors anywhere, not even under Settings -> Storage.

Is this relevant at all?

gts4lwifi:/ $ df
Filesystem                     1K-blocks    Used Available Use% Mounted on
rootfs                           1583124    8704   1574420   1% /
tmpfs                            1682100    1020   1681080   1% /dev
/dev/block/dm-0                   597348  450872    146476  76% /odm
/dev/block/dm-1                  4064008 3219600    844408  80% /system
/dev/block/dm-2                   671392  461272    210120  69% /vendor
tmpfs                            1682100       0   1682100   0% /mnt
tmpfs                            1682100       0   1682100   0% /mnt/secure
/dev/block/sda16                   97232   58448     38784  61% /firmware
/dev/block/sdd7                    12016    4552      7464  38% /dsp
/dev/block/sda5                    28144     184     27960   1% /persist
/dev/block/sda23                  396744    1012    395732   1% /cache
/dev/block/sda6                    16048     620     15428   4% /efs
/dev/block/sda15                   97232   58448     38784  61% /firmware
/dev/block/sda16                  112576   75024     37552  67% /firmware-modem
/dev/block/sda25                   46288      12     46276   1% /omr
tmpfs                            1682100       0   1682100   0% /storage
/dev/block/dm-3                 55230680 1491084  53647436   3% /data
/data/knox/secure_fs/enc_user   55230680 1491084  53647436   3% /data/enc_user
/data/knox/secure_fs/enc_media  55230680 1491084  53647436   3% /data/knox/secure_fs/enc_media
/data/knox/secure_fs/enc_media  55210200 1583244  53626956   3% /mnt/shell/enc_emulated
/data/media                     55210200 1583244  53626956   3% /storage/emulated
/mnt/media_rw/0000-0000        250019840  733952 249285888   1% /storage/0000-0000

gts4lwifi:/ $ ls -l /dev/block/dm-3
brw------- 1 root root 254,   3 2017-02-14 18:56 /dev/block/dm-3

/data is mounted from /dev/block/dm-3 in Android, not from /dev/block/sda27. Could that be part of the problem? On the other hand, I see other dm devices in that output, too, and those are still mountable from their sda devices.

Let's see what mount has to say:

gts4lwifi:/ $ mount | grep /data
/dev/block/dm-3 on /data type ext4 (rw,seclabel,nosuid,nodev,noatime,journal_checksum,noauto_da_alloc,resgid=5555,i_version)
/data/knox/secure_fs/enc_user on /data/enc_user type ecryptfs (rw,seclabel,nodev,relatime,ecryptfs_fnek_sig=3046d5c9d28bb630,ecryptfs_sig=3046d5c9d28bb630,userid=0,sdp_enabled,partition_id=0,ecryptfs_cipher=aes,ecryptfs_key_bytes=32,ecryptfs_enable_cc,ecryptfs_passthrough,base=,label=)
/data/knox/secure_fs/enc_media on /data/knox/secure_fs/enc_media type ecryptfs (rw,seclabel,nodev,relatime,ecryptfs_fnek_sig=3046d5c9d28bb630,ecryptfs_sig=3046d5c9d28bb630,userid=0,sdp_enabled,partition_id=1,ecryptfs_cipher=aes,ecryptfs_key_bytes=32,ecryptfs_enable_cc,ecryptfs_passthrough,base=,label=)
/data/knox/secure_fs/enc_media on /mnt/shell/enc_emulated type sdcardfs (rw,nosuid,nodev,noexec,noatime,fsuid=1000,fsgid=1000,gid=9997,multiuser,derive_gid,reserved=20MB)
/data/media on /mnt/runtime/default/emulated type sdcardfs (rw,nosuid,nodev,noexec,noatime,fsuid=1023,fsgid=1023,gid=1015,multiuser,mask=6,derive_gid,reserved=20MB)
/data/media on /storage/emulated type sdcardfs (rw,nosuid,nodev,noexec,noatime,fsuid=1023,fsgid=1023,gid=1015,multiuser,mask=6,derive_gid,reserved=20MB)
/data/media on /mnt/runtime/read/emulated type sdcardfs (rw,nosuid,nodev,noexec,noatime,fsuid=1023,fsgid=1023,gid=9997,multiuser,mask=23,derive_gid,reserved=20MB)
/data/media on /mnt/runtime/write/emulated type sdcardfs (rw,nosuid,nodev,noexec,noatime,fsuid=1023,fsgid=1023,gid=9997,multiuser,mask=7,derive_gid,reserved=20MB)

So /data is ext4. Why can't e2fsck read a superblock from /dev/block/sda27 then? Weird.

Hopefully you can make some sense of all of this.

 

[ashyx]

Bad magic number in superblock.
Superblock invalid, trying backup blocks...

And then it fails. What's going on here?

TWRP's fdisk doesn't show file-system code types, so I can't verify that this is supposed to be an ext4 fs. I can't see another tool that will tell me this, short of catting the raw sda device.

I can install stock firmware and it comes up at the installation screen with no error. I guess I'll run through the Android installation and satisfy myself that the file-system seems OK that way. If not, I guess I'll have to reinstall with the normal CSC file (not HOME_CSC) and force a full format of the device. I'm hoping there's some other explanation than file-system damage, however.



That's great.

Android reinstalled without issue. There are no errors anywhere, not even under Settings -> Storage.

Is this relevant at all?

gts4lwifi:/ $ df
Filesystem                     1K-blocks    Used Available Use% Mounted on
rootfs                           1583124    8704   1574420   1% /
tmpfs                            1682100    1020   1681080   1% /dev
/dev/block/dm-0                   597348  450872    146476  76% /odm
/dev/block/dm-1                  4064008 3219600    844408  80% /system
/dev/block/dm-2                   671392  461272    210120  69% /vendor
tmpfs                            1682100       0   1682100   0% /mnt
tmpfs                            1682100       0   1682100   0% /mnt/secure
/dev/block/sda16                   97232   58448     38784  61% /firmware
/dev/block/sdd7                    12016    4552      7464  38% /dsp
/dev/block/sda5                    28144     184     27960   1% /persist
/dev/block/sda23                  396744    1012    395732   1% /cache
/dev/block/sda6                    16048     620     15428   4% /efs
/dev/block/sda15                   97232   58448     38784  61% /firmware
/dev/block/sda16                  112576   75024     37552  67% /firmware-modem
/dev/block/sda25                   46288      12     46276   1% /omr
tmpfs                            1682100       0   1682100   0% /storage
/dev/block/dm-3                 55230680 1491084  53647436   3% /data
/data/knox/secure_fs/enc_user   55230680 1491084  53647436   3% /data/enc_user
/data/knox/secure_fs/enc_media  55230680 1491084  53647436   3% /data/knox/secure_fs/enc_media
/data/knox/secure_fs/enc_media  55210200 1583244  53626956   3% /mnt/shell/enc_emulated
/data/media                     55210200 1583244  53626956   3% /storage/emulated
/mnt/media_rw/0000-0000        250019840  733952 249285888   1% /storage/0000-0000

gts4lwifi:/ $ ls -l /dev/block/dm-3
brw------- 1 root root 254,   3 2017-02-14 18:56 /dev/block/dm-3

/data is mounted from /dev/block/dm-3 in Android, not from /dev/block/sda27. Could that be part of the problem? On the other hand, I see other dm devices in that output, too, and those are still mountable from their sda devices.

Let's see what mount has to say:

gts4lwifi:/ $ mount | grep /data
/dev/block/dm-3 on /data type ext4 (rw,seclabel,nosuid,nodev,noatime,journal_checksum,noauto_da_alloc,resgid=5555,i_version)
/data/knox/secure_fs/enc_user on /data/enc_user type ecryptfs (rw,seclabel,nodev,relatime,ecryptfs_fnek_sig=3046d5c9d28bb630,ecryptfs_sig=3046d5c9d28bb630,userid=0,sdp_enabled,partition_id=0,ecryptfs_cipher=aes,ecryptfs_key_bytes=32,ecryptfs_enable_cc,ecryptfs_passthrough,base=,label=)
/data/knox/secure_fs/enc_media on /data/knox/secure_fs/enc_media type ecryptfs (rw,seclabel,nodev,relatime,ecryptfs_fnek_sig=3046d5c9d28bb630,ecryptfs_sig=3046d5c9d28bb630,userid=0,sdp_enabled,partition_id=1,ecryptfs_cipher=aes,ecryptfs_key_bytes=32,ecryptfs_enable_cc,ecryptfs_passthrough,base=,label=)
/data/knox/secure_fs/enc_media on /mnt/shell/enc_emulated type sdcardfs (rw,nosuid,nodev,noexec,noatime,fsuid=1000,fsgid=1000,gid=9997,multiuser,derive_gid,reserved=20MB)
/data/media on /mnt/runtime/default/emulated type sdcardfs (rw,nosuid,nodev,noexec,noatime,fsuid=1023,fsgid=1023,gid=1015,multiuser,mask=6,derive_gid,reserved=20MB)
/data/media on /storage/emulated type sdcardfs (rw,nosuid,nodev,noexec,noatime,fsuid=1023,fsgid=1023,gid=1015,multiuser,mask=6,derive_gid,reserved=20MB)
/data/media on /mnt/runtime/read/emulated type sdcardfs (rw,nosuid,nodev,noexec,noatime,fsuid=1023,fsgid=1023,gid=9997,multiuser,mask=23,derive_gid,reserved=20MB)
/data/media on /mnt/runtime/write/emulated type sdcardfs (rw,nosuid,nodev,noexec,noatime,fsuid=1023,fsgid=1023,gid=9997,multiuser,mask=7,derive_gid,reserved=20MB)

So /data is ext4. Why can't e2fsck read a superblock from /dev/block/sda27 then? Weird.

Hopefully you can make some sense of all of this.

Seems e2fsck doesn't work on encrypted partitons?
In any case /data looks fine as it's mounted as device mapper partion 3

Can you post the out of the following via recovery again?:


ls -laR /dev/block > external_sd/block2.txt

 

[suzook]

Getting my replacement tab today, or tomorrow. Is this a go? Or still testing? I'm willing to test.

 

[ashyx]

Getting my replacement tab today, or tomorrow. Is this a go? Or still testing? I'm willing to test.

We have it booting and all partitions mounting.
Custom kernel patched to disable dm-verity and selinux.
Patch to disable encryption.
At the moment hit a snag with Formatting /data.

 

[ianmacd]

Can you post the out of the following via recovery again?:

ls -laR /dev/block > external_sd/block2.txt

Attached.

This one was made using adb in Android, so I got permission denied on /dev/block/vold, but since you're not interested in that, I think this is sufficient.

 

[ashyx]

Attached.

This one was made using adb in Android, so I got permission denied on /dev/block/vold, but since you're not interested in that, I think this is sufficient.

Ok thanks, I think I have fixed the Format issue with DATA.

https://androidfilehost.com/?fid=1322778262904010852

 

[ianmacd]

Ok thanks, I think I have fixed the Format issue with DATA.

After flashing the boot1 boot image and the forced encryption disabler ZIP, I formatted DATA and... bingo! No error this time.

I then attempted to boot to system, but when the device came back up, it reported Device Storage Damaged again. All it will allow me to do at that point is reset it, but even after the reset, it still comes back up with Storage Damaged again. Weird, because I can mount /data fine in TWRP. I do note that both /sdcard and /data are sharing /dev/block/sda27. Is that correct?

Anyway, at this point, only a reflash of stock firmware restores order.

A log is attached for you.

I'll play with this a while longer and see if I can get anywhere.

 

[ashyx]

After flashing the boot1 boot image and the forced encryption disabler ZIP, I formatted DATA and... bingo! No error this time.

I then attempted to boot to system, but when the device came back up, it reported Device Storage Damaged again. All it will allow me to do at that point is reset it, but even after the reset, it still comes back up with Storage Damaged again. Weird, because I can mount /data fine in TWRP. I do note that both /sdcard and /data are sharing /dev/block/sda27. Is that correct?

Anyway, at this point, only a reflash of stock firmware restores order.

A log is attached for you.

I'll play with this a while longer and see if I can get anywhere.

Good stuff, TWRP log says everything is OK, no errors or mount issues...:victory:

I'm guessing the storage damaged error is due to encryption being removed. Seems it doesn't like that.

Anyway just for the sake of it I have uploaded the custom patched kernel. Maybe you'll have more luck with this (if it boots?)
https://androidfilehost.com/?fid=1322778262904010898

EDIT. I suggest may be flashing the userdata.img.ext.lz4 image in the AP part of the firmware with Odin.
Will need tar'ing first.
It shouldn't re-encrypt the device.

 

[ianmacd]

Good stuff, TWRP log says everything is OK, no errors or mount issues...:victory:

I'm guessing the storage damaged error is due to encryption being removed. Seems it doesn't like that.

Anyway just for the sake of it I have uploaded the custom patched kernel. Maybe you'll have more luck with this (if it boots?)
https://androidfilehost.com/?fid=1322778262904010898

EDIT. I suggest may be flashing the userdata.img.ext.lz4 image in the AP part of the firmware with Odin.
Will need tar'ing first.
It shouldn't re-encrypt the device.

I'm pulling my hair out a bit at this point.

The custom patched kernel results in Verification Failed, even when I flash it from Odin with no other modifications on the system. But when I tried the old boot1 image as a scientific control, that, too, gave me Verification Failed. Hitting the reset button demanded by the device reinstalls the machine, but it still comes up with Verification Failed afterwards.

Aargh. You may recall I had this earlier in the testing, too. Some combination of reinstalling stock firmware + a custom boot partition, plus performing actions in TWRP eventually results in a set of circumstances that allows the post-verification failure reset action to succeed, at which point the system comes up clean, with the custom boot partition presumably still intact. However, I can't find a predictable set of actions to yield this result every time. And I don't understand why these boot images fail verification in the first place, because they disable dm-verity. Right? They should just boot clean after flashing.

Anyway, once I can cleanly boot to system again, I try formatting DATA, but then when I reboot, it's the familiar Device Storage Corrupt again.

I've extracted userdata.img.ext4.lz4 from the AP firmware, tarred and flashed it in Odin, but it makes no difference. Now I'm not sure whether I tried flashing it both before and after the reboot following formatting DATA. I definitely tried it after the reboot, so I'll also try doing it before.

Something in this device really doesn't like DATA being formatted. It doesn't seem security-related, however, because the complaint is data corruption, not a security violation per se. I assume TWRP is calling mke2fs to do the formatting. I'm wondering if we might try tweaking a parameter there. Maybe I should run dumpe2fs on /dev/block/sda27 before formatting it and see if that yields anything notable. I hope dumpe2fs is part of the TWRP suite.

TWRP can mount and read /data just fine after formatting. It looks and behaves like /data on any other TWRP'ed device. It's only when booting to system that it all goes to hell. It's the last important piece of the puzzle. When we figure that out, we'll have a usable recovery.

BTW, if you are getting sick of this (or even if you aren't), I'd be happy to sync your TWRP tree and start experimenting with it at this end, too. I'm unfamiliar with TWRP internals, but I could certainly build it and try some changes of my own. The same goes for editing the DTB. I could do that, too, although it probably needs no further edits at this stage. I'd love to have your working custom kernel .config, too.

I guess what I'm saying is that you needn't be the only person able to do the building. I'd love to learn from you how to do this stuff (because the TWRP documentation on the subject is sorely lacking). You've applied the same methodology to the Tab S2, the Tab S3 and now the Tab S4. Next year, when the Tab S5 will presumably surface, it'd be great if the onus weren't on you alone to produce TWRP for it. A one page how-to from you would probably save 100 hours of experimentation with only the official TWRP for guidance.

 

[ianmacd]

@ashyx,

Here's some additional information, after another session spent wrestling with the device.

Since I can format /data and then mount it in TWRP, I thought I'd have a go at installing Magisk, just to observe its behaviour and try to glean some new data points. So, I reinstalled the stock firmware, reflashed the boot1 boot image, formatted /data and then flashed Magisk. It installed without errors.

I then booted into system without first installing the forced encryption disabler, because Magisk should theoretically have done the same work. Just as the system was about to come up, it automatically rebooted. Damn, I thought. This was doubtless going to lead to a bootloop, Verification Failed or Device Storage Corrupt. To my surprise, however, it spent a long while with the flashing Samsung logo and then came up. No doubt that time was spent re-encrypting /data.

Well, that was interesting, because it meant that it wasn't necessarily the removal of encryption that was causing the Corrupt Storage error.

So, I then reinstalled the stock firmware and repeated the above steps, only this time I added a flash of the forced encryption zip prior to installing Magisk. Sure enough, after installing Magisk and rebooting to system, the device complained that its storage was corrupt.

Just to be sure, I repeated the first installation all over again, this time leaving out the forced encryption disabler again. Again, after the lack of encryption caused a reboot to re-encrypt, the device came back up.

So, I quickly reinstalled Android on it and verified that it had, in fact, been rooted. It had!

Root is great progress. It's reproducible, but of limited use with an encrypted /data. But the interesting data point here is that it's not the actual removal of encryption that results in Corrupt Storage, rather that something seems to be amiss with the forced encryption disabler ZIP. Could you please double-check that file for errors?

I'm hopeful we're on the right track here.

UPDATE:

I've checked the forced encryption disabler myself, and I see this in your replacement for /vendor/etc/fstab.qcom:

/dev/block/bootdevice/by-name/userdata  /data                   ext4    noatime,nosuid,nodev,noauto_da_alloc,discard,journal_async_commit,data=ordered,errors=panic     wait,check,encryptable=footer

If I look at the original, however, I see this:

/dev/block/bootdevice/by-name/userdata   /data              ext4   noatime,nosuid,nodev,noauto_da_alloc,discard,journal_checksum,data=ordered,errors=panic   wait,check,forceencrypt=footer,quota

Why the removal of journal_checksum and the addition of journal_async_commit? I see from the mount man page that using the latter will also automatically enable the former, but why the change?

And why no quota included at the end of the encryptable mount option?

I'm going to try bringing these in-line with the original to see what happens.

2nd UPDATE:

Well, the good news is that restoring those options stops the machine complaining about Corrupt Storage.

The bad news is that the machine ignores encryptable entirely and still re-encrypts the device on first boot. Is this option perhaps clashing with something in the DTB?

I'm attaching the fstab.qcom that I'm now flashing.

 

[ashyx]

@ashyx,

Here's some additional information, after another session spent wrestling with the device.

Since I can format /data and then mount it in TWRP, I thought I'd have a go at installing Magisk, just to observe its behaviour and try to glean some new data points. So, I reinstalled the stock firmware, reflashed the boot1 boot image, formatted /data and then flashed Magisk. It installed without errors.

I then booted into system without first installing the forced encryption disabler, because Magisk should theoretically have done the same work. Just as the system was about to come up, it automatically rebooted. Damn, I thought. This was doubtless going to lead to a bootloop, Verification Failed or Device Storage Corrupt. To my surprise, however, it spent a long while with the flashing Samsung logo and then came up. No doubt that time was spent re-encrypting /data.

Well, that was interesting, because it meant that it wasn't necessarily the removal of encryption that was causing the Corrupt Storage error.

So, I then reinstalled the stock firmware and repeated the above steps, only this time I added a flash of the forced encryption zip prior to installing Magisk. Sure enough, after installing Magisk and rebooting to system, the device complained that its storage was corrupt.

Just to be sure, I repeated the first installation all over again, this time leaving out the forced encryption disabler again. Again, after the lack of encryption caused a reboot to re-encrypt, the device came back up.

So, I quickly reinstalled Android on it and verified that it had, in fact, been rooted. It had!

Root is great progress. It's reproducible, but of limited use with an encrypted /data. But the interesting data point here is that it's not the actual removal of encryption that results in Corrupt Storage, rather that something seems to be amiss with the forced encryption disabler ZIP. Could you please double-check that file for errors?

I'm hopeful we're on the right track here.

UPDATE:

I've checked the forced encryption disabler myself, and I see this in your replacement for /vendor/etc/fstab.qcom:

/dev/block/bootdevice/by-name/userdata  /data                   ext4    noatime,nosuid,nodev,noauto_da_alloc,discard,journal_async_commit,data=ordered,errors=panic     wait,check,encryptable=footer

If I look at the original, however, I see this:

/dev/block/bootdevice/by-name/userdata   /data              ext4   noatime,nosuid,nodev,noauto_da_alloc,discard,journal_checksum,data=ordered,errors=panic   wait,check,forceencrypt=footer,quota

Why the removal of journal_checksum and the addition of journal_async_commit? I see from the mount man page that using the latter will also automatically enable the former, but why the change?

And why no quota included at the end of the encryptable mount option?

I'm going to try bringing these in-line with the original to see what happens.

2nd UPDATE:

Well, the good news is that restoring those options stops the machine complaining about Corrupt Storage.

The bad news is that the machine ignores encryptable entirely and still re-encrypts the device on first boot.

I'm attaching the fstab.qcom that I'm now flashing.

Holy smokes I've got a massive apology for wasting your time! When I looked at the Fstab you posted it was completely different to the one in the disabler patch.

I realised I had included the WRONG fstab!
:silly:

I can only apologise for this stupid mistake. Too many things going on at once.
I have uploaded the correct fstab patch:
https://androidfilehost.com/?fid=1322778262904011574
Quota needs to be removed or the device will re-encrypt.
Thanks for pointing this out, I'll now go and crawl under a rock for a while... :crying:

 

[ianmacd]

Holy smokes I've got a massive apology for wasting your time! When I looked at the Fstab you posted it was completely different to the one in the disabler patch.

I realised I had included the WRONG fstab!
:silly:

I can only apologise for this stupid mistake. Too many things going on at once.
I have uploaded the correct fstab patch:
https://androidfilehost.com/?fid=1322778262904011574
Quota needs to be removed or the device will re-encrypt.
Thanks for pointing this out, I'll now go and crawl under a rock for a while... :crying:

Come out from under your rock, because that was it. With /data reformatted and the fstab flashed with quota removed, the system now comes up with /data unencrypted. Fantastic!

I'm going to reinstall the stock firmware in a few minutes and retrace the procedure all the way from the top, just to verify my findings, but it appears we now have everything mounting, data formatting, and the ability to root.

As far as I can tell, all that's left is getting adb going. Or does TWRP also rely for that on USB Debugging being on in system? Because that gets turned off with every device reset, of course. Well, I'll quickly reinstall Android, turn the option on, reboot to TWRP and answer my own question.

UPDATE:

I have my answer: adb is still not working in TWRP, even after turning on USB Debugging in system.

 

[ashyx]

Come out from under your rock, because that was it. With /data reformatted and the fstab flashed with quota removed, the system now comes up with /data unencrypted. Fantastic!

I'm going to reinstall the stock firmware in a few minutes and retrace the procedure all the way from the top, just to verify my findings, but it appears we now have everything mounting, data formatting, and the ability to root.

As far as I can tell, all that's left is getting adb going. Or does TWRP also rely for that on USB Debugging being on in system? Because that gets turned off with every device reset, of course. Well, I'll quickly reinstall Android, turn the option on, reboot to TWRP and answer my own question.

Debugging isn't required for recovery as it plays no part. It's only the OS that is dependent on debugging being enabled.
ADB should be the first thing that is enabled by init.
Odd it's not working, have you tried it with MTP disabled?

Also is the custom patched kernel booting OK now?

By the way thanks for all your assistance with this, it's been an absolute pleasure to not have to hold someone's hand through the whole process.
Seems you've got a good grasp on this stuff and expect to see good things come from you. I have already noticed your custom kernel thread. Great stuff. :good::good::good:

 

[ianmacd]

Debugging isn't required for recovery as it plays no port. It's only the OS that is dependent on debugging being enabled.
ADB should be the first thing that is enabled by init.

Odd it's not working, have you tried it with MTP disabled?

Thanks for clarifying that USB Debugging has no bearing on TWRP.

Yes, I tried it with MTP disabled, too: no dice.

adb in TWRP is a nice-to-have, but not essential. I suspect a final DTB edit will be needed to nail it.

I'm reinstalling the stock firmware as I type this, and will then go through the entire procedure of installing TWRP, removing dm-verity via the modded boot image, removing forced encryption, formatting /data, installing Magisk, and finally installing Android.

I'll report back when finished.

UPDATE:

I just saw that you edited your post while I was replying, so I'll reply now to the stuff you added.

Also is the custom patched kernel booting OK now?

If you mean the one you uploaded last night, I haven't tried it since. I've been using boot1 from post #40 in this thread. I'll use that one for my full procedure run-through in a second.

By the way thanks for all your assistance with this, it's been an absolute pleasure to not have to hold someone's hand through the whole process.
Seems you've got a good grasp on this stuff and expect to see good things come from you. I have already noticed your custom kernel thread. Great stuff.

Thank you.

I'd love to add better TWRP skills to my arsenal. I just need a good teacher. I don't mind being your remote-controlled arms, and I'm incredibly grateful that you sat this out for as long as it took, but I would have loved to be able to watch over your shoulder at the edits you've been making.

---------- Post added at 19:02 ---------- Previous post was at 18:23 ----------

@ashyx,

OK, I have run through the whole procedure from start to finish, even starting by flashing CSC instead of HOME_CSC, just to wipe the slate totally clean.

Here's what I did:

1. Reinstall stock firmware.
2. Reboot to system to verify correct installation of firmware.
3. Install your latest build of TWRP, using Odin.
4. Install last night's boot image with the custom kernel, using Odin.
5. Flash the newly fixed forced encryption disabler ZIP, using TWRP.
6. Format /data.
7. Reboot to TWRP.
8. Install Magisk.
9. Reboot to system. The system reboots itself once during this. Not sure why.
10. Install Android as usual.
11. adb shell, followed by su on the device.
12. Grant root to su using Magisk Manager on device. Root confirmed.

And just to demonstrate that the custom kernel is up and running:

gts4lwifi:/ # uname -a
Linux localhost 4.4.78+ #6 SMP PREEMPT Fri Sep 21 11:44:23 BST 2018 armv8l

Consider me a happy man!

I'm eager to get a custom kernel project started now. As you know, I already fixed the kernel sources a month ago. Can you please send me your .config? It will save me some time if I can see what you had to disable (or even enable) to get this kernel to boot.

Cheers! I can't thank you enough.

 

[ashyx]

Thanks for clarifying that USB Debugging has no bearing on TWRP.

Yes, I tried it with MTP disabled, too: no dice.

adb in TWRP is a nice-to-have, but not essential. I suspect a final DTB edit will be needed to nail it.

I'm reinstalling the stock firmware as I type this, and will then go through the entire procedure of installing TWRP, removing dm-verity via the modded boot image, removing forced encryption, formatting /data, installing Magisk, and finally installing Android.

I'll report back when finished.

UPDATE:

I just saw that you edited your post while I was replying, so I'll reply now to the stuff you added.



If you mean the one you uploaded last night, I haven't tried it since. I've been using boot1 from post #40 in this thread. I'll use that one for my full procedure run-through in a second.



Thank you.

I'd love to add better TWRP skills to my arsenal. I just need a good teacher. I don't mind being your remote-controlled arms, and I'm incredibly grateful that you sat this out for as long as it took, but I would have loved to be able to watch over your shoulder at the edits you've been making.

---------- Post added at 19:02 ---------- Previous post was at 18:23 ----------

@ashyx,

OK, I have run through the whole procedure from start to finish, even starting by flashing CSC instead of HOME_CSC, just to wipe the slate totally clean.

Here's what I did:

Reinstall stock firmware.
Reboot to system to verify correct installation of firmware.
Install your latest build of TWRP, using Odin.
Install last night's boot image with the custom kernel, using Odin.
Flash the newly fixed forced encryption disabler ZIP, using TWRP.
Format /data.
Reboot to TWRP.
Install Magisk.
Reboot to system. The system reboots itself once during this. Not sure why.
Install Android as usual.
adb shell, followed by su on the device.
Grant root to su using Magisk Manager on device. Root confirmed.


And just to demonstrate that the custom kernel is up and running:



Consider me a happy man!

I'm eager to get a custom kernel project started now. As you know, I already fixed the kernel sources a month ago. Can you please send me your .config? It will save me some time if I can see what you had to disable (or even enable) to get this kernel to boot.

Cheers! I can't thank you enough.

No worries . Much appreciate all your time and assistance.
I'll try and get both twrp and kernel up on github as soon as I get chance.
I'll also update this thread with a proper release page in the OP.

 

[ianmacd]

No worries . Much appreciate all your time and assistance.
I'll try and get both twrp and kernel up on github as soon as I get chance.
I'll also update this thread with a proper release page in the OP.

Cheers, @ashyx.

I only wish all of my time spent on XDA were as fruitful and as satisfying as the last week and 6 pages have been. It's been a pleasure to do this project with you.

This morning, I was actually wondering whether we were going to be able to continue with this, what with you not having the device and that troublesome data corruption issue plaguing me. It just goes to show that it's always darkest before the dawn, and that big effects sometimes have very simple causes. Thank goodness I discovered that fstab wasn't right.

And now I don't have to keep booting into Windows any more. Yay!

Have a great weekend, and if you're ever heading to Amsterdam, look me up. I'll buy you a beer or three.