FORUMS
Remove All Ads from XDA

[DEV][COMPLETE] Project Treble conversion

5,795 posts
Thanks Meter: 7,584
 
By CosmicDan, Senior Member on 15th May 2018, 12:52 AM
Thread Closed Email Thread
UPDATE4: New Treble guide is posted here. Ask all your questions here that aren't specific to other parts of Treble (e.g. the ROM or the TWRP). Thread lock requested. Thanks all!

UPDATE3: The new TWRP Installer with Treble Manager (for guided repartitioning) is now released. See this thread.

UDPATE2: Experimental RR builds for Treble is now released. This contains the required Vendor Pack for GSI use. See this thread.


UPDATE: We have succeed in booting a Phh-Treble GSI! See screenshot at Post #57. Stay tuned for user releases!


This thread will be kept for posterity, but I will request locking once we release stuff. Thanks you all for the support, and thanks to @ghpranav for lots of help and @phhusson for pointing me in the right direction on the last stretch.

Original post below

----------------------------------------------

Alright, here is my WIP guide for converting the Mi A1 to a Treble device and ROM.

NOT FOR NEWBIES. This is all manual stuff right now and if you do something wrong you CAN firm-brick your device! The project is incomplete and for developers/researchers only.

Please refrain from "good luck" and "can't wait for this" and "OMG you are the King of Kings ComsicDan, plz have my babies" and such posts.

Also note that you will NEED to get into EDL mode to restore stock partition table, if you need to, by flashing stock factory image with MiFlash or my Low-Level Backup Tool. MiFlash will WIPE your Userdata completely and your persist so you MUST make a low-level backup to restore original persist.img, and manually backup internal storage. But really it's perfectly fine to keep using a non-treble ROM even with partition table changes (there's no need to revert for any functional reason that I can think of, unless you somehow get a ROM that's bigger than 2.4GB). Note that the final update ZIP that will automatically repartition can also restore stock partitions, without the need for EDL stuff.


Prerequisite - Low-level backup, TWRP backup and Internal Storage backup

  1. First make a full backup of all partitions, via EDL (low-level which gets all partitions). You can use my new Low-Level tool here. I recommend using the skip-systems-and-userdata list since it takes like an hour, and you can backup system/userdata faster in TWRP and PC manually.
  2. Make a full TWRP backup of our current ROM. System needs to be erased during this procedure. After re-partitioning however, you can simply restore this backup (unless your system is > 2.4GB for some reason, which is very unlikely).
  3. Make a full manual backup of your internal storage files to PC. Userdata partition needs to be completely erased if/when you restore the original stock partitions and firmware. You can postpone this process to later however, for if/when you want to actually restore to stock partition layour.


Phase 1 - Repartition device for vendor_a and vendor_b


This is relatively simple. Our device has a standard GPT table, so we can use sgdisk which is already included in our TWRP build.
  1. Boot into recovery and fire up an adb shell
  2. View current partition table...
    Code:
    sgdisk /dev/block/mmcblk0 --print
    ...and confirm these values...
    Code:
    ...
      25         1185792         7477247   3.0 GiB     FFFF  system_a
      26         7477248        13768703   3.0 GiB     FFFF  system_b
    ...
      49        14550032        61071326   22.2 GiB    FFFF  userdata
    The important values are the partition numbers (25, 26 and 49), and the next two numbers for both system partitions (start/end sectors). The start/end sector of userdata isn't important, only the partition number of 49 is. If your sector start/ends are different, you will need to adjust some of the values in these steps - that's up to you to figure out. But do please let me know what your differences are, for science.
  3. Remove system_a, and create a new system_a partition of size ~2.4GB. Don't worry, this is plenty - my current RR build with a heavy AROMA Gapps is only ~1.9GB. We're taking off ~600MB, which is 1228800 sectors (600*1024*1024/512) so the new end sector will be 6248447 (7477247 - 1228800):
    Code:
    sgdisk /dev/block/mmcblk0 --delete 25
    sgdisk /dev/block/mmcblk0 --new=25:1185792:6248447
    sgdisk /dev/block/mmcblk0 --change-name=25:system_a
  4. Create vendor_a with the reclaimed ~600MB:
    Code:
    sgdisk /dev/block/mmcblk0 --new=50:6248448:7477247
    sgdisk /dev/block/mmcblk0 --change-name=50:vendor_a
    ...notice the partition number of 50 - one after the userdata (49).
  5. Do similarly to above, only for system_b and vendor_b:
    Code:
    sgdisk /dev/block/mmcblk0 --delete 26
    sgdisk /dev/block/mmcblk0 --new=26:7477248:12539903
    sgdisk /dev/block/mmcblk0 --change-name=26:system_b
    sgdisk /dev/block/mmcblk0 --new=51:12539904:13768703
    sgdisk /dev/block/mmcblk0 --change-name=51:vendor_b
  6. Quit shell and reboot recovery...
    Code:
    adb reboot recovery
    ...and verify the new partitions exist...
    Code:
    adb shell ls -la /dev/block/bootdevice/by-name/
    ...should show this at the bottom...
    Code:
    ...
    lrwxrwxrwx    1 root     root            21 Jan  5  1970 userdata -> /dev/block/mmcblk0p49
    lrwxrwxrwx    1 root     root            21 Jan  5  1970 vendor_a -> /dev/block/mmcblk0p50
    lrwxrwxrwx    1 root     root            21 Jan  5  1970 vendor_b -> /dev/block/mmcblk0p51


Repartition is done! You can now restore your TWRP backup (only system) and system works perfectly as before.


Roadblocks of Phase 1
  • How best to deliver this partition table change to users? The obvious answer is an update ZIP that just scripts it all (that's why I used sgdisk in this write-up). But then this requires the user to manually backup, reboot recovery, then restore backup. Is that ideal?
  • Another solution would be to export the new GPT, and make a MiFlash-flashable package. This could be another option for future, but would require new system and vendor images too - may as well be a whole flash package. That's big. We'll look into that once we actually get a Treble ROM working (e.g. stock).


Phase 1b revert - Restore stock partition map

If you want to go back to original partition table for some reason, you will need to flash an official fastboot ROM via EDL mode in MiFlash. After researching, I have discovered that there is no way around this. The way MiFlash works is it actually adjusts the GPT info dynamically based on *your* partition map (64GB vs 32GB have different userdata sizes) and for this to be done it needs to flash a full package. So, to restore stock partition map, you must flash a stock ROM in MiFlash via EDL mode which wipes everything. I hope you made a low-level and internal-storage backup like I warned - because after that you'll want to restore your backup to get persist and stuff back. Flashing in fastboot mode is not enough because it does not restore stock partition table.

Full steps for restoring to stock:
  1. Boot into EDL mode
  2. Load MiFlash, select any Oreo fastboot image (I use 8.1.10 since it's what I have) > Refresh. Should show COM30 (or whatever your QDLoader COM port is).
  3. Reboot EDL by holding Power + VolDown and then 'fastboot oem edl' again
  4. Run the restore in my Low-level flash tool of your backup
  5. Reboot back into recovery by holding power + volume-up for ~10 seconds.
  6. "Skip" on password screen
  7. Wipe > Format Data > type "yes" to remove stock ROM encryption
  8. Restore your TWRP backup
  9. Done!


Phase 2 - Patch kernel DTB and TWRP

This requires Linux. I use ChaletOS in VirtualBox and have my home folder Samba shared, mapped to a network drive on my Windows host.

Probably when all this is finalized and working, we will use a custom kernel built from source. But for now lets just patch our existing kernel. I assume you already have TWRP installed.

Part 1 - Kernel DTB
  1. Download the DTB split tool from:
    https://github.com/dianlujitao/split-appended-dtb
    ...to your Linux machine. Just get the binary, not the .c source file.
  2. Dump existing boot.img while in TWRP and pull from device:
    Code:
    adb shell dd if=/dev/block/bootdevice/by-name/boot of=/tmp/boot.img
    adb pull /tmp/boot.img
  3. Unpack this boot.img. I use AIK by @osm0sis from:
    https://forum.xda-developers.com/sho....php?t=2073775
    ... In this case, we need boot.img-zImage. Copy it to the Linux machine folder with DTB split.
  4. Split DTB from zImage on Linux machine:
    Code:
    ./split-appended-dtb boot.img-zImage
    ...should produce 1 DTB. Total output will be two files - "kernel" and "dtbdump_1.dtb".
  5. Decompile DTB on Linux machine:
    Code:
    # if you need to install it
    sudo apt install device-tree-compiler
    dtc -O dts -I dtb -o dtbdump_1.dts dtbdump_1.dtb
  6. Edit dtbdump_1.dts in whatever text editor. Search for fstab and edit the vendor entry, adding "slotselect" to fsmgr_flags (so it's the same as system):
    Code:
    fsmgr_flags = "wait,slotselect";
    NOTE: This is on my RR kernel. I need to check any difference with stock or other kernels.
  7. Recompile the dts to dtb and build a new zImage:
    Code:
    dtc -O dtb -I dts -o dtbdump_1_new.dtb dtbdump_1.dts
    cat kernel dtbdump_1_new.dtb > boot.img-zImage.new
  8. Copy the boot.img-zImage.new back to your kernel kitchen tool, replacing the old one.
That's the device tree patched to mount seamless/slot vendor. But don't repack the boot.img yet, we need to edit RAMDisk.



Part 2 - TWRP patch

This is just so we can see Vendor (and Persist, because why not) in TWRP. This is so:
  • Vendor can be formatted (required before use), backed up/restored and flashed to
  • [Bonus] Persist can be backed up/restored and flashed to (for e.g. Persist repair ZIP's that should come soon).

Only one step is needed - edit /etc/recovery.fstab to add vendor mount (and also persist because why not) with flags:
Code:
/vendor          ext4             /dev/block/bootdevice/by-name/vendor     flags=slotselect;backup=1;wipeingui;display="Vendor"
/persist         ext4             /dev/block/bootdevice/by-name/persist    flags=backup=1;display="Persist"
That's it.

For your convenience, I have attached a Treble-modded TWRP installer to this thread with these changes. Don't forget to flash Magisk after re-installing TWRP. Big thanks to @ghpranav


Phase 3 - Compile a Treble ROM (for Vendor files)

Already succeeded. We just need to cleanup and push.

Phase 4 - Treble-ize an installed ROM
  • Is this possible? Not sure... I'll check it out later sometime


Did I miss anything? Probably. Let me know!
The Following 105 Users Say Thank You to CosmicDan For This Useful Post: [ View ] Gift CosmicDan Ad-Free
 
 
15th May 2018, 01:12 AM |#2  
CosmicDan's Avatar
OP Senior Member
Flag Sydney
Thanks Meter: 7,584
 
Donate to Me
More
I just realized we could use resize2fs to first shrink the system filesystem, and that would save a reboot step. Obviously a TWRP should be done first anyway, though.

Oh, but a reboot would still be necessary for TWRP to see the vendor partition so it can be written to. So... That seems pointless I think.

Hmm...
The Following 25 Users Say Thank You to CosmicDan For This Useful Post: [ View ] Gift CosmicDan Ad-Free
15th May 2018, 01:25 AM |#3  
joeyhuab's Avatar
Recognized Developer
Flag Antipolo
Thanks Meter: 4,514
 
Donate to Me
More
Just finished reading everything. You did a wonderful job explaining this and I'm eager to test this out even if my device is my daily driver. With that being said, I really hope inexperienced users would stay away from this thread and refrain asking silly questions. I especially want them to stay away from complaining that they bricked their device, etc etc. By the looks of this, it should be easy considering you got the partition sectors noted down and most if not all users have the 64GB variant. It should be fairly easy making the ROM boot as long as it caters to A/B partitioning (already available in XDA) and that the Vendor HALs wouldn't cause further trouble.

With all of that, I'm looking forward to testing Android P as soon as we got treble tested up and running.

Thanks for taking your time to explain this thoroughly and I appreciate all the efforts you've put into this, especially automating the process by making a backup tool and hopefully, a working TWRP for Treble.

Looking at the Treble threads, I see some devs have started working on Android P. Link here.

Sent from my Xiaomi Mi A1 using XDA Labs
Attached Thumbnails
Click image for larger version

Name:	Screenshot_XDA Labs_20180515-082802.png
Views:	4012
Size:	185.3 KB
ID:	4501393  
The Following 3 Users Say Thank You to joeyhuab For This Useful Post: [ View ]
15th May 2018, 01:58 AM |#4  
CosmicDan's Avatar
OP Senior Member
Flag Sydney
Thanks Meter: 7,584
 
Donate to Me
More
Quote:
Originally Posted by joeyhuab

Just finished reading everything. You did a wonderful job explaining this and I'm eager to test this out even if my device is my daily driver. With that being said, I really hope inexperienced users would stay away from this thread and refrain asking silly questions. I especially want them to stay away from complaining that they bricked their device, etc etc. By the looks of this, it should be easy considering you got the partition sectors noted down and most if not all users have the 64GB variant. It should be fairly easy making the ROM boot as long as it caters to A/B partitioning (already available in XDA) and that the Vendor HALs wouldn't cause further trouble.

With all of that, I'm looking forward to testing Android P as soon as we got treble tested up and running.

Thanks for taking your time to explain this thoroughly and I appreciate all the efforts you've put into this, especially automating the process by making a backup tool and hopefully, a working TWRP for Treble.

Looking at the Treble threads, I see some devs have started working on Android P. Link here.

Sent from my Xiaomi Mi A1 using XDA Labs

Thanks mate! Yep I've seen them but based on Phh's work, at least Oreo Treble GSI depends on VTS which isn't released yet. So I'm not sure how long until we get it.

I just learned that in order to get stock GPT back you basically have to fully erase the device via flash_factory in MiFlash. Technical details added to the OP.

So, apart from low-level flash, you will also want to backup internal storage before reverting - since factory flash wipes absolutely everything and TWRP won't backup storage. Then of course you can use my low-level tool to restore partitions via EDL to get your (e.g.) persist and stuff back, then restore the associated TWRP backup.
The Following 4 Users Say Thank You to CosmicDan For This Useful Post: [ View ] Gift CosmicDan Ad-Free
15th May 2018, 04:43 AM |#5  
Senior Member
Flag Behind You
Thanks Meter: 117
 
Donate to Me
More
Just a reminder ,

EDL mode uses rawprogram0 , prog_emmc_firehose_8953_ddr.mbn , patch0 or partition isn't it ? Flash_factory will erase all the partition and causes device lost of IMEI , ESN, PSN and others sensors settings . and only flash-able during Fastboot mode .

fastboot %* getvar product 2>&1 | findstr /r /c:"^product: *tissot" || @ECHO "error : Missmatching image and device" & exit /B 1
fastboot %* oem unlock || @ECHO "unlock error" & exit /B 1
fastboot %* flash partition %~dp0images\gpt_both0.bin || @ECHO "Flash gpt_both0 error" & exit /B 1
fastboot %* flash modem_a %~dp0images\NON-HLOS.bin || @ECHO "Flash modem_a error" & exit /B 1
fastboot %* flash modem_b %~dp0images\NON-HLOS.bin || @ECHO "Flash modem_b error" & exit /B 1
fastboot %* flash sbl1 %~dp0images\sbl1.mbn || @ECHO "Flash sbl1 error" & exit /B 1
fastboot %* flash sbl1bak %~dp0images\sbl1.mbn || @ECHO "Flash sbl1bak error" & exit /B 1
fastboot %* flash rpm %~dp0images\rpm.mbn || @ECHO "Flash rpm error" & exit /B 1
fastboot %* flash rpmbak %~dp0images\rpm.mbn || @ECHO "Flash rpmbak error" & exit /B 1
fastboot %* flash tz %~dp0images\tz.mbn || @ECHO "Flash tz error" & exit /B 1
fastboot %* flash tzbak %~dp0images\tz.mbn || @ECHO "Flash tzbak error" & exit /B 1
fastboot %* flash devcfg %~dp0images\devcfg.mbn || @ECHO "Flash devcfg error" & exit /B 1
fastboot %* flash devcfgbak %~dp0images\devcfg.mbn || @ECHO "Flash devcfgbak error" & exit /B 1
fastboot %* flash dsp %~dp0images\adspso.bin || @ECHO "Flash dsp error" & exit /B 1
fastboot %* flash fsg %~dp0images\fs_image.tar.gz.mbn.img || @ECHO "Flash fs_image error" & exit /B 1
fastboot %* flash sec %~dp0images\sec.dat || @ECHO "Flash sec error" & exit /B 1
fastboot %* flash splash %~dp0images\splash.img || @ECHO "Flash splash error" & exit /B 1
fastboot %* flash aboot %~dp0images\emmc_appsboot.mbn || @ECHO "Flash aboot error" & exit /B 1
fastboot %* flash abootbak %~dp0images\emmc_appsboot.mbn || @ECHO "Flash abootbak error" & exit /B 1
fastboot %* flash boot_a %~dp0images\boot.img || @ECHO "Flash boot_a error" & exit /B 1
fastboot %* flash boot_b %~dp0images\boot.img || @ECHO "Flash boot_b error" & exit /B 1
fastboot %* flash system_a %~dp0images\system.img || @ECHO "Flash system_a error" & exit /B 1
fastboot %* flash system_b %~dp0images\system.img || @ECHO "Flash system_a error" & exit /B 1
fastboot %* flash lksecapp %~dp0images\lksecapp.mbn || @ECHO "Flash lksecapp error" & exit /B 1
fastboot %* flash lksecappbak %~dp0images\lksecapp.mbn || @ECHO "Flash lksecappbak error" & exit /B 1
fastboot %* flash cmnlib %~dp0images\cmnlib.mbn || @ECHO "Flash cmnlib error" & exit /B 1
fastboot %* flash cmnlibbak %~dp0images\cmnlib.mbn || @ECHO "Flash cmnlibbak error" & exit /B 1
fastboot %* flash cmnlib64 %~dp0images\cmnlib64.mbn || @ECHO "Flash cmnlib64 error" & exit /B 1
fastboot %* flash cmnlib64bak %~dp0images\cmnlib64.mbn || @ECHO "Flash cmnlib64bak error" & exit /B 1
fastboot %* flash keymaster %~dp0images\keymaster.mbn || @ECHO "Flash keymaster error" & exit /B 1
fastboot %* flash keymasterbak %~dp0images\keymaster.mbn || @ECHO "Flash keymasterbak error" & exit /B 1
fastboot %* flash userdata %~dp0images\userdata.img || @ECHO "Flash userdata error" & exit /B 1
fastboot %* flash persist %~dp0images\persist.img || @ECHO "Flash persist error" & exit /B 1
fastboot %* erase modemst1
fastboot %* erase modemst2
fastboot %* erase config || @ECHO "Erase config error" & exit /B 1
fastboot %* reboot bootloader
ping 127.0.0.1 -n 5 -w 1000 > nul
fastboot %* set_active a
fastboot %* reboot

also , those code are NOT compatible for Mi A1 as modem uses modem.img not NON-HLOS.bin, sbl.img not sbl.mbn etc . for more information you may compare with flash_all with flash_factory .
15th May 2018, 05:15 AM |#6  
murtaza1326's Avatar
Senior Member
Flag godhra
Thanks Meter: 638
 
Donate to Me
More
well just curious i read whole thread but i didn't get that is
repartitioning affect A/B partition system or
we are lucky and have A/B and Treble both at same time?

also except developer if someone tries and flash this successfully then do post SS 😃👍🏻

p.s don't tell me to try it neither i am developer or researcher 😅
Regards
15th May 2018, 05:21 AM |#7  
CosmicDan's Avatar
OP Senior Member
Flag Sydney
Thanks Meter: 7,584
 
Donate to Me
More
Quote:
Originally Posted by JoKeRLeE

*snip*

Yes you are right, flash_factory is only for fastboot mode. It doesn't matter what is chose when in EDL mode as it will flash everything.

But I think EDL mode is still needed, since MiFlash recalculates the partition table. Maybe it does in fastboot mode too but I don't know, it just flashes to "partition" via fastboot.

YMMV. I do it in EDL and it works fine.

Don't know or care if any of the firmware files are incompatible (are you sure? That seems strange if so, why would they be in the A1 package if they're not) since we restore a backup from my tool anyway which does all partitions exactly how they were on your device. The only purpose of MiFlash in this case is to rewrite partition table.


Quote:
Originally Posted by murtaza1326

well just curious i read whole thread but i didn't get that is
repartitioning affect A/B partition system or
we are lucky and have A/B and Treble both at same time?

also except developer if someone tries and flash this successfully then do post SS 😃👍🏻 regards

p.s don't tell me to try it neither i am developer or researcher 😅
Regards

Yes, as you can see it's "vendor_a" and "vendor_b", one for each slot.

There are no screenshots lol, unless you want to see a screenshot of TWRP with a Vendor mount listing ? 😅
The Following 4 Users Say Thank You to CosmicDan For This Useful Post: [ View ] Gift CosmicDan Ad-Free
15th May 2018, 05:26 AM |#8  
murtaza1326's Avatar
Senior Member
Flag godhra
Thanks Meter: 638
 
Donate to Me
More
Quote:
Originally Posted by CosmicDan

Yes, as you can see it's "vendor_a" and "vendor_b", one for each slot.

There are no screenshots lol, unless you want to see a screenshot of TWRP with a Vendor mount listing ? 😅

well yeah i missed that

about screen shot that treble app on playstore or any other thing which state both
also i wonder if GSI images are flashable or moded treble roms should be there 😅
15th May 2018, 06:14 AM |#9  
khusika's Avatar
Recognized Contributor
Flag Malang
Thanks Meter: 4,018
 
Donate to Me
More
Interesting.

As i know, we also need to work with .dts to make it possible with treble inside kernel source with A/B partition layout.
15th May 2018, 06:21 AM |#10  
CosmicDan's Avatar
OP Senior Member
Flag Sydney
Thanks Meter: 7,584
 
Donate to Me
More
Quote:
Originally Posted by murtaza1326

well yeah i missed that

about screen shot that treble app on playstore or any other thing which state both
also i wonder if GSI images are flashable or moded treble roms should be there 😅

Not there yet mate, patience 😋

Quote:
Originally Posted by khusika

Interesting.

As i know, we also need to work with .dts to make it possible with treble inside kernel source with A/B partition layout.

Really? This might be what I'm stuck on. No idea what it is though.

I just get recovery loop if I try to Treble-ize a ROM, and GSI's kick me to fastboot. I was suspecting that it could be kernel, but I am seeking help from Phh about it.

Do you have any links that might point me in the right direction? Meanwhile I'll see what I find in Google.
The Following User Says Thank You to CosmicDan For This Useful Post: [ View ] Gift CosmicDan Ad-Free
15th May 2018, 06:21 AM |#11  
TrueMS's Avatar
Senior Member
Thanks Meter: 113
 
More
Great job. Just a question, if you restore by doing flash_factory.bat - it deletes your IMEI, or not?
Thread Closed Subscribe to Thread
Previous Thread Next Thread
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes