Create TWRP Flashable from existing TWRP Flashable
Create TWRP Flashable zip from Existing TWRP Flashable
1) Get an existing TWRP flashable
You can download the latest TWRP Nougat flashables for Potter from here:
[Nougat][Stock][Rom] TWRP Flashable Stock Builds
The latest TWRP Oreo flashables for Potter are here:
[Oreo][Stock][Rom] TWRP Flashable Stock Builds
I also put together some TWRP flashables for other devices, such as Cedric (G5) and Sanders (G5S Plus). Have a search under my profile at Android File Host:
Downloads for Android Devices by nzedpred
Extract it to a folder, e.g. "next", that represents the next version that you will be creating. Enter the folder.
All commands are run from within this folder. If a folder is created as part of the instructions, I assume that the folder is not entered.
2) Convert sparse data images to img
With the sdat2img tool, convert the system, oem, and modem images to raw images:
Code:
sdat2img.py system.transfer.list system.new.dat system.img
sdat2img.py oem.transfer.list oem.new.dat oem.img
sdat2img.py modem.transfer.list modem.new.dat modem.img
3) Analyse the OTA and apply patching and updates
Unzip the ota upgrade files into a sub-folder ota.
Code:
unzip path/to/ota/Blur_Version.x.y.z.zip -d ota
In the OTA extract, open up the updater-script in a text editor
Code:
ota/META-INF/com/google/android/updater-script
The first parts of the script are to check for valid OEM, Recovery and System partitions from the previous version. The parts we're interested in start below the line:
Code:
# ---- start making changes here ----
New in Oreo
In Nougat, many of the files below are named e.g. system.new.dat. Oreo may have these files named e.g. system.new.dat.br. These "br" files are Brotli compressed files. Look for any files that are named as such, e.g.
Code:
ls ota/*.br
ota/dsp.new.dat.br ota/system.new.dat.br
In the example above, I get two files that are Brotli compressed. Decompress these using the following:
Code:
brotli --decompress --input ota/dsp.new.dat.br --output ota/dsp.new.dat
brotli --decompress --input ota/system.new.dat.br --output ota/system.new.dat
** System **
The first line that makes updates to the system image is this one for Nougat:
Code:
block_image_update("/dev/block/bootdevice/by-name/system", package_extract_file("system.transfer.list"), "system.new.dat", "system.patch.dat")
or Oreo:
Code:
block_image_update("/dev/block/bootdevice/by-name/system", package_extract_file("system.transfer.list"), "system.new.dat.br", "system.patch.dat")
The only difference between Nougat and Oreo is the "br" in system.new.dat.br. As we have decompressed the Brotli files in the previous steps, we only need to use sytem.new.dat.
The equivalent command to imitate this command is the following:
Code:
BlockImageUpdate system.img ota/system.transfer.list ota/system.new.dat ota/system.patch.dat
Check that the error code returned is 0.
** Boot **
The next line that makes updates to the boot image is this one:
Code:
apply_patch("EMMC:/dev/block/bootdevice/by-name/boot:16777216:f6ee50c0900378319080912820b5c20f4bb7051c:16777216:19b3ba799fd8f57588ff3736f1a1c0070417f4c2",
"-", 19b3ba799fd8f57588ff3736f1a1c0070417f4c2, 16777216,
f6ee50c0900378319080912820b5c20f4bb7051c,
package_extract_file("patch/boot.img.p"))
The equivalent command to imitate this is the following:
Code:
ApplyPatch boot.img - 19b3ba799fd8f57588ff3736f1a1c0070417f4c2 16777216 f6ee50c0900378319080912820b5c20f4bb7051c ota/patch/boot.img.p
NOTE: You will need to update the strings above to match what is in the OTA update script, which will differ from OTA to OTA.
NOTE: This will fail if we use the version straight from the previous flashable zip, as that version may have already been patched to disable dm-verity and disable forced encryption. We need to get a version from the stock image.
Also note, that as part of creating these zips, I leave the original unmodified boot.img file as a copy, boot-stock.img or similar. When patching, simply rename it to boot.img before doing the ApplyPatch command
Code:
mv boot-stock.img boot.img -fv
Check that the error code returned is 0.
** Bootloader **
The bootloader also gets updated quite often. Note that as part of making these zips safer, we don't update the bootloader, so skip right past them. These lines look something like this:
Code:
ui_print("updating sbl1 ...");
assert(package_extract_file("sbl1.mbn", "/tmp/sbl1.mbn"),
apply_raw_image("/tmp/sbl1.mbn", "sbl1"),
delete("/tmp/sbl1.mbn"));
Refer to post
#13 to see details of how to work out which partitions are bootloader-related. In brief, the following are bootloader-related:
- aboot
- rpm
- tz
- devcfg
- cmnlib
- cmnlib64
- keymaster
- prov
- sbl1
** MODEM **
When the modem is updated, it tends to be done using a few different techniques. Sometimes it uses one of the approaches above, other times it is untouched, and finally it can have a combination of deleting, patching and copying new files.
To apply changes that are done by deleting, patching and copying, first set up a mount point, mount the image, and make sure you have any of the tools in the path of the root user, as these need to be run as root.
Code:
mkdir modem
sudo su
mount modem.img modem
export PATH=$PATH:/path/to/tools
An example of deleting:
Code:
ui_print("Removing unneeded files from modem...");
delete("/modem/image/Ver_Info.txt", "/modem/image/cmnlib.b04",
"/modem/image/fpctzappfingerprint.b04",
"/modem/image/fpctzappfingerprint.b05", "/modem/image/modem.b17",
"/modem/image/qdsp6m.qdb", "/modem/image/dhsecapp.b00",
etc...
The equivalent of these are:
Code:
rm modem/image/Ver_Info.txt
rm modem/image/cmnlib.b04
rm modem/image/fpctzappfingerprint.b04
etc...
Take very special care to remove the leading slash before modem. You don't want to risk deleting files on your PC's filesystem.
An example of patching:
Code:
ui_print("Patching modem files...");
apply_patch("/modem/image/adsp.b00", "-",
42ae9e4a8a04b70938c6fda6bef2ad7063ccba15, 532,
6df377596db8273c268691fb87380c416128502c,
package_extract_file("patch/modem/image/adsp.b00.p")) ||
abort("E3008: Failed to apply patch to /modem/image/adsp.b00");
apply_patch("/modem/image/adsp.b01", "-",
13500067ce0564e2d45780d5511271f8d195a598, 6920,
3ee7f84d3e81a44725554ac24d001cff21a636ab,
package_extract_file("patch/modem/image/adsp.b01.p")) ||
abort("E3008: Failed to apply patch to /modem/image/adsp.b01");
etc...
The equivalent of these are:
Code:
ApplyPatch modem/image/adsp.b00 - 42ae9e4a8a04b70938c6fda6bef2ad7063ccba15 532 6df377596db8273c268691fb87380c416128502c ota/patch/modem/image/adsp.b00.p
ApplyPatch modem/image/adsp.b01 - 13500067ce0564e2d45780d5511271f8d195a598 6920 3ee7f84d3e81a44725554ac24d001cff21a636ab ota/patch/modem/image/adsp.b01.p
etc...
An example of copying:
Code:
ui_print("Unpacking new files in modem ...");
assert(package_extract_dir("modem", "/modem"));
The equivalent of this is:
Code:
cp -rv ota/modem/* modem/
You will probably also see something like:
Code:
ui_print("Symlinks and permissions in modem ...");
set_metadata_recursive("/modem/", "uid", 0, "gid", 0, "dmode", 0755, "fmode", 0644, "capabilities", 0x0);
That sets permissions of folders and files to 0755 and 0644 respectively. This can be achieved by:
Code:
find modem/ -type d -exec chmod 0755 {} \;
find modem/ -type f -exec chmod 0644 {} \;
When finished, unmount modem.img, exit the root shell, and remove the mount folder:
Code:
umount modem
exit
rmdir modem
** OEM **
The line that updates the oem image is this one:
Code:
block_image_update("/dev/block/bootdevice/by-name/oem", package_extract_file("oem.transfer.list"), "oem.new.dat", "oem.patch.dat")
The equivalent command to imitate this is the following:
Code:
BlockImageUpdate oem.img ota/oem.transfer.list ota/oem.new.dat ota/oem.patch.dat
Check that the error code returned is 0.
** DSP **
Occasionally the dsp partition is also updated. It will use one of the techniques above, e.g.
Code:
block_image_update("/dev/block/bootdevice/by-name/dsp", package_extract_file("dsp.transfer.list"), "dsp.new.dat.br", "dsp.patch.dat")
Has the equivalent command:
Code:
BlockImageUpdate adspso.bin ota/dsp.transfer.list ota/dsp.new.dat ota/dsp.patch.dat
Note that the equivalent of the dsp partition is the adspso.bin file.
** FSG **
If FSG is updated, you will often see something like this:
Code:
assert(package_extract_file("fsg.mbn", "/tmp/fsg.mbn"),
apply_raw_image("/tmp/fsg.mbn", "fsg"),
delete("/tmp/fsg.mbn"));
This doesn't represent a patch being applied. Instead the file would just be copied from the OTA folder into the base folder. This applies to any other file that hasn't been patched in one of the ways above (except the bootloader).
** Others **
At this point, check if there are other (non-bootloader) partitions in the updater-script. This should have covered off all, but a future device may have other partitions that follow this (or a new) technique.
One that is present that I haven't used here, is logo.bin being copied over the logo partition. I tend to leave this as-is, as I prefer/recommend that people flash the logo as a one-off, and preferably grab a "Hidden N/A" from the themes forum.
4) Disable dm-verity and forced encryption
There are two different techniques to disabling dm-verity and forced encryption, one for Nougat and one for Oreo. Refer to the appropriate section below.
4a) Nougat - Disable dm-verity and forced encryption in boot
Nougat has the flags for enabling/disabling dm-verity and forced encryption in the boot image. These instructions are based upon looking at the scripts in the files here:
https://build.nethunter.com/android-tools/no-verity-opt-encrypt/
Make a copy of the boot.img file, so that we have the original for the next ota. This is important, as without the original, the next time we try to apply the patch, it will fail.
Code:
cp boot.img boot-stock.img
Extract the boot image:
Modify the fstab file in a text editor:
Code:
bootimg/ramdisk/fstab.qcom
Remove any instances of verify. For example, the below line:
Code:
/dev/block/bootdevice/by-name/system /system ext4 ro,barrier=1,discard wait[U],verify[/U]
Can be changed to:
Code:
/dev/block/bootdevice/by-name/system /system ext4 ro,barrier=1,discard wait
If the section "wait,verify" was in fact just "verify", we would need to replace it with "defaults"
Replace any instances of forceencrypt or forcefdeorfbe with encryptable
E.g. the below line:
Code:
/dev/block/bootdevice/by-name/userdata /data f2fs rw,discard,nosuid,nodev,noatime,nodiratime,nobarrier,inline_xattr,inline_data wait,check,formattable,[U]forceencrypt[/U]=/dev/block/bootdevice/by-name/metadata
Can be changed to:
Code:
/dev/block/bootdevice/by-name/userdata /data f2fs rw,discard,nosuid,nodev,noatime,nodiratime,nobarrier,inline_xattr,inline_data wait,check,formattable,[U]encryptable[/U]=/dev/block/bootdevice/by-name/metadata
Save and close the file.
Repack the boot image:
Note the above line will replace your previous boot.img file, but that's OK because we made the copy to boot-stock.img.
4b) Oreo - Disable dm-verity and forced encryption
Oreo is different to Nougat with respect to these flags. Oreo uses a "device tree" in the boot image to mount the vendor partition. In the case of potter, and presumably the other recent Moto devices, vendor is actually a symlink to a folder in system. So, we need to make a change to the boot image, and the system partition.
4b) i) Oreo - Disable dm-verity
The process in broad terms is this:
- Backup the boot image
- Unpack the boot image
- Decompress the dt.img file (if applicable), to get the underlying dtb file (device tree blob)
- Hexedit the device tree blob, removing instances of verify
- Compress the dtb file
- Pack the boot image
To assist with the next OTA, make a copy of the boot image. Then use mkboot to extract the boot image to its component parts:
Code:
cp -v boot.img boot-stock.img
mkboot boot.img bootimg
Look at the extracted boot image:
Code:
ls -l bootimg/
-rw-r--r-- 1 user user 418560 Sep 9 09:12 dt.img
-rw-r--r-- 1 user user 466 Sep 9 09:12 img_info
-rw-r--r-- 1 user user 9211208 Sep 9 09:12 kernel
drwxr-xr-x 21 user user 4096 Sep 9 09:12 ramdisk
-rw-r--r-- 1 user user 1273390 Sep 9 09:12 ramdisk.packed
The dt.img is the device tree, which holds the mount information that we need to edit. Note that dt.img may be compressed - for Potter it is lz4 compressed, for Cedric for example, it is not compressed - it is raw data. To find out whether or not it is compressed, use the following command and check the output
The output could be one of the following:
Code:
bootimg/dt.img: LZ4 compressed data (v1.4+)
bootimg/dt.img: data
If the output says that it is LZ4 compressed, use the following to decompress - we will call the decompressed file dtb.img:
Code:
lz4 -d bootimg/dt.img bootimg/dtb.img
Now we need to use a hex editor to view and manipulate the dtb.img (or dt.img if not compressed) file.
Note that the dtc command (device tree compiler) should be able to convert these dtb files into source files. However I have had no success even with the latest versions. In future it may be possible (and preferable) to convert to source, and edit the source text and recompile to dtb.
or
Now, search for instances of ",verify" (without the quotes). When they are found, you can also scroll up to see what the previous lines were, to make sure they are editing instances of "fstab", for example:
Code:
32E44 00 00 00 01 61 6e 64 72 6f 69 64 00 00 00 00 03 00 00 00 11 00 00 00 21 ....android............!
32E5C 61 6e 64 72 6f 69 64 2c 66 69 72 6d 77 61 72 65 00 00 00 00 00 00 00 01 android,firmware........
[COLOR="red"] 32E74 66 73 74 61 62 00 00 00 00 00 00 03 00 00 00 0e 00 00 00 21 61 6e 64 72 fstab..............!andr[/COLOR]
32E8C 6f 69 64 2c 66 73 74 61 62 00 00 00 00 00 00 01 73 79 73 74 65 6d 00 00 oid,fstab.......system..
32EA4 00 00 00 03 00 00 00 0f 00 00 00 21 61 6e 64 72 6f 69 64 2c 73 79 73 74 ...........!android,syst
32EBC 65 6d 00 00 00 00 00 03 00 00 00 35 00 00 03 72 2f 64 65 76 2f 62 6c 6f em.........5...r/dev/blo
32ED4 63 6b 2f 70 6c 61 74 66 6f 72 6d 2f 73 6f 63 2f 37 38 32 34 39 30 30 2e ck/platform/soc/7824900.
32EEC 73 64 68 63 69 2f 62 79 2d 6e 61 6d 65 2f 73 79 73 74 65 6d 00 00 00 00 sdhci/by-name/system....
32F04 00 00 00 03 00 00 00 05 00 00 00 71 65 78 74 34 00 00 00 00 00 00 00 03 ...........qext4........
32F1C 00 00 00 15 00 00 69 bc 72 6f 2c 62 61 72 72 69 65 72 3d 31 2c 64 69 73 ......i.ro,barrier=1,dis
32F34 63 61 72 64 00 00 00 00 00 00 00 03 00 00 00 0c 00 00 69 c6 77 61 69 74 card..............i.wait
[COLOR="Red"] 32F4C 2c 76 65 72 69 66 79 00 00 00 00 03 00 00 00 03 00 00 02 b4 6f 6b 00 00 ,verify.............ok..[/COLOR]
There will be several instances of this. In all cases, we want to overwrite the ",verify" with zeroes. So the last line above would become:
Code:
32F4C 00 00 00 00 00 00 00 00 00 00 00 03 00 00 00 03 00 00 02 b4 6f 6b 00 00 ....................ok..
Make sure to repeat for all, then save and exit the hex editor.
Now complete the last few steps - the first set of commands if the dt.img file was compressed:
Code:
rm -v bootimg/dt.img
lz4 -9 bootimg/dtb.img bootimg/dt.img
rm -v bootimg/dtb.img
mkboot bootimg boot.img
Or the following if dt.img was not compressed:
The boot image has now been recreated with dm-verity disabled.
4b) ii) Oreo - Disable forced encryption
The process here is similar to what is done in Nougat, just that we need to mount system.img first, and then apply to a file at system/vendor/etc/fstab.qcom.
It is important to know that merely mounting system.img will result in changes to the file. This could cause a future OTA to not work. The approach below will work, however it is now recommended that system.img is mounted read-only, the fstab.qcom file is copied and edited, and a script is used to replace the fstab.qcom file at install time. This way system.img is kept intact, there is no need to keep a > 1 GB file as a backup, and there will be no issues with future OTAs. This is the approach taken by the latest flashable zips with the Aroma installer.
In the below, I use vim to edit, but you can use any other text editor. I also create a backup of system.img, just in case I make a mistake. Note that it is a large file so you wouldn't want to keep it in the final zip.
Another thing to note is that this change appears to have no effect if dm-verity is not disabled.
Code:
cp -v system.img /path/to/keep/system.img.backup
mkdir system
sudo su
mount system.img system
vim system/vendor/etc/fstab.qcom
Here, apply the same approach as Nougat - replace "forceencrypt" with "encryptable". Close and save, then unmount system.img, and exit the root shell. Then remove the system folder.
Code:
umount system
exit
rmdir -v system
Installing Magisk immediately after flashing the rom, and before rebooting, will also disable dm-verity and forced encryption.
5) Convert images to sparse images, then to sparse data
Make a temporary folder for the spasre image files that we will create:
Code:
mkdir tmp
img2simg system.img tmp/system.img
img2simg oem.img tmp/oem.img
img2simg modem.img tmp/modem.img
Create the sparse data files from the sparse images:
Code:
img2sdat.py tmp/system.img -v 4 -p system
img2sdat.py tmp/oem.img -v 4 -p oem
img2sdat.py tmp/modem.img -v 4 -p modem
6) Update the actual updater-script
Use your text editor to update the updater-script file:
Code:
vim META-INF/com/google/android/updater-script
Update any lines that refer to the version being installed. It's good to have these lines in there, but are optional. They could be removed entirely if so desired...
Code:
ui_print("Target: motorola/potter/potter:7.0/NPNS25.137-93-8/10:user/release-keys");
As it is information only, it isn't necessary to get the values exactly right. You can get the correct values from the ota updater script here:
Code:
ota/META-INF/com/google/android/updater-script
Update - remove format of modemst1 and modemst2
The builds that I have provided to date erase the modemst1 and modemst2 partitions. Whilst that is fine for devices normally as they get recreated on boot, in cases where users have flashed a custom Oreo rom and then reverted to stock, there have been reports of losing IMEI and other capabilities. Whilst this is caused by Oreo roms changing ownership of the /persist/rfs folder, the fact that these partitions are erased can leave people without IMEI, relying upon a backup to get back. So, if the previous flashable zip has lines to format the modemst1/2 partitions, remove them. The lines look like the following, with the format commands being the ones that do the erase:
Code:
ui_print("Erasing modemst1 ...");
format("raw", "EMMC", "/dev/block/bootdevice/by-name/modemst1", "0", "/modemst1");
ui_print(" ");
ui_print("Erasing modemst2 ...");
format("raw", "EMMC", "/dev/block/bootdevice/by-name/modemst2", "0", "/modemst2");
ui_print(" ");
7) Remove working files and folders and zip
The folders we created in previous steps can now be removed:
The temporary img files can now be removed:
Code:
rm system.img oem.img modem.img
Zip up the remaining files into the parent folder (I like to keep the update tree clean) - replace version_info with an appropriate string, e.g. NPNS25.137-93-14, OPS28.85-13:
Code:
zip ../twrp-flashable-potter-[i]version_info[/i].zip -r *
Note - this will also include the boot-stock.img file. Best to keep it there so we can use it for the next patch!
8) Prepare for flashing
Copy the newly created zip into your phone's SD card
BACKUP BACKUP BACKUP!!! Do a TWRP backup of course! NO EXCUSES!
Flash the zip file.
Voila!
Edit log
- 2018-07-07 - Added reference to post #10, as it details how to do a modem update for the NPNS25.137-93-14 OTA
- 2018-07-10 - Added recommendation to remove lines from OTA updater-script that erase modemst1 and modemst2
- 2018-08-19 - Updated for Oreo.
- 2018-09-09 - Tidied up the steps to disable dm-verity for Oreo, other tidying up. Catered for dt.img being either compressed (e.g. Potter) or not (e.g. Cedric).
