[DEV][LINUX/OSX] IMG Patch Tools | sdat2img for OTA zips

erfanoabdi · Jul 21, 2017

most part of Introduction from taken sdat2img thread

if you want extract non OTA zip i suggest to use sdat2img tool
FOR AB OTA ZIPS USE THIS SCRIPT AND PAYLOAD EXTRACTOR

in this tools i made block_image_update() function from android recovery to a binary to run on PC and patch system.img with DAT files on OTA package to update system.img by OTA

also we have apply_patch() function for patching boot.img and other files like firmwares

Introduction
You probably know already that starting from Android 5.x (Lollipop) compiled roms (aosp,cm,stock) are not compressed anymore the way they used to be on previous android versions. On previous versions all content inside /system folder that has to be extracted within our device was either uncompressed (simple /system folder inside our flashable zip) or compressed in a system.img file, which it is a ext4 compressed file; both of these, anyway, were readable and we could see all system files (app,framework, etc).
The problem comes in >=5.0 versions, this method is not used anymore. Why? Because roms started to be always larger, so it is necessary to compress them even more.

What does new Android zips (full roms, but also OTAs) contain?
New Android flashable zips are made this way:

boot.img (kernel)
file_contexts (selinux related)
META-INF (folder containing scripts)
system.new.dat (compressed /system partition)
system.patch.dat (for OTAs)
system.transfer.list (see explanation below)
and other patch files (.p only on OTA zip)

What does updater-script contains then?
The updater-script uses a brand new function: block_image_update(), this method basically decompresses necessary files inside the device. Let's study it.
From google git source code, if we go inside the new file /bootable/recovery/updater/blockimg.c, we find at the end of it the registration of the function block_image_update() as the method BlockImageUpdateFn() which starts at line 254. Here finally we find all information we need to know about the decompression of the .dat file(s). First file we analyze is system.transfer.list which Google tells us:

The transfer list is a text file containing commands to transfer data from one place to another on the target partition.

But what each line means?:

First line is the version number of the transfer list; 1 for android 5.0.x, 2 for android 5.1.x, 3 for android 6.0.x, 4 for android 7.x
Second line is the total number of blocks being written
Third line is how many stash entries are needed simultaneously (only on versions >= 2)
Fourth line is the maximum number of blocks that will be stashed simultaneously (only on versions >= 2)
Fifth line and subsequent lines are all individual transfer commands.

all transfer commands is :

bsdiff
*erase
free
imgdiff
move
*new
stash
*zero

BlockImageUpdate is reading system.transfer.list and executing all commands
But BlockImageVerify doesn’t execute * commands, which not to make changes on system.img just verifying update can happen

original sdat2img tool only support "new" command

Ok, but how to Patch the system.img with OTA files?
All instructions are below. binaries are involved. Please read carefully step by step.

You can use/modify these files and/or include them in your work as long as proper credits and a link to this thread are given.

If you have questions or problems
write here

Thanks
- @xpirt , for original sdat2img tool and useful thread

XDA:DevDB Information
IMG Patch Tools, sdat2img for OTA zips, Tool/Utility for all devices (see above for details)

Contributors
erfanoabdi
Source Code: https://github.com/erfanoabdi/imgpatchtools

Version Information
Status: Testing

Created 2017-07-21
Last Updated 2020-01-23

erfanoabdi · Jul 21, 2017

Usage

Code:

./BlockImageUpdate <system.img> <system.transfer.list> <system.new.dat> <system.patch.dat>
args:

<system.img> = block device (or file) to modify in-place
<system.transfer.list> = transfer list (blob) from OTA/rom zip
<system.new.dat> = new data stream from OTA/rom zip
<system.patch.dat> = patch stream from OTA/rom zip

Code:

./ApplyPatch <file> <target> <tgt_sha1> <size> <init_sha1(1)> <patch(1)> [init_sha1(2)] [patch(2)]...
args:

<file> = source file from rom zip
<target> = target file (use "-" to patch source file)
<tgt_sha1> = target SHA1 Sum after patching
<size> = file size
<init_sha1> = file SHA1 sum
<patch> = patch file (.p) from OTA zip

Code:

usage: ./scriptpatcher.sh <updater-script>
args:

<updater-script> = updater-script from OTA zip to patch recovery commands

Example
for example from updater-script of OTA we have:

Code:

block_image_update("/dev/block/bootdevice/by-name/system", package_extract_file("system.transfer.list"), "system.new.dat", "system.patch.dat")

apply_patch("EMMC:/dev/block/bootdevice/by-name/boot:33554432:f32a854298814c18b12d56412f6e3a31afc95e42:33554432:0041a4df844d4b14c0085921d84572f48cc79ff4",
            "-", 0041a4df844d4b14c0085921d84572f48cc79ff4, 33554432,
            f32a854298814c18b12d56412f6e3a31afc95e42,
            package_extract_file("patch/boot.img.p"))

after getting system.img and boot.img from firmware This is equals of previous functions on PC with this tools:

Code:

~$ ./BlockImageUpdate system.img system.transfer.list system.new.dat system.patch.dat
~$ ./ApplyPatch boot.img - 0041a4df844d4b14c0085921d84572f48cc79ff4 33554432 f32a854298814c18b12d56412f6e3a31afc95e42

scriptpatcher.sh will generate all commands automatically from updater script so run it like:

Code:

~$ ./scriptpatcher.sh META-INF/com/google/android/updater-script > fullpatch.sh

check fullpatch.sh your self, you need to provide all images and files in correct name and patch as mentioned in mount and other commands of fullpatch.sh

Building Requirements
For Building this tool you need :

zlib
libbz2
openssl

It currently supports Linux x86/x64 & MacOS, Not tested on Windows.

Compile and Build command:

Code:

make

Youtube

erfanoabdi · Jul 21, 2017

Download links:
GitHub Release

Changelog:
GitHub Commits

Known Bugs/Issues:

Plz test and report

Harvey_Spectar · Jul 22, 2017

Nice work

rootlinux · Jul 22, 2017

Very nice

Sent from my LG-LS997 using Tapatalk

daniel_m · Jul 30, 2017

Wow, I tried to apply OTAs manually myself, but never got it to work. Getting the propper tools from you is awesome!

One tiny question though: Does applying/verifying a system update to system.img require a lot of temporary space or memory?

I am running Ubuntu x64 in a VM (VirtualBox) on Windows 10. Verifying a system update doesn't really do much, except keeping the CPU busy.

This is the output of "./BlockImageVerify system.img system.transfer.list system.new.dat system.patch.dat":

Code:

performing verification
creating cache dir cache
cache dir: cache
blockimg version is 3
maximum stash entries 236
creating stash cache/537ed49fb5f6c32dc3d205d78b6084fe54f70cd3/

The cache folder remains empty, there is no harddisk activity and only the CPU is working at 100 %. I interrupted BlockImageVerify after 30 minutes.

erfanoabdi · Jul 31, 2017

daniel_m said:
Wow, I tried to apply OTAs manually myself, but never got it to work. Getting the propper tools from you is awesome!

One tiny question though: Does applying/verifying a system update to system.img require a lot of temporary space or memory?

I am running Ubuntu x64 in a VM (VirtualBox) on Windows 10. Verifying a system update doesn't really do much, except keeping the CPU busy.

This is the output of "./BlockImageVerify system.img system.transfer.list system.new.dat system.patch.dat":

The cache folder remains empty, there is no harddisk activity and only the CPU is working at 100 %. I interrupted BlockImageVerify after 30 minutes.

Ah actually yes you need free space on RAM as much as both patch.dat and new.dat sizes
It's on my todo list to fix it, https://github.com/erfanoabdi/imgpatchtools/blob/master/blockimg/blockimg.cpp#L383

evangelico793 · Aug 21, 2017

How to extract the file system.new.dat

erfanoabdi · Aug 21, 2017

evangelico793 said:
How to extract the file system.new.dat

You can find it in OTA zip

Motorola? See this : motorola.erfanabdi.ir

evangelico793 · Aug 21, 2017

erfanoabdi said:
You can find it in OTA zip

Motorola? See this : motorola.erfanabdi.ir

No longer look at the picture

evangelico793 · Aug 21, 2017

erfanoabdi said:
You can find it in OTA zip

Motorola? See this : motorola.erfanabdi.ir

Something Looks Wrong ReCheck inputs Or No Update Available
for moto g4 play 6.0.1

erfanoabdi · Aug 21, 2017

evangelico793 said:
No longer look at the picture

Please send picture again
404 not found

evangelico793 said:
Something Looks Wrong ReCheck inputs Or No Update Available
for moto g4 play 6.0.1

You are doing something wrong see thread for help and examples

Q9Nap · Sep 3, 2017

@erfanoabdi
Thank you a ton for these tools!
They work like a charm :good:
I knew there must be a way to do this, but it's definitely above my paygrade.

Q9Nap · Sep 7, 2017

@erfanoabdi
I've been trying scriptpatcher.sh on various updater-scripts and have noticed some bugs.
If you're interested in looking at the files, let me know and I'll post them.

erfanoabdi · Sep 7, 2017

Q9Nap said:
@erfanoabdi
I've been trying scriptpatcher.sh on various updater-scripts and have noticed some bugs.
If you're interested in looking at the files, let me know and I'll post them.

Yeah, there's lots of bugs in that script
I appreciate any kind of help
So far i know :
Only supporting /dev/block/boot.../by-name partitions
Can't mount ext4 in macOS
I disabled verify, but we can verify images
And i haven't tested it so much

If it gets more complicated Maybe we have to rewrite it on python.

Q9Nap · Sep 8, 2017

erfanoabdi said:
Yeah, there's lots of bugs in that script
I appreciate any kind of help
So far i know :
Only supporting /dev/block/boot.../by-name partitions
Can't mount ext4 in macOS
I disabled verify, but we can verify images
And i haven't tested it so much

If it gets more complicated Maybe we have to rewrite it on python.

*removed*

Q9Nap · Sep 23, 2017

@erfanoabdi

I noticed that there was an update to the source today to enable the bonus file argument.
I tried to create a recovery.img with a boot.img, recovery-from-boot.p, and recovery-resource.dat.

I tried patching to create a new recovery image and also tried patching the boot image in place.
It always returns "Unknown patch file format".

I am able to use these files to create recovery with the on-device applypatch binary, so not sure why it's throwing the "unknown patch file format" error.

erfanoabdi · Sep 23, 2017

Q9Nap said:
@erfanoabdi

I noticed that there was an update to the source today to enable the bonus file argument.
I tried to create a recovery.img with a boot.img, recovery-from-boot.p, and recovery-resource.dat.

I tried patching to create a new recovery image:

#Output:
creating cache dir cache
cache dir: cache
patch boot.img:
failed to stat "recovery.img": No such file or directory
Unknown patch file format
Done with error code : 1

and also tried patching the boot image in place:

#Output:
creating cache dir cache
cache dir: cache
patch boot.img:
Unknown patch file format
Done with error code : 1

I am able to use these files to create recovery with the on-device applypatch binary, so not sure why it's throwing the "unknown patch file format" error.

Any ideas? Is my syntax incorrect?

That was chainfire commit changes how can i say "no" to his PR
I didn't had time to check it
I'll check it out when I could

erfanoabdi · Sep 24, 2017

Q9Nap said:
@erfanoabdi

I noticed that there was an update to the source today to enable the bonus file argument.
I tried to create a recovery.img with a boot.img, recovery-from-boot.p, and recovery-resource.dat.

I tried patching to create a new recovery image:

#Output:
creating cache dir cache
cache dir: cache
patch boot.img:
failed to stat "recovery.img": No such file or directory
Unknown patch file format
Done with error code : 1

and also tried patching the boot image in place:

#Output:
creating cache dir cache
cache dir: cache
patch boot.img:
Unknown patch file format
Done with error code : 1

I am able to use these files to create recovery with the on-device applypatch binary, so not sure why it's throwing the "unknown patch file format" error.

Any ideas? Is my syntax incorrect?

I've tested recovery-from-boot by chainfire and i can confirm its working
Also i fixed no such file error
Try to compile new source

Q9Nap · Sep 24, 2017

*edit*
Fixed!

[DEV][LINUX/OSX] IMG Patch Tools | sdat2img for OTA zips

Recognized Developer

Recognized Developer

Recognized Developer

Senior Member

Inactive Recognized Contributor

Senior Member

Recognized Developer

Member

Recognized Developer

Member

Member

Recognized Developer

Senior Member

Senior Member

Recognized Developer

Senior Member

Senior Member

Recognized Developer

Recognized Developer

Senior Member

Similar threads

Top Liked Posts