[DEV][LINUX/OSX] IMG Patch Tools | sdat2img for OTA zips

Search This thread

erfanoabdi

Recognized Developer
Jan 3, 2015
1,010
3,156
Tehran
erfanoabdi.github.io
OTA-Updates.png

most part of Introduction from taken sdat2img thread
if you want extract non OTA zip i suggest to use sdat2img tool
FOR AB OTA ZIPS USE THIS SCRIPT AND PAYLOAD EXTRACTOR

in this tools i made block_image_update() function from android recovery to a binary to run on PC and patch system.img with DAT files on OTA package to update system.img by OTA

also we have apply_patch() function for patching boot.img and other files like firmwares

Introduction
You probably know already that starting from Android 5.x (Lollipop) compiled roms (aosp,cm,stock) are not compressed anymore the way they used to be on previous android versions. On previous versions all content inside /system folder that has to be extracted within our device was either uncompressed (simple /system folder inside our flashable zip) or compressed in a system.img file, which it is a ext4 compressed file; both of these, anyway, were readable and we could see all system files (app,framework, etc).
The problem comes in >=5.0 versions, this method is not used anymore. Why? Because roms started to be always larger, so it is necessary to compress them even more.


What does new Android zips (full roms, but also OTAs) contain?
New Android flashable zips are made this way:
boot.img (kernel)
file_contexts (selinux related)
META-INF (folder containing scripts)
system.new.dat (compressed /system partition)
system.patch.dat (for OTAs)
system.transfer.list (see explanation below)
and other patch files (.p only on OTA zip)

What does updater-script contains then?
The updater-script uses a brand new function: block_image_update(), this method basically decompresses necessary files inside the device. Let's study it.
From google git source code, if we go inside the new file /bootable/recovery/updater/blockimg.c, we find at the end of it the registration of the function block_image_update() as the method BlockImageUpdateFn() which starts at line 254. Here finally we find all information we need to know about the decompression of the .dat file(s). First file we analyze is system.transfer.list which Google tells us:

The transfer list is a text file containing commands to transfer data from one place to another on the target partition.

But what each line means?:

First line is the version number of the transfer list; 1 for android 5.0.x, 2 for android 5.1.x, 3 for android 6.0.x, 4 for android 7.x
Second line is the total number of blocks being written
Third line is how many stash entries are needed simultaneously (only on versions >= 2)
Fourth line is the maximum number of blocks that will be stashed simultaneously (only on versions >= 2)
Fifth line and subsequent lines are all individual transfer commands.
all transfer commands is :
  • bsdiff
  • *erase
  • free
  • imgdiff
  • move
  • *new
  • stash
  • *zero
BlockImageUpdate is reading system.transfer.list and executing all commands
But BlockImageVerify doesn’t execute * commands, which not to make changes on system.img just verifying update can happen

original sdat2img tool only support "new" command


Ok, but how to Patch the system.img with OTA files?
All instructions are below. binaries are involved. Please read carefully step by step.


You can use/modify these files and/or include them in your work as long as proper credits and a link to this thread are given.

If you have questions or problems
write here ;)

Thanks
- @xpirt , for original sdat2img tool and useful thread

XDA:DevDB Information
IMG Patch Tools, sdat2img for OTA zips, Tool/Utility for all devices (see above for details)

Contributors
erfanoabdi
Source Code: https://github.com/erfanoabdi/imgpatchtools


Version Information
Status: Testing

Created 2017-07-21
Last Updated 2020-01-23
 

erfanoabdi

Recognized Developer
Jan 3, 2015
1,010
3,156
Tehran
erfanoabdi.github.io
Usage
Code:
./BlockImageUpdate <system.img> <system.transfer.list> <system.new.dat> <system.patch.dat>
args:

<system.img> = block device (or file) to modify in-place
<system.transfer.list> = transfer list (blob) from OTA/rom zip
<system.new.dat> = new data stream from OTA/rom zip
<system.patch.dat> = patch stream from OTA/rom zip
Code:
./ApplyPatch <file> <target> <tgt_sha1> <size> <init_sha1(1)> <patch(1)> [init_sha1(2)] [patch(2)]...
args:

<file> = source file from rom zip
<target> = target file (use "-" to patch source file)
<tgt_sha1> = target SHA1 Sum after patching
<size> = file size
<init_sha1> = file SHA1 sum
<patch> = patch file (.p) from OTA zip
Code:
usage: ./scriptpatcher.sh <updater-script>
args:

<updater-script> = updater-script from OTA zip to patch recovery commands

Example
for example from updater-script of OTA we have:
Code:
block_image_update("/dev/block/bootdevice/by-name/system", package_extract_file("system.transfer.list"), "system.new.dat", "system.patch.dat")

apply_patch("EMMC:/dev/block/bootdevice/by-name/boot:33554432:f32a854298814c18b12d56412f6e3a31afc95e42:33554432:0041a4df844d4b14c0085921d84572f48cc79ff4",
            "-", 0041a4df844d4b14c0085921d84572f48cc79ff4, 33554432,
            f32a854298814c18b12d56412f6e3a31afc95e42,
            package_extract_file("patch/boot.img.p"))
after getting system.img and boot.img from firmware This is equals of previous functions on PC with this tools:

Code:
~$ ./BlockImageUpdate system.img system.transfer.list system.new.dat system.patch.dat
~$ ./ApplyPatch boot.img - 0041a4df844d4b14c0085921d84572f48cc79ff4 33554432 f32a854298814c18b12d56412f6e3a31afc95e42
scriptpatcher.sh will generate all commands automatically from updater script so run it like:

Code:
~$ ./scriptpatcher.sh META-INF/com/google/android/updater-script > fullpatch.sh
check fullpatch.sh your self, you need to provide all images and files in correct name and patch as mentioned in mount and other commands of fullpatch.sh

Building Requirements
For Building this tool you need :
  • zlib
  • libbz2
  • openssl
It currently supports Linux x86/x64 & MacOS, Not tested on Windows.

Compile and Build command:
Code:
make

Youtube
 
Last edited:

daniel_m

Senior Member
Oct 7, 2012
668
290
Wow, I tried to apply OTAs manually myself, but never got it to work. Getting the propper tools from you is awesome!

One tiny question though: Does applying/verifying a system update to system.img require a lot of temporary space or memory?

I am running Ubuntu x64 in a VM (VirtualBox) on Windows 10. Verifying a system update doesn't really do much, except keeping the CPU busy.

This is the output of "./BlockImageVerify system.img system.transfer.list system.new.dat system.patch.dat":
Code:
performing verification
creating cache dir cache
cache dir: cache
blockimg version is 3
maximum stash entries 236
creating stash cache/537ed49fb5f6c32dc3d205d78b6084fe54f70cd3/
The cache folder remains empty, there is no harddisk activity and only the CPU is working at 100 %. I interrupted BlockImageVerify after 30 minutes.
 

erfanoabdi

Recognized Developer
Jan 3, 2015
1,010
3,156
Tehran
erfanoabdi.github.io
Wow, I tried to apply OTAs manually myself, but never got it to work. Getting the propper tools from you is awesome!

One tiny question though: Does applying/verifying a system update to system.img require a lot of temporary space or memory?

I am running Ubuntu x64 in a VM (VirtualBox) on Windows 10. Verifying a system update doesn't really do much, except keeping the CPU busy.

This is the output of "./BlockImageVerify system.img system.transfer.list system.new.dat system.patch.dat":

The cache folder remains empty, there is no harddisk activity and only the CPU is working at 100 %. I interrupted BlockImageVerify after 30 minutes.
;)
Ah actually yes you need free space on RAM as much as both patch.dat and new.dat sizes
It's on my todo list to fix it, https://github.com/erfanoabdi/imgpatchtools/blob/master/blockimg/blockimg.cpp#L383
 
  • Like
Reactions: Q9Nap

Q9Nap

Senior Member
Feb 13, 2009
1,018
1,883
@erfanoabdi
I've been trying scriptpatcher.sh on various updater-scripts and have noticed some bugs.
If you're interested in looking at the files, let me know and I'll post them.
 

erfanoabdi

Recognized Developer
Jan 3, 2015
1,010
3,156
Tehran
erfanoabdi.github.io
@erfanoabdi
I've been trying scriptpatcher.sh on various updater-scripts and have noticed some bugs.
If you're interested in looking at the files, let me know and I'll post them.
Yeah, there's lots of bugs in that script
I appreciate any kind of help
So far i know :
Only supporting /dev/block/boot.../by-name partitions
Can't mount ext4 in macOS
I disabled verify, but we can verify images
And i haven't tested it so much

If it gets more complicated Maybe we have to rewrite it on python.
 
  • Like
Reactions: Q9Nap

Q9Nap

Senior Member
Feb 13, 2009
1,018
1,883
Yeah, there's lots of bugs in that script
I appreciate any kind of help
So far i know :
Only supporting /dev/block/boot.../by-name partitions
Can't mount ext4 in macOS
I disabled verify, but we can verify images
And i haven't tested it so much

If it gets more complicated Maybe we have to rewrite it on python.

*removed*
 
Last edited:

Q9Nap

Senior Member
Feb 13, 2009
1,018
1,883
@erfanoabdi

I noticed that there was an update to the source today to enable the bonus file argument.
I tried to create a recovery.img with a boot.img, recovery-from-boot.p, and recovery-resource.dat.

I tried patching to create a new recovery image and also tried patching the boot image in place.
It always returns "Unknown patch file format".

I am able to use these files to create recovery with the on-device applypatch binary, so not sure why it's throwing the "unknown patch file format" error.
 
Last edited:

erfanoabdi

Recognized Developer
Jan 3, 2015
1,010
3,156
Tehran
erfanoabdi.github.io
@erfanoabdi

I noticed that there was an update to the source today to enable the bonus file argument.
I tried to create a recovery.img with a boot.img, recovery-from-boot.p, and recovery-resource.dat.

I tried patching to create a new recovery image:

#Output:
creating cache dir cache
cache dir: cache
patch boot.img:
failed to stat "recovery.img": No such file or directory
Unknown patch file format
Done with error code : 1

and also tried patching the boot image in place:


#Output:
creating cache dir cache
cache dir: cache
patch boot.img:
Unknown patch file format
Done with error code : 1

I am able to use these files to create recovery with the on-device applypatch binary, so not sure why it's throwing the "unknown patch file format" error.

Any ideas? Is my syntax incorrect?
That was chainfire commit changes how can i say "no" to his PR
I didn't had time to check it
I'll check it out when I could
 
  • Like
Reactions: Q9Nap

erfanoabdi

Recognized Developer
Jan 3, 2015
1,010
3,156
Tehran
erfanoabdi.github.io
@erfanoabdi

I noticed that there was an update to the source today to enable the bonus file argument.
I tried to create a recovery.img with a boot.img, recovery-from-boot.p, and recovery-resource.dat.

I tried patching to create a new recovery image:

#Output:
creating cache dir cache
cache dir: cache
patch boot.img:
failed to stat "recovery.img": No such file or directory
Unknown patch file format
Done with error code : 1

and also tried patching the boot image in place:


#Output:
creating cache dir cache
cache dir: cache
patch boot.img:
Unknown patch file format
Done with error code : 1

I am able to use these files to create recovery with the on-device applypatch binary, so not sure why it's throwing the "unknown patch file format" error.

Any ideas? Is my syntax incorrect?
I've tested recovery-from-boot by chainfire and i can confirm its working
Also i fixed no such file error
Try to compile new source ;)
 
  • Like
Reactions: Q9Nap

Top Liked Posts

  • There are no posts matching your filters.
  • 30
    OTA-Updates.png

    most part of Introduction from taken sdat2img thread
    if you want extract non OTA zip i suggest to use sdat2img tool
    FOR AB OTA ZIPS USE THIS SCRIPT AND PAYLOAD EXTRACTOR

    in this tools i made block_image_update() function from android recovery to a binary to run on PC and patch system.img with DAT files on OTA package to update system.img by OTA

    also we have apply_patch() function for patching boot.img and other files like firmwares

    Introduction
    You probably know already that starting from Android 5.x (Lollipop) compiled roms (aosp,cm,stock) are not compressed anymore the way they used to be on previous android versions. On previous versions all content inside /system folder that has to be extracted within our device was either uncompressed (simple /system folder inside our flashable zip) or compressed in a system.img file, which it is a ext4 compressed file; both of these, anyway, were readable and we could see all system files (app,framework, etc).
    The problem comes in >=5.0 versions, this method is not used anymore. Why? Because roms started to be always larger, so it is necessary to compress them even more.


    What does new Android zips (full roms, but also OTAs) contain?
    New Android flashable zips are made this way:
    boot.img (kernel)
    file_contexts (selinux related)
    META-INF (folder containing scripts)
    system.new.dat (compressed /system partition)
    system.patch.dat (for OTAs)
    system.transfer.list (see explanation below)
    and other patch files (.p only on OTA zip)

    What does updater-script contains then?
    The updater-script uses a brand new function: block_image_update(), this method basically decompresses necessary files inside the device. Let's study it.
    From google git source code, if we go inside the new file /bootable/recovery/updater/blockimg.c, we find at the end of it the registration of the function block_image_update() as the method BlockImageUpdateFn() which starts at line 254. Here finally we find all information we need to know about the decompression of the .dat file(s). First file we analyze is system.transfer.list which Google tells us:

    The transfer list is a text file containing commands to transfer data from one place to another on the target partition.

    But what each line means?:

    First line is the version number of the transfer list; 1 for android 5.0.x, 2 for android 5.1.x, 3 for android 6.0.x, 4 for android 7.x
    Second line is the total number of blocks being written
    Third line is how many stash entries are needed simultaneously (only on versions >= 2)
    Fourth line is the maximum number of blocks that will be stashed simultaneously (only on versions >= 2)
    Fifth line and subsequent lines are all individual transfer commands.
    all transfer commands is :
    • bsdiff
    • *erase
    • free
    • imgdiff
    • move
    • *new
    • stash
    • *zero
    BlockImageUpdate is reading system.transfer.list and executing all commands
    But BlockImageVerify doesn’t execute * commands, which not to make changes on system.img just verifying update can happen

    original sdat2img tool only support "new" command


    Ok, but how to Patch the system.img with OTA files?
    All instructions are below. binaries are involved. Please read carefully step by step.


    You can use/modify these files and/or include them in your work as long as proper credits and a link to this thread are given.

    If you have questions or problems
    write here ;)

    Thanks
    - @xpirt , for original sdat2img tool and useful thread

    XDA:DevDB Information
    IMG Patch Tools, sdat2img for OTA zips, Tool/Utility for all devices (see above for details)

    Contributors
    erfanoabdi
    Source Code: https://github.com/erfanoabdi/imgpatchtools


    Version Information
    Status: Testing

    Created 2017-07-21
    Last Updated 2020-01-23
    8
    Usage
    Code:
    ./BlockImageUpdate <system.img> <system.transfer.list> <system.new.dat> <system.patch.dat>
    args:
    
    <system.img> = block device (or file) to modify in-place
    <system.transfer.list> = transfer list (blob) from OTA/rom zip
    <system.new.dat> = new data stream from OTA/rom zip
    <system.patch.dat> = patch stream from OTA/rom zip
    Code:
    ./ApplyPatch <file> <target> <tgt_sha1> <size> <init_sha1(1)> <patch(1)> [init_sha1(2)] [patch(2)]...
    args:
    
    <file> = source file from rom zip
    <target> = target file (use "-" to patch source file)
    <tgt_sha1> = target SHA1 Sum after patching
    <size> = file size
    <init_sha1> = file SHA1 sum
    <patch> = patch file (.p) from OTA zip
    Code:
    usage: ./scriptpatcher.sh <updater-script>
    args:
    
    <updater-script> = updater-script from OTA zip to patch recovery commands

    Example
    for example from updater-script of OTA we have:
    Code:
    block_image_update("/dev/block/bootdevice/by-name/system", package_extract_file("system.transfer.list"), "system.new.dat", "system.patch.dat")
    
    apply_patch("EMMC:/dev/block/bootdevice/by-name/boot:33554432:f32a854298814c18b12d56412f6e3a31afc95e42:33554432:0041a4df844d4b14c0085921d84572f48cc79ff4",
                "-", 0041a4df844d4b14c0085921d84572f48cc79ff4, 33554432,
                f32a854298814c18b12d56412f6e3a31afc95e42,
                package_extract_file("patch/boot.img.p"))
    after getting system.img and boot.img from firmware This is equals of previous functions on PC with this tools:

    Code:
    ~$ ./BlockImageUpdate system.img system.transfer.list system.new.dat system.patch.dat
    ~$ ./ApplyPatch boot.img - 0041a4df844d4b14c0085921d84572f48cc79ff4 33554432 f32a854298814c18b12d56412f6e3a31afc95e42
    scriptpatcher.sh will generate all commands automatically from updater script so run it like:

    Code:
    ~$ ./scriptpatcher.sh META-INF/com/google/android/updater-script > fullpatch.sh
    check fullpatch.sh your self, you need to provide all images and files in correct name and patch as mentioned in mount and other commands of fullpatch.sh

    Building Requirements
    For Building this tool you need :
    • zlib
    • libbz2
    • openssl
    It currently supports Linux x86/x64 & MacOS, Not tested on Windows.

    Compile and Build command:
    Code:
    make

    Youtube
    5
    Download links:
    GitHub Release

    Changelog:
    GitHub Commits

    Known Bugs/Issues:
    • Plz test and report
    4
    So I finally found a fully working solution:
    Use ApplyPatch and imgdiff compiled on Ubuntu 14.04;
    Use BlockImageUpdate, BlockImageVerify, and scriptpatcher.sh compiled on Ubuntu 16.04;
    On Ubuntu 16.04+.
    Here's a link to my binaries that are fully working on Ubuntu 16.04:
    https://mega.nz/#!TRIjkZAB!c12pNem1FHfQMwAoMlY8-R4iw4Dnr_eG_xtcUWQGHbI
    2
    How to extract the file system.new.dat
    You can find it in OTA zip

    Motorola? See this : motorola.erfanabdi.ir