FORUMS
Remove All Ads from XDA
Honor 7x
Win an Honor 7X!

[DEV][LINUX/OSX] IMG Patch Tools | sdat2img for OTA zips

773 posts
Thanks Meter: 922
 
Post Reply Email Thread
OTA

most part of Introduction from taken sdat2img thread
if you want extract non OTA zip i suggest to use sdat2img tool

in this tools i made block_image_update() function from android recovery to a binary to run on PC and patch system.img with DAT files on OTA package to update system.img by OTA

also we have apply_patch() function for patching boot.img and other files like firmwares

Introduction
You probably know already that starting from Android 5.x (Lollipop) compiled roms (aosp,cm,stock) are not compressed anymore the way they used to be on previous android versions. On previous versions all content inside /system folder that has to be extracted within our device was either uncompressed (simple /system folder inside our flashable zip) or compressed in a system.img file, which it is a ext4 compressed file; both of these, anyway, were readable and we could see all system files (app,framework, etc).
The problem comes in >=5.0 versions, this method is not used anymore. Why? Because roms started to be always larger, so it is necessary to compress them even more.


What does new Android zips (full roms, but also OTAs) contain?
New Android flashable zips are made this way:
boot.img (kernel)
file_contexts (selinux related)
META-INF (folder containing scripts)
system.new.dat (compressed /system partition)
system.patch.dat (for OTAs)
system.transfer.list (see explanation below)
and other patch files (.p only on OTA zip)
What does updater-script contains then?
The updater-script uses a brand new function: block_image_update(), this method basically decompresses necessary files inside the device. Let's study it.
From google git source code, if we go inside the new file /bootable/recovery/updater/blockimg.c, we find at the end of it the registration of the function block_image_update() as the method BlockImageUpdateFn() which starts at line 254. Here finally we find all information we need to know about the decompression of the .dat file(s). First file we analyze is system.transfer.list which Google tells us:

Quote:

The transfer list is a text file containing commands to transfer data from one place to another on the target partition.

But what each line means?:

Quote:

First line is the version number of the transfer list; 1 for android 5.0.x, 2 for android 5.1.x, 3 for android 6.0.x, 4 for android 7.x
Second line is the total number of blocks being written
Third line is how many stash entries are needed simultaneously (only on versions >= 2)
Fourth line is the maximum number of blocks that will be stashed simultaneously (only on versions >= 2)
Fifth line and subsequent lines are all individual transfer commands.

all transfer commands is :
  • bsdiff
  • *erase
  • free
  • imgdiff
  • move
  • *new
  • stash
  • *zero
BlockImageUpdate is reading system.transfer.list and executing all commands
But BlockImageVerify doesn’t execute * commands, which not to make changes on system.img just verifying update can happen

original sdat2img tool only support "new" command


Ok, but how to Patch the system.img with OTA files?
All instructions are below. binaries are involved. Please read carefully step by step.


You can use/modify these files and/or include them in your work as long as proper credits and a link to this thread are given.

If you have questions or problems
write here

Thanks
- @xpirt , for original sdat2img tool and useful thread

XDA:DevDB Information
IMG Patch Tools, sdat2img for OTA zips, Tool/Utility for all devices (see above for details)

Contributors
erfanoabdi
Source Code: https://github.com/erfanoabdi/imgpatchtools


Version Information
Status: Testing

Created 2017-07-21
Last Updated 2017-07-24
The Following 11 Users Say Thank You to erfanoabdi For This Useful Post: [ View ] Gift erfanoabdi Ad-Free
 
 
21st July 2017, 11:16 PM |#2  
erfanoabdi's Avatar
OP Senior Member
Flag Famagusta
Thanks Meter: 922
 
More
Usage
Code:
./BlockImageUpdate <system.img> <system.transfer.list> <system.new.dat> <system.patch.dat>
args:

<system.img> = block device (or file) to modify in-place
<system.transfer.list> = transfer list (blob) from OTA/rom zip
<system.new.dat> = new data stream from OTA/rom zip
<system.patch.dat> = patch stream from OTA/rom zip
Code:
./ApplyPatch <file> <target> <tgt_sha1> <size> <init_sha1(1)> <patch(1)> [init_sha1(2)] [patch(2)]...
args:

<file> = source file from rom zip
<target> = target file (use "-" to patch source file)
<tgt_sha1> = target SHA1 Sum after patching
<size> = file size
<init_sha1> = file SHA1 sum
<patch> = patch file (.p) from OTA zip
Code:
usage: ./scriptpatcher.sh <updater-script>
args:

<updater-script> = updater-script from OTA zip to patch recovery commands
Example
for example from updater-script of OTA we have:
Code:
block_image_update("/dev/block/bootdevice/by-name/system", package_extract_file("system.transfer.list"), "system.new.dat", "system.patch.dat")

apply_patch("EMMC:/dev/block/bootdevice/by-name/boot:33554432:f32a854298814c18b12d56412f6e3a31afc95e42:33554432:0041a4df844d4b14c0085921d84572f48cc79ff4",
            "-", 0041a4df844d4b14c0085921d84572f48cc79ff4, 33554432,
            f32a854298814c18b12d56412f6e3a31afc95e42,
            package_extract_file("patch/boot.img.p"))
after getting system.img and boot.img from firmware This is equals of previous functions on PC with this tools:

Code:
~$ ./BlockImageUpdate system.img system.transfer.list system.new.dat system.patch.dat
~$ ./ApplyPatch boot.img - 0041a4df844d4b14c0085921d84572f48cc79ff4 33554432 f32a854298814c18b12d56412f6e3a31afc95e42
scriptpatcher.sh will generate all commands automatically from updater script so run it like:

Code:
~$ ./scriptpatcher.sh META-INF/com/google/android/updater-script > fullpatch.sh
check fullpatch.sh your self, you need to provide all images and files in correct name and patch as mentioned in mount and other commands of fullpatch.sh

Building Requirements
For Building this tool you need :
  • zlib
  • libbz2
  • openssl
It currently supports Linux x86/x64 & MacOS, Not tested on Windows.

Compile and Build command:
Code:
make
Youtube
The Following 5 Users Say Thank You to erfanoabdi For This Useful Post: [ View ] Gift erfanoabdi Ad-Free
21st July 2017, 11:16 PM |#3  
erfanoabdi's Avatar
OP Senior Member
Flag Famagusta
Thanks Meter: 922
 
More
Download links:
GitHub Release

Changelog:
GitHub Commits

Known Bugs/Issues:
  • Plz test and report
The Following 3 Users Say Thank You to erfanoabdi For This Useful Post: [ View ] Gift erfanoabdi Ad-Free
22nd July 2017, 07:56 AM |#5  
rootlinux's Avatar
Recognized Contributor
Flag Cleveland
Thanks Meter: 3,613
 
Donate to Me
More
Very nice

Sent from my LG-LS997 using Tapatalk
30th July 2017, 08:07 PM |#6  
Senior Member
Thanks Meter: 239
 
More
Wow, I tried to apply OTAs manually myself, but never got it to work. Getting the propper tools from you is awesome!

One tiny question though: Does applying/verifying a system update to system.img require a lot of temporary space or memory?

I am running Ubuntu x64 in a VM (VirtualBox) on Windows 10. Verifying a system update doesn't really do much, except keeping the CPU busy.

This is the output of "./BlockImageVerify system.img system.transfer.list system.new.dat system.patch.dat":
Code:
performing verification
creating cache dir cache
cache dir: cache
blockimg version is 3
maximum stash entries 236
creating stash cache/537ed49fb5f6c32dc3d205d78b6084fe54f70cd3/
The cache folder remains empty, there is no harddisk activity and only the CPU is working at 100 %. I interrupted BlockImageVerify after 30 minutes.
31st July 2017, 07:46 AM |#7  
erfanoabdi's Avatar
OP Senior Member
Flag Famagusta
Thanks Meter: 922
 
More
Quote:
Originally Posted by daniel_m

Wow, I tried to apply OTAs manually myself, but never got it to work. Getting the propper tools from you is awesome!

One tiny question though: Does applying/verifying a system update to system.img require a lot of temporary space or memory?

I am running Ubuntu x64 in a VM (VirtualBox) on Windows 10. Verifying a system update doesn't really do much, except keeping the CPU busy.

This is the output of "./BlockImageVerify system.img system.transfer.list system.new.dat system.patch.dat":

The cache folder remains empty, there is no harddisk activity and only the CPU is working at 100 %. I interrupted BlockImageVerify after 30 minutes.


Ah actually yes you need free space on RAM as much as both patch.dat and new.dat sizes
It's on my todo list to fix it, https://github.com/erfanoabdi/imgpat...ckimg.cpp#L383
The Following User Says Thank You to erfanoabdi For This Useful Post: [ View ] Gift erfanoabdi Ad-Free
21st August 2017, 08:28 PM |#8  
evangelico793's Avatar
Junior Member
Thanks Meter: 0
 
More
How to extract the file system.new.dat
21st August 2017, 08:45 PM |#9  
erfanoabdi's Avatar
OP Senior Member
Flag Famagusta
Thanks Meter: 922
 
More
Quote:
Originally Posted by evangelico793

How to extract the file system.new.dat

You can find it in OTA zip

Motorola? See this : motorola.erfanabdi.ir
The Following User Says Thank You to erfanoabdi For This Useful Post: [ View ] Gift erfanoabdi Ad-Free
21st August 2017, 09:05 PM |#10  
evangelico793's Avatar
Junior Member
Thanks Meter: 0
 
More
Quote:
Originally Posted by erfanoabdi

You can find it in OTA zip

Motorola? See this : motorola.erfanabdi.ir

No longer look at the picture
Attached Thumbnails
Click image for larger version

Name:	Sem título.jpg
Views:	248
Size:	193.7 KB
ID:	4248536  
21st August 2017, 09:31 PM |#11  
evangelico793's Avatar
Junior Member
Thanks Meter: 0
 
More
Quote:
Originally Posted by erfanoabdi

You can find it in OTA zip

Motorola? See this : motorola.erfanabdi.ir

Something Looks Wrong ReCheck inputs Or No Update Available
for moto g4 play 6.0.1
Post Reply Subscribe to Thread
Previous Thread Next Thread
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes