FORUMS
Remove All Ads from XDA

[DEV] sdat2img 1.0 - img2sdat 1.2 ◦ Unpack/re-pack android DAT files

4,889 posts
Thanks Meter: 20,366
 
Post Reply Email Thread
Introduction
You probably know already that starting from Android 5.x (Lollipop) compiled roms (aosp,cm,stock) are not compressed anymore the way they used to be on previous android versions. On previous versions all content inside /system folder that has to be extracted within our device was either uncompressed (simple /system folder inside our flashable zip) or compressed in a system.img file, which it is a ext4 compressed file; both of these, anyway, were readable and we could see all system files (app,framework, etc).
The problem comes in >=5.0 versions, this method is not used anymore. Why? Because roms started to be always larger, so it is necessary to compress them even more.


What does new Android zips (full roms, but also otas) contain?
New Android flashable zips are made this way:
boot.img (kernel)
file_contexts (selinux related)
META-INF (folder containing scripts)
system.new.dat (compressed /system partition)
system.patch.dat (for OTAs)
system.transfer.list (see explanation below)

What does updater-script contains then?
The updater-script uses a brand new function: block_image_update(), this method basically decompresses necessary files inside the device. Let's study it.
From google git source code, if we go inside the new file /bootable/recovery/updater/blockimg.c, we find at the end of it the registration of the function block_image_update() as the method BlockImageUpdateFn() which starts at line 254. Here finally we find all information we need to know about the decompression of the .dat file(s). First file we analyze is system.transfer.list which Google tells us:

Quote:

The transfer list is a text file containing commands to transfer data from one place to another on the target partition.


But what each line means?:
Quote:

First line is the version number of the transfer list; 1 for android 5.0.x, 2 for android 5.1.x, 3 for android 6.0.x, 4 for android 7.x/8.x
Second line is the total number of blocks being written
Third line is how many stash entries are needed simultaneously (only on versions >= 2)
Fourth line is the maximum number of blocks that will be stashed simultaneously (only on versions >= 2)
Fifth line and subsequent lines are all individual transfer commands.


Ok, but how to decompress the system.new.dat file?
All instructions are below. sdat2img, rimg2sdat and img2sdat binaries are involved. Please read carefully step by step.


NOTE for OTAs
If you are looking on decompressing system.patch.dat file or .p files, therefore reproduce the patching system on your PC, check imgpatchtools out by @erfanoabdi.


You can use/modify these files and/or include them in your work as long as proper credits and a link to this thread are given.

If you have questions or problems write here


Are you building from source and want to revert to the old flashing method without DAT files?
Do this (thx @Elluel):
1 - Delete this line
2 - Make sure you have this commit


Thanks
- howellzhu & luxi78 - for initial source codes
- all contributors from github
The Following 287 Users Say Thank You to xpirt For This Useful Post: [ View ]
 
 
20th December 2014, 05:03 PM |#2  
xpirt's Avatar
OP Recognized Developer / Recognized Contributor
Thanks Meter: 20,366
 
Donate to Me
More
Ubuntu
Ubuntu 64-bit - Guide

Step 1 - Decompressing = DAT (sparse data) -> EXT4 (raw image)

We're now using sdat2img binary, the usage is very simple (make sure you have python 2.7+ installed):
./sdat2img.py <transfer_list> <system_new_file> [system_img]
- <transfer_list> = input, system.transfer.list from rom zip
- <system_new_file> = input, system.new.dat from rom zip
- [system_img] = output ext4 raw image file

and a quick example:
./sdat2img.py system.transfer.list system.new.dat system.img
by running this command you will get as output the file my_new_system.img which is the raw ext4 image.


Step 2 - Decompress EXT4 (raw image) -> OUTPUT folder -> Compress EXT4 (raw image)

Now we need to mount or ext4 raw image into an output folder so we can see apks/jars etc.
To do this we need to type this command:
sudo mount -t ext4 -o loop system.img output/
As you can see there is a new folder called output which we can edit/modify/delete your files (not able to? see here)

Now we need to compress it back to a raw ext4 image, to do this we need the make_ext4fs binary. Make sure you have the file_contexts file (taken from the Rom zip) inside the make_ext4fs path. Then type this (got issues? see here)
./make_ext4fs -T 0 -S file_contexts -l 1073741824 -a system system_new.img output/
You will get the new raw ext4 image called 'system_new.img' ready for the next step.

Step 3 - Converting = EXT4 (raw image) -> IMG (sparse image)

Now we need to convert the ext4 raw image into a sparse image. For this you need img2simg binary you can find here (thx to @A.S._id).
The usage is simple:
img2simg <raw_image_file> <sparse_image_file>
Pretty self-explanatory, the output will be a new sparse image (.img).


Step 4 - Converting = IMG (sparse image) -> DAT (sparse data)

Now we need the img2sdat binary, the usage is very simple (make sure you have python 2.7+ installed):
./img2sdat.py <system_img>
- <system_img> = name of input sparse image file (from step 3)

As you can see the output is composed by system.transfer.list, (system.patch.dat) & system.new.dat, ready to be replaced inside your Rom zip.


DOWNLOADs
sdat2img.py
- github.com
make_ext4fs
- mega.co.nz
img2sdat.py
- github.com
The Following 129 Users Say Thank You to xpirt For This Useful Post: [ View ]
20th December 2014, 05:03 PM |#3  
xpirt's Avatar
OP Recognized Developer / Recognized Contributor
Thanks Meter: 20,366
 
Donate to Me
More
Windows
Windows 32/64 bit - Guide

Step 1 - Decompressing = DAT (sparse data) -> EXT4 (raw image)

We're now using sdat2img.py binary, the usage is very simple (make sure you have python 3.x installed):
sdat2img.py <transfer_list> <system_new_file> <system_ext4>
- <transfer_list> = input, system.transfer.list from rom zip
- <system_new_file> = input, system.new.dat from rom zip
- <system_ext4> = output ext4 raw image file

and a quick example:
sdat2img.py system.transfer.list system.new.dat system.img
by running this command you will get as output the file my_new_system.img which is the raw ext4 image.


Step 2 - Decompress EXT4 (raw image) -> OUTPUT folder -> Compress EXT4 (raw image)

Now you should either move to Ubuntu x64/x86 or follow the 2nd step from the Ubuntu guide on cygwin 32 bit.

Step 3 - Converting = EXT4 (raw image) -> IMG (sparse image)

Now we need to convert the ext4 raw image into a sparse image. For this you need img2simg binary you can find here (thx to @A.S._id).
The usage is simple:
img2simg.exe <raw_image_file> <sparse_image_file>
Pretty self-explanatory, the output will be a new sparse image (.img).


Step 4 - Converting = IMG (sparse image) -> DAT (sparse data)

Now we need the img2sdat binary, the usage is very simple (make sure you have python 2.7+ installed):
img2sdat.py <system_img>
- <system_img> = name of input sparse image file (from step 3)

As you can see the output is composed by system.transfer.list, (system.patch.dat) & system.new.dat, ready to be replaced inside your Rom zip.


DOWNLOADs
sdat2img.py
- github.com
make_ext4fs
- mega.co.nz
img2sdat.py
- github.com
The Following 94 Users Say Thank You to xpirt For This Useful Post: [ View ]
20th December 2014, 05:03 PM |#4  
xpirt's Avatar
OP Recognized Developer / Recognized Contributor
Thanks Meter: 20,366
 
Donate to Me
More
Source - Changelog
sdat2img
Repo: https://github.com/xpirt/sdat2img
Changelog: https://github.com/xpirt/sdat2img/commits/master
Source: https://github.com/xpirt/sdat2img/bl...er/sdat2img.py

img2sdat
Repo: https://github.com/xpirt/img2sdat
Changelog: https://github.com/xpirt/img2sdat/commits/master
Source: https://github.com/xpirt/img2sdat/bl...er/img2sdat.py
The Following 40 Users Say Thank You to xpirt For This Useful Post: [ View ]
20th December 2014, 06:49 PM |#5  
01 Vlatce's Avatar
Senior Member
Thanks Meter: 102
 
More
Is there any tool to browse (and/or manipulate) these .dat files in Windows?
The Following User Says Thank You to 01 Vlatce For This Useful Post: [ View ] Gift 01 Vlatce Ad-Free
20th December 2014, 09:14 PM |#6  
xpirt's Avatar
OP Recognized Developer / Recognized Contributor
Thanks Meter: 20,366
 
Donate to Me
More
Quote:
Originally Posted by 01 Vlatce

Is there any tool to browse (and/or manipulate) these .dat files in Windows?

i don't think so, though you can install ubuntu in a virtual machine as 64-bit and decompress there without problems.

edit: windows binaries added to OP.
The Following 6 Users Say Thank You to xpirt For This Useful Post: [ View ]
20th December 2014, 10:11 PM |#7  
01 Vlatce's Avatar
Senior Member
Thanks Meter: 102
 
More
Quote:
Originally Posted by xpirt

i don't think so, though you can install ubuntu in a virtual machine as 64-bit and decompress there without problems.

I found one, Ext2Explore, it can browse and extract lollipop .dat files
The Following 10 Users Say Thank You to 01 Vlatce For This Useful Post: [ View ] Gift 01 Vlatce Ad-Free
21st December 2014, 11:50 AM |#8  
Senior Member
Thanks Meter: 135
 
More
Quote:
Originally Posted by 01 Vlatce

I found one, Ext2Explore, it can browse and extract lollipop .dat files

Can you explain how to work in more detail ? Thanks.
21st December 2014, 02:49 PM |#9  
01 Vlatce's Avatar
Senior Member
Thanks Meter: 102
 
More
Quote:
Originally Posted by chihliouma

Can you explain how to work in more detail ? Thanks.

With that application you can open the system.new.dat file like a zip archive

Sent from my Nexus 5
The Following 2 Users Say Thank You to 01 Vlatce For This Useful Post: [ View ] Gift 01 Vlatce Ad-Free
22nd December 2014, 01:30 AM |#10  
Incredible_Culp's Avatar
Senior Member
Flag Loud City
Thanks Meter: 987
 
More
Quote:
Originally Posted by xpirt

i don't think so, though you can install ubuntu in a virtual machine as 64-bit and decompress there without problems.

Quote:
Originally Posted by 01 Vlatce

I found one, Ext2Explore, it can browse and extract lollipop .dat files

Confirmed.

Works perfectly..

Thanks a lot! This will save a lot of time theming.. (pulling the system, settings, framework without having to flash the rom first)

Oh, just for s&g's i extracted ("decompiled") the system.new file, the files (app, prive-app, framework, media, etc) all got placed on my desktop -- i deleted the system.new file from the rom -- i then placed the folders and files that got extracted and placed them in the rom -- i flashed it -- it booted! lol.. didnt think it would... but anyways, there's that.

For those who want to try:
--- > its basically self explanatory.
- once downloaded (the link above) extract the contents from the zip (i just went ahead and extracted to my desktop)
- right click the linux penguin
- run as administrator - the window will pop up
- go to file > open image > find your system.new image (should already be extracted from the rom and placed on your desktop)
- click open
- you should now see your image (or path) in the windows -- double click them (i doubled clicked both the right and left side)
- now you should be able to see all the folders

---> if you want to extract those folders
- just go up to the top
- click save and choose an extraction (i chose Desktop)

Hoped i helped -- any questions or confusion, dont be shy to ask -- i probably went too fast, idk lol

Enjoy!
The Following 12 Users Say Thank You to Incredible_Culp For This Useful Post: [ View ] Gift Incredible_Culp Ad-Free
22nd December 2014, 05:15 AM |#11  
webdroidmt's Avatar
Senior Member
Flag NJ
Thanks Meter: 1,119
 
More
Quote:
Originally Posted by Incredible_Culp

Confirmed.

Works perfectly..

Thanks a lot! This will save a lot of time theming.. (pulling the system, settings, framework without having to flash the rom first)

Oh, just for s&g's i extracted ("decompiled") the system.new file, the files (app, prive-app, framework, media, etc) all got placed on my desktop -- i deleted the system.new file from the rom -- i then placed the folders and files that got extracted and placed them in the rom -- i flashed it -- it booted! lol.. didnt think it would... but anyways, there's that.

For those who want to try:
--- > its basically self explanatory.
- once downloaded (the link above) extract the contents from the zip (i just went ahead and extracted to my desktop)
- right click the linux penguin
- run as administrator - the window will pop up
- go to file > open image > find your system.new image (should already be extracted from the rom and placed on your desktop)
- click open
- you should now see your image (or path) in the windows -- double click them (i doubled clicked both the right and left side)
- now you should be able to see all the folders

---> if you want to extract those folders
- just go up to the top
- click save and choose an extraction (i chose Desktop)

Hoped i helped -- any questions or confusion, dont be shy to ask -- i probably went too fast, idk lol

Enjoy!

I can't get this to work at all. I'm using Win 7 Pro and I've actually used this app before with stock Nexus 7 factory images and it worked fine.. What file did you actually download? I d/l 2 different versions, one was beta 2.0 and the other was a zip that said 2.2.71 but the about on the exe says 2.1, which is the version I've been using for quite a while.

With the beta 2.0, there's not even an option to open a file and with the 2.1, it errors out every time. After opening system.new.dat image and double clicking, it just hangs and the program stops responding. If I try clicking on it and selecting save, it allows me to select a folder to save it in but I just get a dialog box that says saving files and then hangs at 24% every time, finally crashing with a Microsoft C++ runtime error.

Oh btw, you didn't go too fast LOL, this is really a simple app to use but it's just not working for me with these block image files. Maybe I'm missing something but it's got me scratching my head. If you have any ideas, I'm all ears.

Thanks,
Mike T
Post Reply Subscribe to Thread

Tags
android 5, dat, decompress, img, lollipop
Previous Thread Next Thread
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes