[DEV] sdat2img 1.1 - img2sdat 1.7 ◦ Unpack/re-pack android DAT files

Search This thread

ST Dog

Senior Member
Jun 22, 2013
1,122
341
ls -l ?

What are the permissions?

Is that the executable or a symlink?


The original error still looked to be looking in the wrong place.
 

ST Dog

Senior Member
Jun 22, 2013
1,122
341
So I'm not sure. I use the tools a few years ago and it worked.

Looking back at the first posts there's a download for make_ext4fs in with the python scripts. I'm sure that's what is supposed to be executed.
 

paxar331

Member
Mar 13, 2020
29
3
I placed sdat2img.py in the same directory as system.new.dat.br:
https://i.imgur.com/5hJKw6M.png

Then I executed the following:
Code:
E:\HIS Documents\Computer guides\LG G7\Softwares\lineage\lineage zip extract>C:\
ProgramData\Python\python sdat2img.py system.transfer.list system.new.dat system
.img

Unfortunately no luck for me:
Code:
Android Nougat 7.x / Oreo 8.x detected!

Traceback (most recent call last):
  File "sdat2img.py", line 143, in <module>
    main(TRANSFER_LIST_FILE, NEW_DATA_FILE, OUTPUT_IMAGE_FILE)
  File "sdat2img.py", line 92, in main
    new_data_file = open(NEW_DATA_FILE, 'rb')
FileNotFoundError: [Errno 2] No such file or directory: 'system.new.dat'

https://i.imgur.com/32tYXnL.png

I already made brotli was installed:
https://forum.xda-developers.com/showpost.php?p=82038917&postcount=68

Edit: I had to manually specify the file extension, i.e. system.new.dat.br, instead of, system.new.dat

https://i.imgur.com/q2uNM73.png
https://i.imgur.com/GDW4pF7.png
 
Last edited:

BlueMan_86

Senior Member
Jul 2, 2015
75
7
Novi Sad
Guys, gathered a simple script, converts and extracts. In Russian and English languages. The main task of Extractor is only to unpack, without installing additional software on the PC. Only Windows 64\32bit ;)
Follow the instructions...
Update script v3.6 - Updated sdat2img (DATE: 2018-10-27 10:33:21 CEST)
Is v3.6 Latest Version? And do you have github or post just only for this script so i can follow?
Thanks for your work...
 

VBabbar

Member
  • Feb 4, 2021
    27
    9
    Ubuntu

    Ubuntu 64-bit - Guide

    Step 1 - Decompressing = DAT (sparse data) -> EXT4 (raw image)
    We're now using sdat2img binary, the usage is very simple (make sure you have python 2.7+ installed):
    ./sdat2img.py <transfer_list> <system_new_file> [system_img]​

    - <transfer_list> = input, system.transfer.list from rom zip
    - <system_new_file> = input, system.new.dat from rom zip
    - [system_img] = output ext4 raw image file

    and a quick example:
    ./sdat2img.py system.transfer.list system.new.dat system.img​

    by running this command you will get as output the file my_new_system.img which is the raw ext4 image.




    Step 2 - Decompress EXT4 (raw image) -> OUTPUT folder -> Compress EXT4 (raw image)
    Now we need to mount or ext4 raw image into an output folder so we can see apks/jars etc.
    To do this we need to type this command:
    sudo mount -t ext4 -o loop system.img output/​

    As you can see there is a new folder called output which we can edit/modify/delete your files (not able to? see here)

    Now we need to compress it back to a raw ext4 image, to do this we need the make_ext4fs binary. Make sure you have the file_contexts file (taken from the Rom zip) inside the make_ext4fs path. Then type this (got issues? see here)
    ./make_ext4fs -T 0 -S file_contexts -l 1073741824 -a system system_new.img output/​

    You will get the new raw ext4 image called 'system_new.img' ready for the next step.



    Step 3 - Converting = EXT4 (raw image) -> IMG (sparse image)
    Now we need to convert the ext4 raw image into a sparse image. For this you need img2simg binary you can find here (thx to @A.S._id).
    The usage is simple:
    img2simg <raw_image_file> <sparse_image_file>​

    Pretty self-explanatory, the output will be a new sparse image (.img).




    Step 4 - Converting = IMG (sparse image) -> DAT (sparse data)
    Now we need the img2sdat binary, the usage is very simple (make sure you have python 2.7+ installed):
    ./img2sdat.py <system_img>​

    - <system_img> = name of input sparse image file (from step 3)

    As you can see the output is composed by system.transfer.list, (system.patch.dat) & system.new.dat, ready to be replaced inside your Rom zip.




    DOWNLOADs
    sdat2img.py
    - github.com
    make_ext4fs
    - mega.co.nz
    img2sdat.py
    - github.com
    I will try these softwares to extract .dat files but isn't there any ANDROID APP WHICH CAN do this?
     

    dgm_mixtoon

    Senior Member
    Dec 23, 2014
    199
    27
    28
    Santo Domingo
    Introduction
    You probably know already that starting from Android 5.x (Lollipop) compiled roms (aosp,cm,stock) are not compressed anymore the way they used to be on previous android versions. On previous versions all content inside /system folder that has to be extracted within our device was either uncompressed (simple /system folder inside our flashable zip) or compressed in a system.img file, which it is a ext4 compressed file; both of these, anyway, were readable and we could see all system files (app,framework, etc).
    The problem comes in >=5.0 versions, this method is not used anymore. Why? Because roms started to be always larger, so it is necessary to compress them even more.


    What does new Android zips (full roms, but also otas) contain?
    New Android flashable zips are made this way:
    boot.img (kernel)​
    file_contexts (selinux related)​
    META-INF (folder containing scripts)​
    system.new.dat (compressed /system partition)​
    system.patch.dat (for OTAs)​
    system.transfer.list (see explanation below)​


    What does updater-script contains then?
    The updater-script uses a brand new function: block_image_update(), this method basically decompresses necessary files inside the device. Let's study it.
    From google git source code, if we go inside the new file /bootable/recovery/updater/blockimg.c, we find at the end of it the registration of the function block_image_update() as the method BlockImageUpdateFn() which starts at line 254. Here finally we find all information we need to know about the decompression of the .dat file(s). First file we analyze is system.transfer.list which Google tells us:




    But what each line means?:



    Ok, but how to decompress the system.new.dat file?
    All instructions are below. sdat2img, rimg2sdat and img2sdat binaries are involved. Please read carefully step by step.


    NOTE for OTAs
    If you are looking on decompressing system.patch.dat file or .p files, therefore reproduce the patching system on your PC, check imgpatchtools out by @erfanoabdi.


    You can use/modify these files and/or include them in your work as long as proper credits and a link to this thread are given.

    If you have questions or problems write here ;)


    Are you building from source and want to revert to the old flashing method without DAT files?
    Do this (thx @Elluel):
    1 - Delete this line
    2 - Make sure you have this commit


    Thanks
    - howellzhu & luxi78 - for initial source codes
    - all contributors from github
    hello bro, with your tool i can delete magisk from pre-rooted rom.zip?
     

    dgm_mixtoon

    Senior Member
    Dec 23, 2014
    199
    27
    28
    Santo Domingo
    Well magisk is systemless, so you need to focus on the boot.img. if you do it right, the answer is yes
    Unroot forever! Use an image of the same version of android (boot.img) I'm happy, I'm testing it, what consequences can that bring? Is what I did wrong?
    by the way I managed to unpack the rooted .img file and I see several folders with magisk, if I delete them and repackage it working?
    sorry I'm a newbie at this
     

    MPK99

    Senior Member
  • Jun 25, 2017
    329
    130
    @xpirt Hi sir... A small mistake in the img2sdat python script...
    It works only on the image which is named "system", and doesn't work on vendor, product, etc...
    I had to rename them to system, apply the script, get the dat files, then rename them again to their original names, which is a painfully slower task...
    Can you update the script so that it works for any file name?
    Also you might consider updating other scripts that have the same code.
     

    Top Liked Posts

    • There are no posts matching your filters.
    • 1
      hello bro, with your tool i can delete magisk from pre-rooted rom.zip?
      Well magisk is systemless, so you need to focus on the boot.img. if you do it right, the answer is yes
    • 319
      Introduction
      You probably know already that starting from Android 5.x (Lollipop) compiled roms (aosp,cm,stock) are not compressed anymore the way they used to be on previous android versions. On previous versions all content inside /system folder that has to be extracted within our device was either uncompressed (simple /system folder inside our flashable zip) or compressed in a system.img file, which it is a ext4 compressed file; both of these, anyway, were readable and we could see all system files (app,framework, etc).
      The problem comes in >=5.0 versions, this method is not used anymore. Why? Because roms started to be always larger, so it is necessary to compress them even more.


      What does new Android zips (full roms, but also otas) contain?
      New Android flashable zips are made this way:
      boot.img (kernel)
      file_contexts (selinux related)
      META-INF (folder containing scripts)
      system.new.dat (compressed /system partition)
      system.patch.dat (for OTAs)
      system.transfer.list (see explanation below)​


      What does updater-script contains then?
      The updater-script uses a brand new function: block_image_update(), this method basically decompresses necessary files inside the device. Let's study it.
      From google git source code, if we go inside the new file /bootable/recovery/updater/blockimg.c, we find at the end of it the registration of the function block_image_update() as the method BlockImageUpdateFn() which starts at line 254. Here finally we find all information we need to know about the decompression of the .dat file(s). First file we analyze is system.transfer.list which Google tells us:

      The transfer list is a text file containing commands to transfer data from one place to another on the target partition.


      But what each line means?:
      First line is the version number of the transfer list; 1 for android 5.0.x, 2 for android 5.1.x, 3 for android 6.0.x, 4 for android 7.x/8.x
      Second line is the total number of blocks being written
      Third line is how many stash entries are needed simultaneously (only on versions >= 2)
      Fourth line is the maximum number of blocks that will be stashed simultaneously (only on versions >= 2)
      Fifth line and subsequent lines are all individual transfer commands.


      Ok, but how to decompress the system.new.dat file?
      All instructions are below. sdat2img, rimg2sdat and img2sdat binaries are involved. Please read carefully step by step.


      NOTE for OTAs
      If you are looking on decompressing system.patch.dat file or .p files, therefore reproduce the patching system on your PC, check imgpatchtools out by @erfanoabdi.


      You can use/modify these files and/or include them in your work as long as proper credits and a link to this thread are given.

      If you have questions or problems write here ;)


      Are you building from source and want to revert to the old flashing method without DAT files?
      Do this (thx @Elluel):
      1 - Delete this line
      2 - Make sure you have this commit


      Thanks
      - howellzhu & luxi78 - for initial source codes
      - all contributors from github
      140
      Ubuntu

      Ubuntu 64-bit - Guide

      Step 1 - Decompressing = DAT (sparse data) -> EXT4 (raw image)

      We're now using sdat2img binary, the usage is very simple (make sure you have python 2.7+ installed):
      ./sdat2img.py <transfer_list> <system_new_file> [system_img]​
      - <transfer_list> = input, system.transfer.list from rom zip
      - <system_new_file> = input, system.new.dat from rom zip
      - [system_img] = output ext4 raw image file

      and a quick example:
      ./sdat2img.py system.transfer.list system.new.dat system.img​
      by running this command you will get as output the file my_new_system.img which is the raw ext4 image.


      Step 2 - Decompress EXT4 (raw image) -> OUTPUT folder -> Compress EXT4 (raw image)

      Now we need to mount or ext4 raw image into an output folder so we can see apks/jars etc.
      To do this we need to type this command:
      sudo mount -t ext4 -o loop system.img output/​
      As you can see there is a new folder called output which we can edit/modify/delete your files (not able to? see here)

      Now we need to compress it back to a raw ext4 image, to do this we need the make_ext4fs binary. Make sure you have the file_contexts file (taken from the Rom zip) inside the make_ext4fs path. Then type this (got issues? see here)
      ./make_ext4fs -T 0 -S file_contexts -l 1073741824 -a system system_new.img output/​
      You will get the new raw ext4 image called 'system_new.img' ready for the next step.

      Step 3 - Converting = EXT4 (raw image) -> IMG (sparse image)

      Now we need to convert the ext4 raw image into a sparse image. For this you need img2simg binary you can find here (thx to @A.S._id).
      The usage is simple:
      img2simg <raw_image_file> <sparse_image_file>​
      Pretty self-explanatory, the output will be a new sparse image (.img).


      Step 4 - Converting = IMG (sparse image) -> DAT (sparse data)

      Now we need the img2sdat binary, the usage is very simple (make sure you have python 2.7+ installed):
      ./img2sdat.py <system_img>​
      - <system_img> = name of input sparse image file (from step 3)

      As you can see the output is composed by system.transfer.list, (system.patch.dat) & system.new.dat, ready to be replaced inside your Rom zip.


      DOWNLOADs
      sdat2img.py
      - github.com
      make_ext4fs
      - mega.co.nz
      img2sdat.py
      - github.com
      126
      Guys, gathered a simple script, converts and extracts. In Russian and English languages. The main task of Extractor is only to unpack, without installing additional software on the PC. Only Windows 64\32bit ;)
      Follow the instructions...
      View attachment 4388717
      Update script v3.6 - Updated sdat2img (DATE: 2018-10-27 10:33:21 CEST)
      Update script v3.5 - Fix 3 settings item
      Update script v3.4 - Fix 10 settings item
      Update script v3.3 - Updated sdat2img (DATE: 2018-05-25 12:19:12 CEST)
      Update script v3.3 - Updated sdat2img (DATE: 2018-05-25 12:19:12 CEST)
      Update script v3.2 - Add supported vendor.new.dat.br
      Update script v3.1 - Repack sdat2img - https://www.virustotal.com/#/file/5...64c133a65233d7df5effc92fa8dd33f578b/detection
      Update script v3.0 - Add convert
      Update script v2.9 - Add support "system.new.dat.br"
      Update script v2.8 - Fixed compatibility with Windows 32bit
      Update script v2.7 - Updated sdat2img (DATE: 2017-08-25 16:20:11 CST)\add support Android 7.0-8.0
      Update script v2.6 - Updated sdat2img (DATE: 2016-11-23 16:20:11 CST)
      Update script v2.5 - Updated sdat2img (DATE: 2016-09-22 17:44:06 CST)
      Update script v2.4 - Updated sdat2img in latest github source
      Update script v2.2 - Updated sdat2img
      Update script v2.1 - Remove the extra folders, corrected description. add system.img.ext4
      Update script v2.0 - add support Android 6.0
      Update script v1.7 - fix select items
      Update script v1.6 - add system.img, recovery.img, boot.img
      98
      Windows

      Windows 32/64 bit - Guide

      Step 1 - Decompressing = DAT (sparse data) -> EXT4 (raw image)

      We're now using sdat2img.py binary, the usage is very simple (make sure you have python 3.x installed):
      sdat2img.py <transfer_list> <system_new_file> <system_ext4>​
      - <transfer_list> = input, system.transfer.list from rom zip
      - <system_new_file> = input, system.new.dat from rom zip
      - <system_ext4> = output ext4 raw image file

      and a quick example:
      sdat2img.py system.transfer.list system.new.dat system.img​
      by running this command you will get as output the file my_new_system.img which is the raw ext4 image.


      Step 2 - Decompress EXT4 (raw image) -> OUTPUT folder -> Compress EXT4 (raw image)

      Now you should either move to Ubuntu x64/x86 or follow the 2nd step from the Ubuntu guide on cygwin 32 bit.

      Step 3 - Converting = EXT4 (raw image) -> IMG (sparse image)

      Now we need to convert the ext4 raw image into a sparse image. For this you need img2simg binary you can find here (thx to @A.S._id).
      The usage is simple:
      img2simg.exe <raw_image_file> <sparse_image_file>​
      Pretty self-explanatory, the output will be a new sparse image (.img).


      Step 4 - Converting = IMG (sparse image) -> DAT (sparse data)

      Now we need the img2sdat binary, the usage is very simple (make sure you have python 2.7+ installed):
      img2sdat.py <system_img>​
      - <system_img> = name of input sparse image file (from step 3)

      As you can see the output is composed by system.transfer.list, (system.patch.dat) & system.new.dat, ready to be replaced inside your Rom zip.


      DOWNLOADs
      sdat2img.py
      - github.com
      make_ext4fs
      - mega.co.nz
      img2sdat.py
      - github.com
    Our Apps
    Get our official app!
    The best way to access XDA on your phone
    Nav Gestures
    Add swipe gestures to any Android
    One Handed Mode
    Eases uses one hand with your phone