[Tool] SparseConverter v1.0.1

Search This thread

tal.aloni

Member
May 12, 2014
46
103
About:
fastboot stores each file it receives in RAM before flashing it to your Android device.
For this reason, large partitions (such as the /system partition) must be splitted using the "compressed ext4 file system sparse image format",
this format is defined by the AOSP (Android Open Source Project) and was designed for the single purpose of flashing a large partition.

If you tried to flash system.img from backup and received the "Invalid sparse file format at header magi" error, you have come to the right place.

The Solution:
SparseConverter is a tool that can create / decompress compressed ext4 file system sparse image format (e.g. system.img_sparsechunk1).

Note for Motorola phone owners:
The factory images from Motorola contains a 128KB motorola header and a 4KB motorola footer, if you decompress those images you may want to remove the header and footer.
I had no problem using the standard Android fastboot with my unlocked Moto G and flashing images without the Motorola header / footer.
(If you keep the header / footer, then you must use Motorola's fastboot)


Usage Examples:
SparseConverter.exe /compress D:\system.img E:\ 256MB
( will compress D:\system.img to 256MB sparse files starting from E:\system.img_sparsechunk1 )
SparseConverter.exe /decompress E:\system.img_sparsechunk1 D:\system.img
( will decompress E:\system.img_sparsechunk1, E:\system.img_sparsechunk2 and etc. to D:\system.img )

The software may contain bugs and/or limitations that may result in damage to your phone, I take no responsibility for any damage that may occur.

For additional information about the "compressed ext4 file system sparse image format" see libsparse/sparse_format.h

p.s. Source code in C# is provided, Windows XP \ 2003 users must install the .Net Framework 2.0 before running this tool.
 

Attachments

  • SparseConverter_1.0.1.zip
    8.9 KB · Views: 18,008
  • SparseConverter_1.0.1_Source.zip
    19.3 KB · Views: 5,420
Last edited:

tal.aloni

Member
May 12, 2014
46
103
Everything is explained in the original post, You need to use fastboot.exe which is part of the ADT bundle (Android Development Tools).
 

tal.aloni

Member
May 12, 2014
46
103
yes,
fastboot flash system system.img_sparsechunk1
fastboot flash system system.img_sparsechunk2
fastboot flash system system.img_sparsechunk3

make sure the system partition size match the image that you're trying to flash, to flash a bigger image you'll have to reflash the partition table and all the partitions (boot, recovery etc.)
p.s. not sure what "not working" means, what error do you get exactly?
 

gouster3

Senior Member
Oct 22, 2014
298
107
Levice
yes,
fastboot flash system system.img_sparsechunk1
fastboot flash system system.img_sparsechunk2
fastboot flash system system.img_sparsechunk3

make sure the system partition size match the image that you're trying to flash, to flash a bigger image you'll have to reflash the partition table and all the partitions (boot, recovery etc.)
p.s. not sure what "not working" means, what error do you get exactly?

Here is my error code, after attemp to flash system system.img_sparechunk1:
FAILED (remote: flash_cmds error!)

partition size match, it is system dump from another same device.
 

dark_samus

Senior Member
Dec 11, 2012
51
37
Hi, I'm trying to convert a userdata.img file (60GB!) so I can flash it with fastboot onto my phone, when I run the tool I get this
Code:
C:\Users\Matt\Documents\SparseConverter_1.0.0>sparseconverter /compress C:\Users
\Matt\Documents\userdata.img "C:\Users\Matt\Documents\userdata sparse" 256MB
Image size is not a multiple of 4096 bytes

Does this mean that it won't convert my image? Thanks

EDIT: Also I know this image is valid as under linux

Code:
[email protected]:~$ sudo mount -t ext4 -o loop /home/matt/userdata.img /home/matt/ext4
[sudo] password for matt:
[email protected]:~$ sudo su
[email protected]:/home/matt# cd ext4
[email protected]:/home/matt/ext4# ls
anr          bugreports    dontpanic   media        resource-cache  user
app          cam_socket1   dpm         mediadrm     security
app-asec     cam_socket2   drm         misc         shared
app-lib      connectivity  fdAlbum     nfc          system
app-private  dalvik-cache  local       opponvitems  time
backup       data          lost+found  property     tombstones
 
Last edited:

tal.aloni

Member
May 12, 2014
46
103
Hi Matt,
ext4 allocates storage space in units of "blocks". A block is a group of sectors between 1KiB and 64KiB, and the number of sectors must be an integral power of 2. Block size is specified at mkfs time and typically is 4KiB.
The sector size for most flash devices is 4096 bytes.
The "compressed ext4 file system sparse image format" assumes that the image is going to be flashed to a flash device, and thus expects its size to be a multiple of 4096 bytes.
I'm not sure how you created this user data partition image, the size of a proper image (for flashing purposes) should be a multiple of 4096 bytes.

Edit: BlockSize can be set to 512 instead of 4096 in SparseConverter source code (SparseCompressionHelper.cs).
 
Last edited:

hazard666

Senior Member
Jul 19, 2010
358
22
Ohio
Hi Matt,

The sector size for most flash devices is 4096 bytes.
The "compressed ext4 file system sparse image format" assumes that the image is going to be flashed to a flash device, and thus expects its size to be a multiple of 4096 bytes.
I'm not sure how you created this user data partition image, the size of a proper image (for flashing purposes) should be a multiple of 4096 bytes.

I ran into this issue as well. I am trying to compress the userdata image from my phone that was previously bricked and I'd like to find a way to hold onto this data.
Thank you!
 

tal.aloni

Member
May 12, 2014
46
103
I ran into this issue as well. I am trying to compress the userdata image from my phone that was previously bricked and I'd like to find a way to hold onto this data.
Thank you!

How did you create this image? are you sure it's a raw image?
if it's indeed a raw image, you could append a the necessary amount of zero bytes to the image (e.g. using HxD) and then you could compress it.
I'm not sure what you mean by "hold onto this data", this tool is to prepare an image for flashing, if you're trying to access the files there are other methods (e.g. linux).
 

juliospinoza

Senior Member
Jan 9, 2012
770
323
Tepic
Awesome, I use this tool to compress system file of Motorola X2014 of 2.7GB that I was not able to flash it... works great, thank you so much
 

Azlun

Senior Member
May 9, 2013
340
217
Shenzhen
Great progress! In my case
I generate my system.img.raw by
adb root
adb wait-for-device
adb shell "dd if=/dev/block/platform/f9824900.sdhci/by-name/system of=/sdcard/system.img.raw"
adb pull -p /sdcard/system.img.raw

1st Trail below will NOT works, but at the end you will see my solution:
Windows cmd log:
E:\AzlunData\DEV Tools\ImageBakupsCMD\SYS GMS+Root+AzAppsMDP>sparseconverter /compress system.img.raw .\ 536870912
...
pause
E:\AzlunData\DEV Tools\ImageBakupsCMD\SYS GMS+Root+AzAppsMDP>fastboot flash system system.img.raw_
sparsechunk1
target reported max download size of 536870912 bytes
erasing 'system'...
OKAY [ 0.591s]
sending 'system' (524284 KB)...
OKAY [ 1.350s]
writing 'system'...
OKAY [ 13.953s]
finished. total time: 15.897s

E:\AzlunData\DEV Tools\ImageBakupsCMD\SYS GMS+Root+AzAppsMDP>fastboot flash system system.img.raw_
sparsechunk2
target reported max download size of 536870912 bytes
erasing 'system'...
OKAY [ 0.532s]
sending 'system' (524287 KB)...
OKAY [ 1.351s]
writing 'system'...
OKAY [ 8.678s]
finished. total time: 10.565s

E:\AzlunData\DEV Tools\ImageBakupsCMD\SYS GMS+Root+AzAppsMDP>fastboot flash system system.img.raw_
sparsechunk3
target reported max download size of 536870912 bytes
erasing 'system'...
OKAY [ 0.524s]
sending 'system' (411515 KB)...
OKAY [ 1.075s]
writing 'system'...
OKAY [123.069s]
finished. total time: 124.671s

Please note even we pass the fastboot flashing, but it does NOT works after flash is finished, the phone can reboot to Linux logo, but block there, can not start Android Animation and go to desktop.
E:\AzlunData\DEV Tools\ImageBakupsCMD\SYS GMS+Root+AzAppsMDP>adb shell
- exec '/system/bin/sh' failed: No such file or directory (2) -

SO, SOLUTION is:
as an conclusion, it seems fastboot can not manage the multi input images well, BUT it can handle a single compressed image files, so set the max image side to the partition size to 2684354560 can make fastboot works well.
I need to find my original dd output file system.img.raw in 2.5GB, then:
Windows cmd logs:

E:\AzlunData\DEV Tools\ImageBakupsCMD\SYS GMS+Root+AzAppsMDP>sparseconverter /compress system.img.raw .\ 2684354560

E:\AzlunData\DEV Tools\ImageBakupsCMD\SYS GMS+Root+AzAppsMDP>fastboot flash system system.img.raw_sp
arsechunk1
target reported max download size of 536870912 bytes
erasing 'system'...
OKAY [ 0.767s]
sending sparse 'system' (521371 KB)...
OKAY [ 4.530s]
writing 'system'...
OKAY [ 9.577s]
sending sparse 'system' (511116 KB)...
OKAY [ 6.465s]
writing 'system'...
OKAY [ 10.833s]
sending sparse 'system' (445892 KB)...
OKAY [ 6.274s]
writing 'system'...
OKAY [139.377s]
finished. total time: 177.831s

that will bring my system.img.raw back to my phone WELL
you can see my rooted su is there, ^_^:
E:\AzlunData\DEV Tools\ImageBakupsCMD\SYS GMS+Root+AzAppsMDP>adb shell
[email protected]:/ $ su
su
[email protected]:/ #
 
Last edited:

tal.aloni

Member
May 12, 2014
46
103
Matt, hazard666,
I was wrong, there is no restriction on BlockSize , you're welcome to download SparseConverter source code and set BlockSize to 512 (in SparseCompressionHelper.cs),
please let me know if this helps.
 

JulesJam

Senior Member
Aug 9, 2014
2,468
773
Note for Motorola phone owners:
The factory images from Motorola contains a 128KB motorola header and a 4KB motorola footer, if you decompress those images you may want to remove the header and footer.
I had no problem using the standard Android fastboot with my unlocked Moto G and flashing images without the Motorola header / footer.
(If you keep the header / footer, then you must use Motorola's fastboot)
Hi, I used your tool to decompress sparsechunks into a single system image for the Moto X 2014 XT1097, AT&T. This is needed as a preliminary step in order to get root.

The next step is to convert the image into an ext4 format. However, the instructions we have to do that require using linux command simg2img and it doesn't like the header. The errors are:

"Invalid sparse file format at header magi

Failed to read sparse file"

"Bad magic"

or

"Error reading sparse file header"


How do we get rid of this header? Thanks!
 

tal.aloni

Member
May 12, 2014
46
103
How do we get rid of this header? Thanks!
I usually do it manually with HxD (hex editor).

The next step is to convert the image into an ext4 format. However, the instructions we have to do that require using linux command simg2img and it doesn't like the header.
I think you're confused, once the image is decompressed, it's no longer a sparse file, you shouldn't use simg2img (to convert it to raw image, because it's already been done).
instead, you can just mount the image, for ext4 it would be:
Code:
sudo mount -t ext4 -o loop,offset=131072 system.img mnt/
and when you're done customizing:
Code:
sudo ./make_ext4fs -l 840M -a system mysystem.img mnt/
sudo umount mnt

(note that I've used offset=131072 instead of removing the header)
 
Last edited:
  • Like
Reactions: niko99

JulesJam

Senior Member
Aug 9, 2014
2,468
773
I think you're confused, once the image is decompressed, it's no longer a sparse file, so you shouldn't use simg2img.
Yeah, I am definitely confused, lol!

instead, use something along the line of (note that I've used offset=131072 instead of removing the header):
Code:
sudo mount -t ext4 -o loop,offset=131072 system.img mnt/
sudo ./make_ext4fs -l 840M -a system mysystem.img mnt/
sudo umount mnt
Ok great, I will have someone who knows linux try that. What we want is a single file system image in ext4 format so hopefully that will work!

Thanks!
 
  • Like
Reactions: niko99

G4 Man2

Senior Member
May 4, 2011
883
286
Coffeyville
I usually do it manually with HxD (hex editor).


I think you're confused, once the image is decompressed, it's no longer a sparse file, you shouldn't use simg2img (to convert it to raw image, because it's already been done).
instead, you can just mount the image, for ext4 it would be:
Code:
sudo mount -t ext4 -o loop,offset=131072 system.img mnt/
and when you're done customizing:
Code:
sudo ./make_ext4fs -l 840M -a system mysystem.img mnt/
sudo umount mnt

(note that I've used offset=131072 instead of removing the header)

Yeah, I am definitely confused, lol!

Ok great, I will have someone who knows linux try that. What we want is a single file system image in ext4 format so hopefully that will work!

Thanks!

And I'm someone who's running a Linux distro!

After the second command I'm getting a memory allocation error though
"error: do_inode_allocate_extents: Failed to allocate 6524 blocks"
 

JulesJam

Senior Member
Aug 9, 2014
2,468
773
I think you're confused, once the image is decompressed, it's no longer a sparse file, you shouldn't use simg2img (to convert it to raw image, because it's already been done).
Also, just to be complete, the way I am going to modify/root the ext4 image is by using TWRP on a bootloader unlocked device according to these instructions:

http://forum.xda-developers.com/showpost.php?p=60389310&postcount=177


It worked for the XT1096 so it should work for the XT1097. Once we get the rooted ext4 image, we have a tool to flash it to the XT1097 that bypasses the signature check. Our bootloaders are locked so we have to do it this way.
 
Last edited:

Top Liked Posts

  • There are no posts matching your filters.
  • 28
    About:
    fastboot stores each file it receives in RAM before flashing it to your Android device.
    For this reason, large partitions (such as the /system partition) must be splitted using the "compressed ext4 file system sparse image format",
    this format is defined by the AOSP (Android Open Source Project) and was designed for the single purpose of flashing a large partition.

    If you tried to flash system.img from backup and received the "Invalid sparse file format at header magi" error, you have come to the right place.

    The Solution:
    SparseConverter is a tool that can create / decompress compressed ext4 file system sparse image format (e.g. system.img_sparsechunk1).

    Note for Motorola phone owners:
    The factory images from Motorola contains a 128KB motorola header and a 4KB motorola footer, if you decompress those images you may want to remove the header and footer.
    I had no problem using the standard Android fastboot with my unlocked Moto G and flashing images without the Motorola header / footer.
    (If you keep the header / footer, then you must use Motorola's fastboot)


    Usage Examples:
    SparseConverter.exe /compress D:\system.img E:\ 256MB
    ( will compress D:\system.img to 256MB sparse files starting from E:\system.img_sparsechunk1 )
    SparseConverter.exe /decompress E:\system.img_sparsechunk1 D:\system.img
    ( will decompress E:\system.img_sparsechunk1, E:\system.img_sparsechunk2 and etc. to D:\system.img )

    The software may contain bugs and/or limitations that may result in damage to your phone, I take no responsibility for any damage that may occur.

    For additional information about the "compressed ext4 file system sparse image format" see libsparse/sparse_format.h

    p.s. Source code in C# is provided, Windows XP \ 2003 users must install the .Net Framework 2.0 before running this tool.
    3
    I think I figured it out. thanks!

    FYI - for anyone who wants a step-by-step on how to use a hex editor on a motorola image to remove the header and the footer:

    1) Download the hex editor here:

    http://mh-nexus.de/en/hxd/


    2) Click Edit-> Select Block and in the pop-up select dec at the bottom. Then put 0 in the Start-offset and 131072 in the length. Click OK to select the header. Then click Edit-> Delete to delete the header.

    3) At the top where it says hex, change it to dec using the pull down. Click Search-> GoTo and in the pop-up select dec and end (backwards) and in the offset put 4096. The blinking cursor will be on the last pair of the row ABOVE the one you want. Write down the number of the row right below where the blinking cursor is.

    4) Click Edit -> Select Block and in the pop-up select dec, in Start offset put the row number from step #3 and in Length type in 4096. Then click OK then click Edit-> Delete to delete the footer.

    5) Save the file.
    1
    How do we get rid of this header? Thanks!
    I usually do it manually with HxD (hex editor).

    The next step is to convert the image into an ext4 format. However, the instructions we have to do that require using linux command simg2img and it doesn't like the header.
    I think you're confused, once the image is decompressed, it's no longer a sparse file, you shouldn't use simg2img (to convert it to raw image, because it's already been done).
    instead, you can just mount the image, for ext4 it would be:
    Code:
    sudo mount -t ext4 -o loop,offset=131072 system.img mnt/
    and when you're done customizing:
    Code:
    sudo ./make_ext4fs -l 840M -a system mysystem.img mnt/
    sudo umount mnt

    (note that I've used offset=131072 instead of removing the header)
    1
    I think you're confused, once the image is decompressed, it's no longer a sparse file, so you shouldn't use simg2img.
    Yeah, I am definitely confused, lol!

    instead, use something along the line of (note that I've used offset=131072 instead of removing the header):
    Code:
    sudo mount -t ext4 -o loop,offset=131072 system.img mnt/
    sudo ./make_ext4fs -l 840M -a system mysystem.img mnt/
    sudo umount mnt
    Ok great, I will have someone who knows linux try that. What we want is a single file system image in ext4 format so hopefully that will work!

    Thanks!
    1
    Usage Examples:
    SparseConverter.exe /compress D:\system.img E:\ 256MB
    ( will compress D:\system.img to 256MB sparse files starting from E:\system.img_sparsechunk1 )