[Tool] SparseConverter v1.0.1

[tal.aloni]

About:
fastboot stores each file it receives in RAM before flashing it to your Android device.
For this reason, large partitions (such as the /system partition) must be splitted using the "compressed ext4 file system sparse image format",
this format is defined by the AOSP (Android Open Source Project) and was designed for the single purpose of flashing a large partition.

If you tried to flash system.img from backup and received the "Invalid sparse file format at header magi" error, you have come to the right place.

The Solution:
SparseConverter is a tool that can create / decompress compressed ext4 file system sparse image format (e.g. system.img_sparsechunk1).

Note for Motorola phone owners:
The factory images from Motorola contains a 128KB motorola header and a 4KB motorola footer, if you decompress those images you may want to remove the header and footer.
I had no problem using the standard Android fastboot with my unlocked Moto G and flashing images without the Motorola header / footer.
(If you keep the header / footer, then you must use Motorola's fastboot)


Usage Examples:
SparseConverter.exe /compress D:\system.img E:\ 256MB
( will compress D:\system.img to 256MB sparse files starting from E:\system.img_sparsechunk1 )
SparseConverter.exe /decompress E:\system.img_sparsechunk1 D:\system.img
( will decompress E:\system.img_sparsechunk1, E:\system.img_sparsechunk2 and etc. to D:\system.img )

The software may contain bugs and/or limitations that may result in damage to your phone, I take no responsibility for any damage that may occur.

For additional information about the "compressed ext4 file system sparse image format" see libsparse/sparse_format.h

p.s. Source code in C# is provided, Windows XP \ 2003 users must install the .Net Framework 2.0 before running this tool.

 

[gouster3]

How can i flash system.img when is too large. i have compressed it using this tool, now how to flash?

 

[tal.aloni]

Everything is explained in the original post, You need to use fastboot.exe which is part of the ADT bundle (Android Development Tools).

 

[gouster3]

Everything is explained in the original post, You need to use fastboot.exe which is part of the ADT bundle (Android Development Tools).

fastboot flash system system.img_sparsechunk1 ?
Not working for me. :/

It will be downloaded to device, but not flashed.

 

[tal.aloni]

yes,
fastboot flash system system.img_sparsechunk1
fastboot flash system system.img_sparsechunk2
fastboot flash system system.img_sparsechunk3

make sure the system partition size match the image that you're trying to flash, to flash a bigger image you'll have to reflash the partition table and all the partitions (boot, recovery etc.)
p.s. not sure what "not working" means, what error do you get exactly?

 

[gouster3]

yes,
fastboot flash system system.img_sparsechunk1
fastboot flash system system.img_sparsechunk2
fastboot flash system system.img_sparsechunk3

make sure the system partition size match the image that you're trying to flash, to flash a bigger image you'll have to reflash the partition table and all the partitions (boot, recovery etc.)
p.s. not sure what "not working" means, what error do you get exactly?

Here is my error code, after attemp to flash system system.img_sparechunk1:
FAILED (remote: flash_cmds error!)

partition size match, it is system dump from another same device.

 

[tal.aloni]

Sorry, I'm not familiar with this error.

 

[gouster3]

Sorry, I'm not familiar with this error.

I have Rooted my ASUS Tablet, and deleted something in system, now i am soft-bricked.
Now i cannot do somethnig, only use fastboot, butt i cannot flash system.

So i must waiting to developers, to port CWM/TWRP for my device.

 

[dark_samus]

Hi, I'm trying to convert a userdata.img file (60GB!) so I can flash it with fastboot onto my phone, when I run the tool I get this

C:\Users\Matt\Documents\SparseConverter_1.0.0>sparseconverter /compress C:\Users
\Matt\Documents\userdata.img "C:\Users\Matt\Documents\userdata sparse" 256MB
Image size is not a multiple of 4096 bytes

Does this mean that it won't convert my image? Thanks

EDIT: Also I know this image is valid as under linux

[email protected]:~$ sudo mount -t ext4 -o loop /home/matt/userdata.img /home/matt/ext4
[sudo] password for matt:
[email protected]:~$ sudo su
[email protected]:/home/matt# cd ext4
[email protected]:/home/matt/ext4# ls
anr          bugreports    dontpanic   media        resource-cache  user
app          cam_socket1   dpm         mediadrm     security
app-asec     cam_socket2   drm         misc         shared
app-lib      connectivity  fdAlbum     nfc          system
app-private  dalvik-cache  local       opponvitems  time
backup       data          lost+found  property     tombstones

 

[tal.aloni]

Hi Matt,

ext4 allocates storage space in units of "blocks". A block is a group of sectors between 1KiB and 64KiB, and the number of sectors must be an integral power of 2. Block size is specified at mkfs time and typically is 4KiB.

The sector size for most flash devices is 4096 bytes.
The "compressed ext4 file system sparse image format" assumes that the image is going to be flashed to a flash device, and thus expects its size to be a multiple of 4096 bytes.
I'm not sure how you created this user data partition image, the size of a proper image (for flashing purposes) should be a multiple of 4096 bytes.
Edit: BlockSize can be set to 512 instead of 4096 in SparseConverter source code (SparseCompressionHelper.cs).

 

[hazard666]

Hi Matt,

The sector size for most flash devices is 4096 bytes.
The "compressed ext4 file system sparse image format" assumes that the image is going to be flashed to a flash device, and thus expects its size to be a multiple of 4096 bytes.
I'm not sure how you created this user data partition image, the size of a proper image (for flashing purposes) should be a multiple of 4096 bytes.

I ran into this issue as well. I am trying to compress the userdata image from my phone that was previously bricked and I'd like to find a way to hold onto this data.
Thank you!

 

[tal.aloni]

I ran into this issue as well. I am trying to compress the userdata image from my phone that was previously bricked and I'd like to find a way to hold onto this data.
Thank you!

How did you create this image? are you sure it's a raw image?
if it's indeed a raw image, you could append a the necessary amount of zero bytes to the image (e.g. using HxD) and then you could compress it.
I'm not sure what you mean by "hold onto this data", this tool is to prepare an image for flashing, if you're trying to access the files there are other methods (e.g. linux).

 

[juliospinoza]

Awesome, I use this tool to compress system file of Motorola X2014 of 2.7GB that I was not able to flash it... works great, thank you so much

 

[Azlun]

Great progress! In my case
I generate my system.img.raw by
adb root
adb wait-for-device
adb shell "dd if=/dev/block/platform/f9824900.sdhci/by-name/system of=/sdcard/system.img.raw"
adb pull -p /sdcard/system.img.raw

1st Trail below will NOT works, but at the end you will see my solution:
Windows cmd log:
E:\AzlunData\DEV Tools\ImageBakupsCMD\SYS GMS+Root+AzAppsMDP>sparseconverter /compress system.img.raw .\ 536870912
...
pause
E:\AzlunData\DEV Tools\ImageBakupsCMD\SYS GMS+Root+AzAppsMDP>fastboot flash system system.img.raw_
sparsechunk1
target reported max download size of 536870912 bytes
erasing 'system'...
OKAY [ 0.591s]
sending 'system' (524284 KB)...
OKAY [ 1.350s]
writing 'system'...
OKAY [ 13.953s]
finished. total time: 15.897s

E:\AzlunData\DEV Tools\ImageBakupsCMD\SYS GMS+Root+AzAppsMDP>fastboot flash system system.img.raw_
sparsechunk2
target reported max download size of 536870912 bytes
erasing 'system'...
OKAY [ 0.532s]
sending 'system' (524287 KB)...
OKAY [ 1.351s]
writing 'system'...
OKAY [ 8.678s]
finished. total time: 10.565s

E:\AzlunData\DEV Tools\ImageBakupsCMD\SYS GMS+Root+AzAppsMDP>fastboot flash system system.img.raw_
sparsechunk3
target reported max download size of 536870912 bytes
erasing 'system'...
OKAY [ 0.524s]
sending 'system' (411515 KB)...
OKAY [ 1.075s]
writing 'system'...
OKAY [123.069s]
finished. total time: 124.671s

Please note even we pass the fastboot flashing, but it does NOT works after flash is finished, the phone can reboot to Linux logo, but block there, can not start Android Animation and go to desktop.
E:\AzlunData\DEV Tools\ImageBakupsCMD\SYS GMS+Root+AzAppsMDP>adb shell
- exec '/system/bin/sh' failed: No such file or directory (2) -

SO, SOLUTION is:
as an conclusion, it seems fastboot can not manage the multi input images well, BUT it can handle a single compressed image files, so set the max image side to the partition size to 2684354560 can make fastboot works well.
I need to find my original dd output file system.img.raw in 2.5GB, then:
Windows cmd logs:

E:\AzlunData\DEV Tools\ImageBakupsCMD\SYS GMS+Root+AzAppsMDP>sparseconverter /compress system.img.raw .\ 2684354560

E:\AzlunData\DEV Tools\ImageBakupsCMD\SYS GMS+Root+AzAppsMDP>fastboot flash system system.img.raw_sp
arsechunk1
target reported max download size of 536870912 bytes
erasing 'system'...
OKAY [ 0.767s]
sending sparse 'system' (521371 KB)...
OKAY [ 4.530s]
writing 'system'...
OKAY [ 9.577s]
sending sparse 'system' (511116 KB)...
OKAY [ 6.465s]
writing 'system'...
OKAY [ 10.833s]
sending sparse 'system' (445892 KB)...
OKAY [ 6.274s]
writing 'system'...
OKAY [139.377s]
finished. total time: 177.831s

that will bring my system.img.raw back to my phone WELL
you can see my rooted su is there, ^_^:
E:\AzlunData\DEV Tools\ImageBakupsCMD\SYS GMS+Root+AzAppsMDP>adb shell
[email protected]:/ $ su
su
[email protected]:/ #

 

[tal.aloni]

Matt, hazard666,
I was wrong, there is no restriction on BlockSize , you're welcome to download SparseConverter source code and set BlockSize to 512 (in SparseCompressionHelper.cs),
please let me know if this helps.

 

[JulesJam]

Note for Motorola phone owners:
The factory images from Motorola contains a 128KB motorola header and a 4KB motorola footer, if you decompress those images you may want to remove the header and footer.
I had no problem using the standard Android fastboot with my unlocked Moto G and flashing images without the Motorola header / footer.
(If you keep the header / footer, then you must use Motorola's fastboot)

Hi, I used your tool to decompress sparsechunks into a single system image for the Moto X 2014 XT1097, AT&T. This is needed as a preliminary step in order to get root.

The next step is to convert the image into an ext4 format. However, the instructions we have to do that require using linux command simg2img and it doesn't like the header. The errors are:

"Invalid sparse file format at header magi

Failed to read sparse file"

"Bad magic"

or

"Error reading sparse file header"


How do we get rid of this header? Thanks!

 

[tal.aloni]

How do we get rid of this header? Thanks!

I usually do it manually with HxD (hex editor).

The next step is to convert the image into an ext4 format. However, the instructions we have to do that require using linux command simg2img and it doesn't like the header.

I think you're confused, once the image is decompressed, it's no longer a sparse file, you shouldn't use simg2img (to convert it to raw image, because it's already been done).
instead, you can just mount the image, for ext4 it would be:

sudo mount -t ext4 -o loop,offset=131072 system.img mnt/

and when you're done customizing:

sudo ./make_ext4fs -l 840M -a system mysystem.img mnt/
sudo umount mnt

(note that I've used offset=131072 instead of removing the header)

 

[JulesJam]

I think you're confused, once the image is decompressed, it's no longer a sparse file, so you shouldn't use simg2img.

Yeah, I am definitely confused, lol!

instead, use something along the line of (note that I've used offset=131072 instead of removing the header):

sudo mount -t ext4 -o loop,offset=131072 system.img mnt/
sudo ./make_ext4fs -l 840M -a system mysystem.img mnt/
sudo umount mnt

Ok great, I will have someone who knows linux try that. What we want is a single file system image in ext4 format so hopefully that will work!

Thanks!

 

[G4 Man2]

I usually do it manually with HxD (hex editor).


I think you're confused, once the image is decompressed, it's no longer a sparse file, you shouldn't use simg2img (to convert it to raw image, because it's already been done).
instead, you can just mount the image, for ext4 it would be:

sudo mount -t ext4 -o loop,offset=131072 system.img mnt/

and when you're done customizing:

sudo ./make_ext4fs -l 840M -a system mysystem.img mnt/
sudo umount mnt

(note that I've used offset=131072 instead of removing the header)

Yeah, I am definitely confused, lol!

Ok great, I will have someone who knows linux try that. What we want is a single file system image in ext4 format so hopefully that will work!

Thanks!

And I'm someone who's running a Linux distro!

After the second command I'm getting a memory allocation error though
"error: do_inode_allocate_extents: Failed to allocate 6524 blocks"

 

[JulesJam]

I think you're confused, once the image is decompressed, it's no longer a sparse file, you shouldn't use simg2img (to convert it to raw image, because it's already been done).

Also, just to be complete, the way I am going to modify/root the ext4 image is by using TWRP on a bootloader unlocked device according to these instructions:

http://forum.xda-developers.com/showpost.php?p=60389310&postcount=177


It worked for the XT1096 so it should work for the XT1097. Once we get the rooted ext4 image, we have a tool to flash it to the XT1097 that bypasses the signature check. Our bootloaders are locked so we have to do it this way.