FORUMS
Remove All Ads from XDA
Honor View 10

How to backup compressed data from your android device to your computer

1,016 posts
Thanks Meter: 1,832
 
Post Reply Email Thread
Disclaimer
Code:
Whatever you do, it's your decision and you do it on your own risk!
I am not responsible for bricks, any kind of damages, data loss or any other unwanted result!
Everything you do is done on YOUR OWN RESPONSIBILITY!
And I am also not responsible if you lose the chat with your crush.
You might want to transfer your internal storage such as /sdcard to your computer directly and compressed as fast as possible.
So here is how you do it (on linux using bash):

Reboot your phone into TWRP, connect your device to your computer, and run the following command:

Code:
adb shell mkfifo /bk.pipe && adb exec-out 'tar -cvap data/media/0 2>/bk.pipe' | dd of=sdcard.tar & adb shell cat /bk.pipe &
In this example your data (in this case /data/media/0) will be backed up to your computer into the file sdcard.tar.
Compression is done automatically and you will get a nice output on your terminal on which file is being backed up currently.

When everything is finished you should get a message similar to this on your terminal:

Code:
79692237+2 records in
79692238+0 records out
40802425856 bytes (41 GB, 38 GiB) copied, 3550,38 s, 11,5 MB/s
(On the result of this you can see that the speed is around 12 MB/s, means that 1 GB backups in less than 1 minute)

When you get that, it means that your backup has finished and you can safely remove the usb cable.

Note that you can do this with any directory on your device, so if you desire to back up /data, just write data instead of data/media/0.

Example of the resulting sdcard.tar file (opened on my computer):
Example result

Decompressing your data directly to your device might get a little trickier:

Code:
adb shell mkfifo /rst.pipe && adb push sdcard.tar /rst.pipe & adb shell 'cd /; cat /rst.pipe | tar -xv'
If that does not work, you can just extract the content of the tar file normally onto your computer and then copy the files over using adb push or copy the tar file to your phone and extract it using the following command:

Code:
adb push sdcard.tar /sdcard/sdcard.tar
adb shell 'cd /; tar -xvf /sdcard/sdcard.tar'
And that should do the job as well.

---

See also:

tar to pipe but keep -v verbose output separate from STDERR

Transferring binary data over ADB shell (ie fast file transfer using tar)

Compose and pipe a gzipped tar archive in one go

Bash tar and output to log file

GNU Tar

Encrypting and Compressing

How encrypt data/text stream instead of a file?

Post on my blog:

How to backup compressed data from your android device to your computer
The Following 34 Users Say Thank You to xdvs23 For This Useful Post: [ View ] Gift xdvs23 Ad-Free
 
 
19th September 2016, 11:03 PM |#2  
Wickii's Avatar
Senior Member
Thanks Meter: 296
 
Donate to Me
More
Is it possible to do this with windows cmd?
20th September 2016, 12:13 AM |#3  
Senior Member
Thanks Meter: 203
 
More
Hmm, that looks really complicated. How about rsync instead?

(Note: This requires root and it requires rsync... It's present on CyanogenMod and probably other custom roms, but usually not on stock ROMs...)

Code:
1. Create rsync.conf:
[root]
path = /
read only = no
uid = root
gid = root

2. Upload the file to the phone
$ adb push rsync.conf /sdcard/

3. Restart adbd with root permissions
$ adb root

4. Set up TCP forwarding of the rsync daemon through adb
$ adb forward tcp:9873 tcp:873

5. Run the daemon on the phone
$ adb shell /system/xbin/rsync --daemon --config /sdcard/rsync.conf --no-detach

6. Copy files in both directions
rsync -v rsync://localhost:9873/root/
rsync --help
Edit: You can also run rsync via SSH, e.g. with the help of SSHdroid:
Code:
rsync -e "ssh -l root -p 2222" -v <phone-ip>:/
The Following User Says Thank You to michael_ch For This Useful Post: [ View ] Gift michael_ch Ad-Free
20th September 2016, 01:06 AM |#4  
osm0sis's Avatar
Recognized Developer / Recognized Contributor
Flag Halifax
Thanks Meter: 25,932
 
Donate to Me
More
adbsync utility by @Renate NST is pretty perfect for this too. I've got a batch automation for Windows packaged up in my Odds and Ends thread but I'm sure it wouldn't be hard to get something similar setup for Linux.
The Following User Says Thank You to osm0sis For This Useful Post: [ View ]
20th September 2016, 01:13 AM |#5  
Member
Thanks Meter: 12
 
More
Nice!
Is there a way to do this but instead of coping files, copy the raw data from the internal memory? So that I can recover deleted files from the device from the generated raw file.
The Following User Says Thank You to Tarcis For This Useful Post: [ View ] Gift Tarcis Ad-Free
20th September 2016, 04:31 AM |#6  
Member
Thanks Meter: 60
 
More
Quote:
Originally Posted by Tarcis

Nice!
Is there a way to do this but instead of coping files, copy the raw data from the internal memory? So that I can recover deleted files from the device from the generated raw file.

Something like (run in the host console)

Code:
adb exec-out "cat <sdcard partition> 2>/dev/null" > sdcard.raw
Or

Code:
adb exec-out "dd if=<sdcard partition> bs=64k 2>/dev/null" > sdcard.raw
This works on Windows too.

Something like on the first post that works on Windows but without the progress report (which is what the FIFO is for):

Code:
adb exec-out "tar czf - /sdcard" > sdcard.tar.gz
adb exec-out is an undocumented adb command that works like adb shell but without creating a pseudoterminal thus it works with binary content.
The Following 4 Users Say Thank You to char101 For This Useful Post: [ View ] Gift char101 Ad-Free
20th September 2016, 10:24 AM |#7  
Member
Thanks Meter: 8
 
More
Has it been confirmed that all files are backed up this way? Main issue I have when backing up via MTP is that not all files seem to be transferred properly e.g. Carbon/Helium backups missing the json file.
20th September 2016, 01:06 PM |#8  
xdvs23's Avatar
OP Senior Member
Flag Bavaria
Thanks Meter: 1,832
 
Donate to Me
More
Quote:
Originally Posted by tech3475

Has it been confirmed that all files are backed up this way? Main issue I have when backing up via MTP is that not all files seem to be transferred properly e.g. Carbon/Helium backups missing the json file.

As far as I know, tar backups all files recursively in the directory you specified. Hence that should work perfectly fine. I recommend to use TWRP 3+ for this, but you can also try it on android itself.
The Following User Says Thank You to xdvs23 For This Useful Post: [ View ] Gift xdvs23 Ad-Free
20th September 2016, 01:13 PM |#9  
xdvs23's Avatar
OP Senior Member
Flag Bavaria
Thanks Meter: 1,832
 
Donate to Me
More
Quote:
Originally Posted by Tarcis

Nice!
Is there a way to do this but instead of coping files, copy the raw data from the internal memory? So that I can recover deleted files from the device from the generated raw file.

It depends on your device, but in general, you can transfer any binary data over exec-out in highspeed.

Code:
adb root
adb exec-out "dd if=/dev/block/mmcblk0" > mmcblk0.img
In this case it would copy the whole internal storage including all partitions and everything from 0 to end of your internal emmc to your computer.

On newer devices/systems you can also use the following to copy the single partitions (in this example /data) as raw image to your pc:

Code:
adb root
adb exec-out "dd if=/dev/block/bootdevice/by-name/userdata">userdata.img
Examples for partition names: system, data, userdata, cache, boot, LOGO, recovery, ...

If you want to compress all partitions separately into a tar archive, following command should do:

Code:
adb root && adb exec-out 'cd /dev/block/bootdevice/by-name && tar -ca ./' > allpartitions.tar
This would also work on windows.
The Following 2 Users Say Thank You to xdvs23 For This Useful Post: [ View ] Gift xdvs23 Ad-Free
21st September 2016, 03:24 AM |#10  
Member
Thanks Meter: 12
 
More
Quote:
Originally Posted by char101

Something like (run in the host console)

Code:
adb exec-out "cat <sdcard partition> 2>/dev/null" > sdcard.raw
Or

Code:
adb exec-out "dd if=<sdcard partition> bs=64k 2>/dev/null" > sdcard.raw
This works on Windows too.

Something like on the first post that works on Windows but without the progress report (which is what the FIFO is for):

Code:
adb exec-out "tar czf - /sdcard" > sdcard.tar.gz
adb exec-out is an undocumented adb command that works like adb shell but without creating a pseudoterminal thus it works with binary content.



Quote:
Originally Posted by xdvs23

It depends on your device, but in general, you can transfer any binary data over exec-out in highspeed.

Code:
adb root
adb exec-out "dd if=/dev/block/mmcblk0" > mmcblk0.img
In this case it would copy the whole internal storage including all partitions and everything from 0 to end of your internal emmc to your computer.

On newer devices/systems you can also use the following to copy the single partitions (in this example /data) as raw image to your pc:

Code:
adb root
adb exec-out "dd if=/dev/block/bootdevice/by-name/userdata">userdata.img
Examples for partition names: system, data, userdata, cache, boot, LOGO, recovery, ...

If you want to compress all partitions separately into a tar archive, following command should do:

Code:
adb root && adb exec-out 'cd /dev/block/bootdevice/by-name && tar -ca ./' > allpartitions.tar
This would also work on windows.

Thank you so much for the commands, but I ran them on Windows, and I get no error, but the output file is 1kb.
I tried running on dos prompt, I´ll try again tomorrow on cygwin
21st September 2016, 11:31 AM |#11  
Junior Member
Thanks Meter: 3
 
More
thank but too hard for me
can i have another software to do this?
Post Reply Subscribe to Thread

Tags
android, backup, compress, tar, twrp
Previous Thread Next Thread
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes