How to backup compressed data from your android device to your computer

xdvs23

Senior Member
Apr 11, 2015
1,054
1,923
0
20
Bavaria
xdevs23.me
Disclaimer
Code:
Whatever you do, it's your decision and you do it on your own risk!
I am not responsible for bricks, any kind of damages, data loss or any other unwanted result!
Everything you do is done on YOUR OWN RESPONSIBILITY!
And I am also not responsible if you lose the chat with your crush.
You might want to transfer your internal storage such as /sdcard to your computer directly and compressed as fast as possible.
So here is how you do it (on linux using bash):

Reboot your phone into TWRP, connect your device to your computer, and run the following command:

Code:
adb shell mkfifo /bk.pipe && adb exec-out 'tar -cvap data/media/0 2>/bk.pipe' | dd of=sdcard.tar & adb shell cat /bk.pipe &
In this example your data (in this case /data/media/0) will be backed up to your computer into the file sdcard.tar.
Compression is done automatically and you will get a nice output on your terminal on which file is being backed up currently.

When everything is finished you should get a message similar to this on your terminal:

Code:
79692237+2 records in
79692238+0 records out
40802425856 bytes (41 GB, 38 GiB) copied, 3550,38 s, 11,5 MB/s
(On the result of this you can see that the speed is around 12 MB/s, means that 1 GB backups in less than 1 minute)

When you get that, it means that your backup has finished and you can safely remove the usb cable.

Note that you can do this with any directory on your device, so if you desire to back up /data, just write data instead of data/media/0.

Example of the resulting sdcard.tar file (opened on my computer):


Decompressing your data directly to your device might get a little trickier:

Code:
adb shell mkfifo /rst.pipe && adb push sdcard.tar /rst.pipe & adb shell 'cd /; cat /rst.pipe | tar -xv'
If that does not work, you can just extract the content of the tar file normally onto your computer and then copy the files over using adb push or copy the tar file to your phone and extract it using the following command:

Code:
adb push sdcard.tar /sdcard/sdcard.tar
adb shell 'cd /; tar -xvf /sdcard/sdcard.tar'
And that should do the job as well.

---

See also:

tar to pipe but keep -v verbose output separate from STDERR

Transferring binary data over ADB shell (ie fast file transfer using tar)

Compose and pipe a gzipped tar archive in one go

Bash tar and output to log file

GNU Tar

Encrypting and Compressing

How encrypt data/text stream instead of a file?

Post on my blog:

How to backup compressed data from your android device to your computer
 
Last edited:

michael_ch

Senior Member
Oct 27, 2009
233
209
63
Hmm, that looks really complicated. How about rsync instead?

(Note: This requires root and it requires rsync... It's present on CyanogenMod and probably other custom roms, but usually not on stock ROMs...)

Code:
1. Create rsync.conf:
[root]
path = /
read only = no
uid = root
gid = root

2. Upload the file to the phone
$ adb push rsync.conf /sdcard/

3. Restart adbd with root permissions
$ adb root

4. Set up TCP forwarding of the rsync daemon through adb
$ adb forward tcp:9873 tcp:873

5. Run the daemon on the phone
$ adb shell /system/xbin/rsync --daemon --config /sdcard/rsync.conf --no-detach

6. Copy files in both directions
rsync -v rsync://localhost:9873/root/
rsync --help
Edit: You can also run rsync via SSH, e.g. with the help of SSHdroid:
Code:
rsync -e "ssh -l root -p 2222" -v <phone-ip>:/
 
Last edited:
  • Like
Reactions: _that

osm0sis

Senior Recognized Developer / Recognized Contribut
Mar 14, 2012
14,402
32,367
263
Halifax
adbsync utility by @Renate NST is pretty perfect for this too. I've got a batch automation for Windows packaged up in my Odds and Ends thread but I'm sure it wouldn't be hard to get something similar setup for Linux.
 
  • Like
Reactions: nenebear

Tarcis

Senior Member
Mar 11, 2015
57
12
0
Nice!
Is there a way to do this but instead of coping files, copy the raw data from the internal memory? So that I can recover deleted files from the device from the generated raw file.
 
  • Like
Reactions: YourFriendCaspian

char101

Senior Member
Jan 13, 2011
112
78
0
Nice!
Is there a way to do this but instead of coping files, copy the raw data from the internal memory? So that I can recover deleted files from the device from the generated raw file.
Something like (run in the host console)

Code:
adb exec-out "cat <sdcard partition> 2>/dev/null" > sdcard.raw
Or

Code:
adb exec-out "dd if=<sdcard partition> bs=64k 2>/dev/null" > sdcard.raw
This works on Windows too.

Something like on the first post that works on Windows but without the progress report (which is what the FIFO is for):

Code:
adb exec-out "tar czf - /sdcard" > sdcard.tar.gz
adb exec-out is an undocumented adb command that works like adb shell but without creating a pseudoterminal thus it works with binary content.
 

tech3475

Senior Member
Nov 4, 2013
90
10
28
Has it been confirmed that all files are backed up this way? Main issue I have when backing up via MTP is that not all files seem to be transferred properly e.g. Carbon/Helium backups missing the json file.
 

xdvs23

Senior Member
Apr 11, 2015
1,054
1,923
0
20
Bavaria
xdevs23.me
Has it been confirmed that all files are backed up this way? Main issue I have when backing up via MTP is that not all files seem to be transferred properly e.g. Carbon/Helium backups missing the json file.
As far as I know, tar backups all files recursively in the directory you specified. Hence that should work perfectly fine. I recommend to use TWRP 3+ for this, but you can also try it on android itself.
 
  • Like
Reactions: tech3475

xdvs23

Senior Member
Apr 11, 2015
1,054
1,923
0
20
Bavaria
xdevs23.me
Nice!
Is there a way to do this but instead of coping files, copy the raw data from the internal memory? So that I can recover deleted files from the device from the generated raw file.
It depends on your device, but in general, you can transfer any binary data over exec-out in highspeed.

Code:
adb root
adb exec-out "dd if=/dev/block/mmcblk0" > mmcblk0.img
In this case it would copy the whole internal storage including all partitions and everything from 0 to end of your internal emmc to your computer.

On newer devices/systems you can also use the following to copy the single partitions (in this example /data) as raw image to your pc:

Code:
adb root
adb exec-out "dd if=/dev/block/bootdevice/by-name/userdata">userdata.img
Examples for partition names: system, data, userdata, cache, boot, LOGO, recovery, ...

If you want to compress all partitions separately into a tar archive, following command should do:

Code:
adb root && adb exec-out 'cd /dev/block/bootdevice/by-name && tar -ca ./' > allpartitions.tar
This would also work on windows.
 
Last edited:

Tarcis

Senior Member
Mar 11, 2015
57
12
0
Something like (run in the host console)

Code:
adb exec-out "cat <sdcard partition> 2>/dev/null" > sdcard.raw
Or

Code:
adb exec-out "dd if=<sdcard partition> bs=64k 2>/dev/null" > sdcard.raw
This works on Windows too.

Something like on the first post that works on Windows but without the progress report (which is what the FIFO is for):

Code:
adb exec-out "tar czf - /sdcard" > sdcard.tar.gz
adb exec-out is an undocumented adb command that works like adb shell but without creating a pseudoterminal thus it works with binary content.


It depends on your device, but in general, you can transfer any binary data over exec-out in highspeed.

Code:
adb root
adb exec-out "dd if=/dev/block/mmcblk0" > mmcblk0.img
In this case it would copy the whole internal storage including all partitions and everything from 0 to end of your internal emmc to your computer.

On newer devices/systems you can also use the following to copy the single partitions (in this example /data) as raw image to your pc:

Code:
adb root
adb exec-out "dd if=/dev/block/bootdevice/by-name/userdata">userdata.img
Examples for partition names: system, data, userdata, cache, boot, LOGO, recovery, ...

If you want to compress all partitions separately into a tar archive, following command should do:

Code:
adb root && adb exec-out 'cd /dev/block/bootdevice/by-name && tar -ca ./' > allpartitions.tar
This would also work on windows.
Thank you so much for the commands, but I ran them on Windows, and I get no error, but the output file is 1kb.
I tried running on dos prompt, I´ll try again tomorrow on cygwin
 
Last edited:

Emma Tayler

Member
Mar 2, 2016
30
1
0
There are a few ways which you can make a Nandroid backup. The recommended way is to use a custom recovery to create one, and it’s the only way to restore from one. You should be able to use any custom recovery that offers Nandroid backup capabilities — if you don’t want to search around, the best choices are CWM and TWRP. Once you’ve flashed a custom recovery onto your device, you can boot into it and choose to create (or later on, restore from) a Nandroid backup. It’ll go through the process and create a backup file on your microSD card or other equivalent storage location.
 

xdvs23

Senior Member
Apr 11, 2015
1,054
1,923
0
20
Bavaria
xdevs23.me
There are a few ways which you can make a Nandroid backup. The recommended way is to use a custom recovery to create one, and it’s the only way to restore from one. You should be able to use any custom recovery that offers Nandroid backup capabilities — if you don’t want to search around, the best choices are CWM and TWRP. Once you’ve flashed a custom recovery onto your device, you can boot into it and choose to create (or later on, restore from) a Nandroid backup. It’ll go through the process and create a backup file on your microSD card or other equivalent storage location.
Then tell me how to create a nandroid backup from your internal storage (/sdcard or /data/media/0)
 

xdvs23

Senior Member
Apr 11, 2015
1,054
1,923
0
20
Bavaria
xdevs23.me
thank but too hard for me :(
can i have another software to do this?
I'm actually making backup tools which you can then use to create backups easily, maybe I'll also make a program with a user interface for windows where you can do that much easier.

Of course I cant promise anything atm cuz I have some other things to do as well.
 
  • Like
Reactions: Tarcis

Tarcis

Senior Member
Mar 11, 2015
57
12
0
Then open the file using e.g. notepad and show me the contents. It could be an error message or something...
good idea!

The files contained:
allpartitions.tar: /sbin/sh: syntax error: unterminated quoted string on one tr
mmcblk0.img: 0kb
sdcard.raw : /sbin/sh: syntax error: unexpected redirection
userdata:img: dd: can't open '/dev/block/bootdevice/by-name/userdata': No such file or directory