[GUIDE] How to extract, create or edit android adb backups

Search This thread

LucasDupuis32

Member
May 16, 2016
20
2
OnePlus 8
I see, Apple uses bsdtar instead of gnu tar because of licensing, but you can install it from homebrew: https://formulae.brew.sh/formula/gnu-tar

Then check again:
Code:
gtar -tf archive.tar
echo $?
It should display error code different from 0.

It's possible that you can use download gtar without installing it, but no idea.

Also even if the file is a lot bigger than the app directory, either the rest is filled with 00 values, or has damaged entries, or they are in a format not recognized. You would have to open archive.tar with a hex editor with support for very large files, and look at the end of last app file then search the next entry if there's something valid.

For example where it says "(app data stuff)" which app is the latest detected as invalid? So we can start searching for offset.
Indeed, gnu tar gives an error 2 :
Code:
lucasdupuis@MacBook-Air-de-Lucas backup 07:22 % gtar -tf backup.tar
apps/android.overlay.common/_manifest
apps/android.overlay.target/_manifest
apps/android.qvaoverlay.common/_manifest
(...)
apps/com.remarkable.mobile/f/ee98185b-b7a4-4f3d-8159-e51105b7fbda/1957a2ce-2455-4cb3-8bdf-4cfb00277c53.rm
gtar: Skipping to next header
gtar: Exiting with failure status due to previous errors
lucasdupuis@MacBook-Air-de-Lucas backup 07:22 % echo $?
2
com.remarkable.mobile is the latest detected app, but I don't know if it's valid or not ?

Is there an hex editor that you recommend so I can open the .tar file ?
 
Last edited:

scandiun

Senior Member
Jul 12, 2010
1,903
983
Indeed, gnu tar gives an error 2 :
Code:
lucasdupuis@MacBook-Air-de-Lucas backup 07:22 % gtar -tf backup.tar
apps/android.overlay.common/_manifest
apps/android.overlay.target/_manifest
apps/android.qvaoverlay.common/_manifest
(...)
apps/com.remarkable.mobile/f/ee98185b-b7a4-4f3d-8159-e51105b7fbda/1957a2ce-2455-4cb3-8bdf-4cfb00277c53.rm
gtar: Skipping to next header
gtar: Exiting with failure status due to previous errors
lucasdupuis@MacBook-Air-de-Lucas backup 07:22 % echo $?
2
com.remarkable.mobile is the latest detected app, but I don't know if it's valid or not ?

Is there an hex editor that you recommend so I can open the .tar file ?
A-You can check how many files and folders are there for such app:
Code:
gtar -tvf archive.tar apps/com.remarkable.mobile
or
Code:
gtar -tvf archive.tar apps/com.remarkable.mobile/
And check error with $? after any command

It can be that all the files that are for such app are fine but may be more missing or not. If you want to test, you can covert the app to ab file.

You can create a tar with only that app:
Code:
java -jar tar-bin-split.jar -split-android-shared archive.tar
Then, even if has error, check if tar for com.remarkable.mobile exits. Maybe "split-android/com.remarkable.mobile.tar"

Then you can convert to ab file and test:
Code:
java -jar abp.jar pack-kk split-android/com.remarkable.mobile.tar com.remarkable.mobile.ab

Then restore it to some android device or Android Studio virtual device or virtual machine:
Code:
adb restore com.remarkable.mobile.ab

Check that the app is as when you backed it up. If the app has the same notes and data it means that the app is complete inside the original archive.tar. Otherwise maybe that the app does not support adb backup 100% or that some of its files where in the damaged part. You may need to install the app first if the apk is not included which can be seen in the first command A.
 

LucasDupuis32

Member
May 16, 2016
20
2
OnePlus 8
A-You can check how many files and folders are there for such app:
Code:
gtar -tvf archive.tar apps/com.remarkable.mobile
or
Code:
gtar -tvf archive.tar apps/com.remarkable.mobile/
And check error with $? after any command

It can be that all the files that are for such app are fine but may be more missing or not. If you want to test, you can covert the app to ab file.

You can create a tar with only that app:
Code:
java -jar tar-bin-split.jar -split-android-shared archive.tar
Then, even if has error, check if tar for com.remarkable.mobile exits. Maybe "split-android/com.remarkable.mobile.tar"

Then you can convert to ab file and test:
Code:
java -jar abp.jar pack-kk split-android/com.remarkable.mobile.tar com.remarkable.mobile.ab

Then restore it to some android device or Android Studio virtual device or virtual machine:
Code:
adb restore com.remarkable.mobile.ab

Check that the app is as when you backed it up. If the app has the same notes and data it means that the app is complete inside the original archive.tar. Otherwise maybe that the app does not support adb backup 100% or that some of its files where in the damaged part. You may need to install the app first if the apk is not included which can be seen in the first command A.
But I only care about the /sdcard data, I don't need the app data. I don't understand why I'd have to test the com.remarkable.mobile backup
 

scandiun

Senior Member
Jul 12, 2010
1,903
983
But I only care about the /sdcard data, I don't need the app data. I don't understand why I'd have to test the com.remarkable.mobile backup
It would be to check if the corruption started just after the last app or in the middle of the last one. But anyway if you don't want to test it, I would you recommend using bless hex editor:
Code:
bless archive.tar

Then search for apps/com.remarkable.mobile as Text, and the next entry if valid should be shared/ insted of apps/
 

LucasDupuis32

Member
May 16, 2016
20
2
OnePlus 8
It would be to check if the corruption started just after the last app or in the middle of the last one. But anyway if you don't want to test it, I would you recommend using bless hex editor:
Code:
bless archive.tar

Then search for apps/com.remarkable.mobile as Text, and the next entry if valid should be shared/ insted of apps/
I think com.remarkable.mobile is not the last app of the backup. I'll try your method to see
I get an error when I try to use bless :
Code:
lucasdupuis@MacBook-Air-de-Lucas backup 07:22 % bless --folder /Volumes/Lucas/backup\ 07\:22 --file backup.tar
Authorization required
The backup isn't encrypted so I don't see what authorization they're talking about

I found another Hex editor called Hex Fiend, but there are a lot of apps/com.remarkable.mobile text entries, I'm trying to find the last one
 
  • Like
Reactions: scandiun

LucasDupuis32

Member
May 16, 2016
20
2
OnePlus 8
Capture d’écran 2022-08-28 à 22.11.02.png

I found the last apps/com.remarkable.mobile entry. Here's what it looks like after it
 

scandiun

Senior Member
Jul 12, 2010
1,903
983
I found the last apps/com.remarkable.mobile entry. Here's what it looks like after it
Yes that looks like the last entry, unless the things below are data which I'm not sure.
Search for shared/0 after that entry, or for ustar and see if there's any which I doubt. You can just move the slider down or use keys page up/page down and see if there's any recognizable text. If there's none, it's likely that the data is garbage and is irrecoverable.
 
  • Like
Reactions: LucasDupuis32

scandiun

Senior Member
Jul 12, 2010
1,903
983
I can also see some shared/0/DCIM entries (which is the data I'm mostly looking for)
I see, so is possible that the data is recoverable, however there must something wrong between the last apps/ entry and the first shared/0 one, because tar fails in the middle and does not allow to continue. Could be possible to see screenshot of such data? You can send me private message if you think there is something not allowed to be viewed publicly but is to know why tar fails.
 

LucasDupuis32

Member
May 16, 2016
20
2
OnePlus 8
I see, so is possible that the data is recoverable, however there must something wrong between the last apps/ entry and the first shared/0 one, because tar fails in the middle and does not allow to continue. Could be possible to see screenshot of such data? You can send me private message if you think there is something not allowed to be viewed publicly but is to know why tar fails.
I sent you a PM
 
  • Like
Reactions: scandiun

scandiun

Senior Member
Jul 12, 2010
1,903
983
@LucasDupuis32 glad you could extract all your files with the "--ignore-zeros" option (-i) and everything seems to have been recovered. Did you notice anything missing, any picture or video?
 

Top Liked Posts

  • There are no posts matching your filters.
  • 54
    What is an android adb backup?
    An adb backup is a file with ab extension, generated by android's backup manager when we request it via adb shell. This allows you to backup some data of the phone, but is not a replacement of a clockworkmod backup:

    - Java 7 or higher is required because of SYNC_FLUSH mode for the Deflater.
    - If the backup contains apk+data for an app, restore will work. If contains only the data, you must install the app first on the device and then restore. Installing the app later won't work.
    - Some apps include a policy where the apk is never backed up even if specified.
    - Inside an ab file is a tar file, which contains files and folders in a certain order. You have you respect that order, which is not necessarily alphabetical like tar does by default. Such order is listed in "Full local backup infrastructure".
    - Inside the tar file, directories must not have trailing slash, for that reason pax, star or equivalent has to be used.
    - There are some bugs present in the android source code.
    - 'adb backup' or 'adb restore' are pretty slow, between 1 and 2.6 MBps.
    - The adb backups (usually with .ab extension) can be password protected or not. If the device is encrypted this is a must and has to be the same used for the device.
    - Star is recommended instead of pax, which supports path length only up to 100 characters, so will fail in some cases. This usually happens with browsers history.
    - ADB backups are not a replacement for a Nandroid backup. The whole /data partition is not backed up, only a part of it. Also, other partitions like /system, /preload, /cache, modem, RIL, efs, kernel or recovery are never backed up because are not user data. This is to prevent issue when restoring on a different device. It has also the biggest
    - If you have the device encrypted with a password, you must use that particular one for backup creation and restore. You can't create a backup without password or a different one in that particular case, or if you try to restore will fail.
    - star for cygwin for windows is attached (move it to C:\cygwin\bin\star.exe) since there is no package available. It can be compiled from the schily (ftp://ftp.berlios.de/pub/schily/).
    - The best way to test if an adb backup has errors, is to convert it to tar and then check.advantage that doesn't require root to operate, so is totally compatible with stock roms, locked bootloaders and device encryption.
    - There's also a bash script called adb-split.sh that creates individual adb backups for each recognized app from the full one, so you can restore apps individually. Same encryption is preserved to them, if any.

    You can use java or perl, although with perl can be more complicated because requires downloading some modules from cpan and some ssl headers.

    To know more information about types of android backups:
    [Guide] Types of Android backups

    Software needed
    The entire process can be done from Windows, but is better to use a unix-like operating system, like Linux, OS X or BSD, since we should extract the files on a filesystem that preservers file permissions and owners and repack the tar archive. Furthermore some tools like star are easier to get for linux. In such cases, using Virtualbox or VMWare Workstation is highly recommended. Using Windows should work in any case whatsoever.

    Cygwin
    Preferably 32 bits: has more packages and is less prone to errors. Furthermore, the star for cygwin attached is 32 bits.

    Java 7 or higher
    • Oracle Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files 7 if you are going to work with password encrypted backups.
      You need to install the files local_policy.jar and US_export_policy.jar under jre's lib/security folder, for example:
      - For Windows:
      • C:\Program Files\Java\jdk1.7.0_09\jre\lib\security\
      • C:\Program Files\Java\jre7\lib\security\
      • C:\Program Files (x86)\Java\jdk1.7.0_07\jre\lib\security\
      • C:\Program Files (x86)\Java\jre7\lib\security\
      - For Linux or BSD:
      /usr/local/jdk1.7/jre/lib/security/
      /usr/lib/jvm/java-7-openjdk-*/jre/lib/security/
      /usr/local/openjdk7/jre/lib/security/
      - For OS X:
      • /Library/Java/JavaVirtualMachines/jdk1.7.0_09.jdk/Contents/Home/jre/lib/security/

    Perl
    • Perl is available for several operating systems
    • libssl or openssl headers. If you are using Linux or Cygwin is much more easier
    • cpan modules required by use functions

    pax or star
    pax is an archiving utility that compresses in ustar by default. In this case is extremely useful because stores directories without trailing slash. Is available for all operating systems, and in windows can be installed via cygwin's setup.exe or download a native version like gnuwin32 (sometimes available via bsdtar with pax option). pax doesn't work when paths are more than 100 characters lenght, so I recommend star instead.

    star allows storing directories without trailing slash. You can get the ubuntu version 1.5 here. For Windows you can use Cygwin's version included.

    Android Backup Extractor
    Android Backup Extractor is the java application that does all the job. It includes the perl scripts.


    How it works, better with an example
    I will use an example to demonstrate how it works, with java version. If you like to use perl, just grab the perl scripts and is nearly the same.
    This is extracted from the readme file, and will extract a whole adb backup and repack only the data for the game Grand Theft Auto III for android.

    1) Convert the original adb backup to tar format:
    Code:
    java -jar abe.jar unpack nexus7.ab nexus7.tar <password>

    2) Extract the contents of the tar archive. This should be done on a filesystem where the permissions of the files inside the tar are preserved, for example using linux, mac or bsd. Up to two folders may appear, apps and shared:
    Code:
    tar -xvf nexus7.tar

    3) Make a list of all the contents of the original archive in the order they are archived:
    Code:
    tar -tf nexus7.tar > nexus7.list

    4) Create a new list only with the files and folders you want, in proper order. For example for the GTA 3 (you can try savegames instead of all data):
    Code:
    cat nexus7.list | grep com.rockstar.gta3 > gta3.list

    5) Create the new tar archive. The directories stored on tar shouldn't contain trailing slashes, so I use pax or star instead of tar. Pax works also if paths are up to 100 of lenght:
    Code:
    cat gta.list | pax -wd > gta3.tar
    OR
    Code:
    star -c -v -f gta3.tar -no-dirslash list=gta3.list

    6) Create the adb backup from the tar archive. Password is optional:
    Code:
    java -jar abe.jar pack gta3.tar gta3.ab <password>

    Note: if the backup is not encrypted zlib can be used instead for both unpack and pack the ab archive:
    - Quick unpacking:
    Code:
    dd if=nexus7.ab bs=24 skip=1 | openssl zlib -d > nexus7.tar
    - Quick packing:
    Code:
    dd if=nexus7.ab bs=24 count=1 of=gta3.ab ; openssl zlib -in gta3.tar >> gta3.ab

    Split an Android backup
    Use Android Backup Splitter (is a shell script). Works on Cygwin too:
    sh adb-split.sh backup.ab [password if needed]
    Resulting files go in app-ab folder

    # ADB Backup Splitter:
    #
    # This bash script utility generates an adb backup for each item
    # on apps/ folder. That equals to generating an adb backup for
    # each app.
    #
    # shared/0 and shared/1 (sdcards) are ignored, so that means that
    # external data and OBB files, although may be backed up, may not
    # be restored. For doing so backup the folder Android of the sdcards.
    # Creating a backup with -shared flag has know issues (corruption)
    # so is not recommended. Future releases of this script may add
    # the option to automatically detect things from sdcards inside
    # the adb backup file.
    #
    # This script works on Cygwin also, and in general where star does.
    #
    # There may be apps in the backup that don't include the apk. In that
    # particular case, the data won't be restored, so the individual backup
    # is generated for testing purposes only. Installing the apk afterwards won't work.
    # There is generated an html file called apk-missing.html so you can open it and
    # install all applications from Play Store on any of your devices.

    Patch apps to allow adb backup
    Some apps specify in the AndroidManifest.xml android:allowBackup="false" so it won't be backed up by adb backup. If you have root and use Titanium Backup or CWM or TWRP they will do the job, but if you don't want root the only solution is to modify the apk to edit that string. Read here how to do that and the pros and cons:
    [GUIDE] How to enable adb backup for any app changing android:allowBackup

    Links
    Android Backup Splitter (@ Sourceforge)
    Android Backup Extractor (@ Sourceforge)
    Android Backup Extractor (@ GitHub)
    Documentation at Blogspot
    Full local backup infrastructure
    [Guide] Types of Android backups
    Backing Up Android Games
    Titanium Backup Decrypter
    Perl scripts to encrypt/decrypt adb backup files
    [GUIDE] Full Phone Backup without Unlock or Root
    [GUIDE] Internal Memory Data Recovery - Yes We Can!
    [Q] [adb] backup not working for Temple Run

    Related adb backup and restore bugs:
    Issue 55860: adb backup skips the apk for certain apps and doesn't prompt for download when restoring
    Issue 48126: device adb connecting to localhost adbd cannot execute commands anymore since 4.2.2 (error device offline)
    Issue 3254: adb shell doesn't return exit code of program
    Issue 53065: Debugging via ADB stopping the onClick action on ImageView widget
    Issue 53668: ADB : An existing connection was forcibly closed by the remote host
    Issue 54781: adb install: single quotes in filenames cause quoting issues
    Issue 55173: (Game) Offroad Legends does not detect obb when restored via adb restore
    Issue 55178: (Game) Carmageddon Promo does not detect obb when restored via adb restore
    Issue 28303: adb backup doesn't respect -noshared flag
    Issue 32830: adb restore errors not displayed on device
    Issue 34311: Galaxy Nexus gets stuck when restoring adb backup
    Issue 25780: BackupManager causes reboot when BackupAgent missing
    Issue 40691: ab backup and restore Ski Safari does not restore saved data
    Issue 16286: Restoration of phone not working properly
    Issue 39014: Nexus 7 adb restore freezes when restoring udk.android.reader

    Warning about Helium (Carbon): helium backups are just android adb backups that are not compressed with Java Deflater. It means they were created with the option static final boolean COMPRESS_FULL_BACKUPS = false.
    A normal adb restore should work.
    3
    I have tried similar attempt to restore my save of another game but that doesn't work.
    How did you replace the files in the .tar file? I am using 7z to replace the .tar file but nothing happen after i restored.
    Thanks.
    Read the guide. 7z is not compatible, you have to use star or equivalent and meet the requirements.
    2
    Great guide, and can confirm that it does work!

    I managed to use this method to restore my angry birds data from a Froyo phone (HTC Desire), to a Jelly Bean phone (HTC One X+). This is no easy task since you cannot use adb backup/restore on Froyo, but you can use adb pull, and you can't use adb push on Jelly Bean, but you can use adb backup/restore. Just in case anyone is looking to do the same, here is the process I used (requires having adb up and running but does not require root)...

    As the adb backup function doesn't work on Froyo, I had to use adb pull to grab the angry birds data files (highscores.lua and settings.lua). Then, making sure that you have angry birds installed on the jelly bean phone and have played the first level, use the adb backup command to backup just the angry birds app. You can then use scandiuns guide to extract a tar file from the backup file, and then extract the "app" folder from the tar file.

    Once this is done, you will find the highscores.lua and settings.lua files in the apps/com.rovio.angrybirds/f directory. Replace these two files with your original ones that you pulled from the old device. Then package up the app folder using the guide. Then just use adb restore to restore the modified backup file to your new device, and you should have all your old scores restored!

    Hope that helps and thanks to scandiun for the great guide! :good:
    2
    Maybe I'm missing something.

    I've installed Java, ran cygwins setup.exe and I think it installed everything. Also downloaded the Android Backup Extractor. I try running a windows command prompt with:
    java -jar abe.jar unpack nexus7.ab nexus7.tar

    All I get is "java is not an internal or external command". I'm running the command prompt in the android backup extractor folder. Does everything have to be in the same folder? Where do I dump everything?

    Probably java is not in your path, try

    Code:
    which java

    It should result something like
    Code:
    $ which java
    /cygdrive/c/Windows/system32/java

    Add Java to the Windows Path

    Then restart cygwin
    2
    Because i am using windows, so does it mean that i couldn't modify the .tar file in any way?
    thanks.

    Yes, from windows you can use pax. You can download it from cygwin's setup.exe. I've updated the guide, see the step 5.