[GUIDE] How to extract, create or edit android adb backups
- Thread starter scandiun
- Start date
I inform you that Fifa 14 1.3.0 is not compatible with adb backup, so isn't either with Carbon (Helium). You will see that restoring the backup just resets the progress. This happens with more games.
Titanium Backup is compatible and restores your savegame (I've checked it personally), so your only option is to root both devices.
You should send an email requesting support to help@eamobile.com and ask them the exact reason why fifa 14 it's not compatible with adb backup.
Good guide, I found this the most complete & informatice guide online. I had taken an ADB backup (with shared) (encrypted) on my Nexus 4 before unlocking and rooting, but I faced lot of issues while trying to restore. After lot of different tries (including buying Titanium Backup), I decided to give this a shot.. I was on windows the whole time, but the ABE utility would not work properly until I started using Linux. After another battle with the phone, I got the backup restored fully by the following strategy:
My backup of apps and shared storage was ~5.5GB, most of which is the shared storage. The Perl utility was much easier to setup and use compared to the Java one. I used that to pack and unpack (the other "quick" method failed always when I tried to restore). For packing the tar, I used pax in the end after trying star again and again. The only problem I faced, with pax, was the path length limit so some thumbnails were not included in the tar.
Note: Since I did a full backup which included some google apps causing issues when restored. I remove all such files by monitoring the restore process using `adb logcat | grep "Backup"`.
Hope this helps someone out
Cheers,
Prakhar
My backup of apps and shared storage was ~5.5GB, most of which is the shared storage. The Perl utility was much easier to setup and use compared to the Java one. I used that to pack and unpack (the other "quick" method failed always when I tried to restore). For packing the tar, I used pax in the end after trying star again and again. The only problem I faced, with pax, was the path length limit so some thumbnails were not included in the tar.
Note: Since I did a full backup which included some google apps causing issues when restored. I remove all such files by monitoring the restore process using `adb logcat | grep "Backup"`.
Hope this helps someone out
Cheers,
Prakhar
Thanks, I'm about to attempt the same, good to have reminder to stay away from system apps.
For anyone that does a lot of checking of their adb backups using dragomerlin's excellent Android Backup Extractor and are on windows may find this explorer shell extension I put together useful. It lets you quickly go from .ab file to .tar without messing with feeding paths into command line. Basically it consists of a reg file and corresponding batch file so you can just double click on the .ab file (or right click > extract) and a tar file will be produced in the same directory.
abextract.reg:
Code:
Windows Registry Editor Version 5.00
[HKEY_CLASSES_ROOT\.ab]
@="adb backup"
[HKEY_CLASSES_ROOT\.ab\shell]
[HKEY_CLASSES_ROOT\.ab\shell\extract]
[HKEY_CLASSES_ROOT\.ab\shell\extract\command]
@="C:\\android\\tools\\android-backup-extractor-20130526-bin\\abextract.bat \"%1\""
Code:
@echo off
echo extracting %1...
set abfile=%1
set tarfile=%abfile:.ab=.tar%
java -jar C:\android\tools\android-backup-extractor-20130526-bin\abe.jar unpack %abfile% %tarfile%Of course you may have to modify paths to the abe.jar and abextract.bat
And of course this doesn't take encryption into account but I think would be easy to modify for it (maybe even with user input)
First post on XDA woohoo! I thought I'd lurk until I had a grain of sand to give to the ocean of knowledge from which I've taken
ps I'd attach the files but I'm probably not allowed yet
Last edited:
Can't seem to get .tar working...
For some reason, I can't get
to work. I always end up with the error "tar: Cannot connect to C: resolve failed." I'm using Cygwin on Win7, and had the perl package installed, and added Cygwin to my PATH, but I can't think of anything else to fix the problem.
I tried restarting my Nexus 7 (2013) and remaking the backup file, then converting it to tar, but I noticed that my archives are always damaged/corrupted (according to WinRAR). Any ideas?
For some reason, I can't get
Code:
tar -xvf nexus7.tarI tried restarting my Nexus 7 (2013) and remaking the backup file, then converting it to tar, but I noticed that my archives are always damaged/corrupted (according to WinRAR). Any ideas?
It looks like latest 64 bits tar for cygwin is having issues. I recommend you to rename the cygwin folder and install it again but 32 bits.
64-bit TAR fails with "Cannot connect to e: resolve failed"
Ah okay. Thanks for that. I'll re-install with the 32-bit version and report back.
EDIT: Okay, I uninstalled the 64-bit Cygwin (and completely removed it's root folder, shortcuts, etc.) and installed the 32-bit version. At first I was getting the same exact error as above when I used the following:
Code:
tar -xvf C:/Android/backup/nexus7_2.tarSo I decided instead to just make a copy of the .tar that was made in the root directory that contained "abe.jar" file. For some reason, I guess it didn't like that I tried to direct it to the where the file was located. I don't know if that made a difference, but it worked, and I was able to then make the .list out of the extracted .tar.
My follow-up question is however, is that in the example in the OP, they make an individual tar for gta3. Is there a way I can somehow batch all of the apps I want to backup in one line? Or do I have to repeat this for each app I want to backup? Sorry for all the questions, but I'm slowly getting the hang of this!
Last edited:
Thanks for the heads up. There must be some incompatibility with version between version 1 and 2, since a new version is a mayor change. If that happened from 4.4.2 to 4.4.3 we should look to the changes between the two revisions.
I think the only "documentation" is the source code for BackupManagerService.java starting with line 2792...
Funky Android: KOT49H (4.4.2_r1) to KTU84L (4.4.3_r1) AOSP changelog
Using diff or patch you can get all the exact lines that were changed. Git does that also.
Using git you can get the entire diff and filter for a particular file, for example BackupManagerService.java, but probably it's not the only file affected.
well, given that the change seems to be only because of cryptographic reasons and my backup not being encrypted I hex-edited the ab file and changed '2' to '1' and the extractor works.
how to solve this problem?
(kitkat omni 4.4.4)
(kitkat omni 4.4.4)
Code:
Strong AES encryption allowed
Magic: ANDROID BACKUP
Version: 2
Compressed: 1
Algorithm: AES-256
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:58)
Caused by: java.lang.RuntimeException: javax.crypto.BadPaddingException: Given final block not properly padded
at org.nick.abe.AndroidBackup.extractAsTar(AndroidBackup.java:351)
at org.nick.abe.Main.main(Main.java:64)
... 5 more
Caused by: javax.crypto.BadPaddingException: Given final block not properly padded
at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:811)
at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:676)
at com.sun.crypto.provider.AESCipher.engineDoFinal(AESCipher.java:313)
at javax.crypto.Cipher.doFinal(Cipher.java:1922)
at org.nick.abe.AndroidBackup.extractAsTar(AndroidBackup.java:268)
... 6 moreTwo ideas:
- Check that your password is correct
- Restore the backup on the same phone or another to see if everything restore properly. That may greatly depend whether the backup is of a single app or of the whole device.
Is there a way to fix the .tar files created when you unpack an .ab file from a Helium/carbon backup?
The tar files from carbon backups don't need to be "fixed", it's just that they are a different format. They are created using the debug flag I think, so they can be saved directly to the device (which is not allowed by default).
Similar threads
Top Liked Posts
-
There are no posts matching your filters.
-
54What is an android adb backup?
An adb backup is a file with ab extension, generated by android's backup manager when we request it via adb shell. This allows you to backup some data of the phone, but is not a replacement of a clockworkmod backup:
- Java 7 or higher is required because of SYNC_FLUSH mode for the Deflater.
- If the backup contains apk+data for an app, restore will work. If contains only the data, you must install the app first on the device and then restore. Installing the app later won't work.
- Some apps include a policy where the apk is never backed up even if specified.
- Inside an ab file is a tar file, which contains files and folders in a certain order. You have you respect that order, which is not necessarily alphabetical like tar does by default. Such order is listed in "Full local backup infrastructure".
- Inside the tar file, directories must not have trailing slash, for that reason pax, star or equivalent has to be used.
- There are some bugs present in the android source code.
- 'adb backup' or 'adb restore' are pretty slow, between 1 and 2.6 MBps.
- The adb backups (usually with .ab extension) can be password protected or not. If the device is encrypted this is a must and has to be the same used for the device.
- Star is recommended instead of pax, which supports path length only up to 100 characters, so will fail in some cases. This usually happens with browsers history.
- ADB backups are not a replacement for a Nandroid backup. The whole /data partition is not backed up, only a part of it. Also, other partitions like /system, /preload, /cache, modem, RIL, efs, kernel or recovery are never backed up because are not user data. This is to prevent issue when restoring on a different device. It has also the biggest
- If you have the device encrypted with a password, you must use that particular one for backup creation and restore. You can't create a backup without password or a different one in that particular case, or if you try to restore will fail.
- star for cygwin for windows is attached (move it to C:\cygwin\bin\star.exe) since there is no package available. It can be compiled from the schily (ftp://ftp.berlios.de/pub/schily/).
- The best way to test if an adb backup has errors, is to convert it to tar and then check.advantage that doesn't require root to operate, so is totally compatible with stock roms, locked bootloaders and device encryption.
- There's also a bash script called adb-split.sh that creates individual adb backups for each recognized app from the full one, so you can restore apps individually. Same encryption is preserved to them, if any.
You can use java or perl, although with perl can be more complicated because requires downloading some modules from cpan and some ssl headers.
To know more information about types of android backups:
[Guide] Types of Android backups
Software needed
The entire process can be done from Windows, but is better to use a unix-like operating system, like Linux, OS X or BSD, since we should extract the files on a filesystem that preservers file permissions and owners and repack the tar archive. Furthermore some tools like star are easier to get for linux. In such cases, using Virtualbox or VMWare Workstation is highly recommended. Using Windows should work in any case whatsoever.
Cygwin
Preferably 32 bits: has more packages and is less prone to errors. Furthermore, the star for cygwin attached is 32 bits.
Java 7 or higher
- Oracle Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files 7 if you are going to work with password encrypted backups.
You need to install the files local_policy.jar and US_export_policy.jar under jre's lib/security folder, for example:
- For Windows:- C:\Program Files\Java\jdk1.7.0_09\jre\lib\security\
- C:\Program Files\Java\jre7\lib\security\
- C:\Program Files (x86)\Java\jdk1.7.0_07\jre\lib\security\
- C:\Program Files (x86)\Java\jre7\lib\security\
/usr/local/jdk1.7/jre/lib/security/
/usr/lib/jvm/java-7-openjdk-*/jre/lib/security/
/usr/local/openjdk7/jre/lib/security/
- For OS X:- /Library/Java/JavaVirtualMachines/jdk1.7.0_09.jdk/Contents/Home/jre/lib/security/
Perl- Perl is available for several operating systems
- libssl or openssl headers. If you are using Linux or Cygwin is much more easier
- cpan modules required by use functions
pax or star
pax is an archiving utility that compresses in ustar by default. In this case is extremely useful because stores directories without trailing slash. Is available for all operating systems, and in windows can be installed via cygwin's setup.exe or download a native version like gnuwin32 (sometimes available via bsdtar with pax option). pax doesn't work when paths are more than 100 characters lenght, so I recommend star instead.
star allows storing directories without trailing slash. You can get the ubuntu version 1.5 here. For Windows you can use Cygwin's version included.
Android Backup Extractor
Android Backup Extractor is the java application that does all the job. It includes the perl scripts.
How it works, better with an example
I will use an example to demonstrate how it works, with java version. If you like to use perl, just grab the perl scripts and is nearly the same.
This is extracted from the readme file, and will extract a whole adb backup and repack only the data for the game Grand Theft Auto III for android.
1) Convert the original adb backup to tar format:
Code:java -jar abe.jar unpack nexus7.ab nexus7.tar <password>
2) Extract the contents of the tar archive. This should be done on a filesystem where the permissions of the files inside the tar are preserved, for example using linux, mac or bsd. Up to two folders may appear, apps and shared:
Code:tar -xvf nexus7.tar
3) Make a list of all the contents of the original archive in the order they are archived:
Code:tar -tf nexus7.tar > nexus7.list
4) Create a new list only with the files and folders you want, in proper order. For example for the GTA 3 (you can try savegames instead of all data):
Code:cat nexus7.list | grep com.rockstar.gta3 > gta3.list
5) Create the new tar archive. The directories stored on tar shouldn't contain trailing slashes, so I use pax or star instead of tar. Pax works also if paths are up to 100 of lenght:
ORCode:cat gta.list | pax -wd > gta3.tar
Code:star -c -v -f gta3.tar -no-dirslash list=gta3.list
6) Create the adb backup from the tar archive. Password is optional:
Code:java -jar abe.jar pack gta3.tar gta3.ab <password>
Note: if the backup is not encrypted zlib can be used instead for both unpack and pack the ab archive:
- Quick unpacking:
- Quick packing:Code:dd if=nexus7.ab bs=24 skip=1 | openssl zlib -d > nexus7.tar
Code:dd if=nexus7.ab bs=24 count=1 of=gta3.ab ; openssl zlib -in gta3.tar >> gta3.ab
Split an Android backup
Use Android Backup Splitter (is a shell script). Works on Cygwin too:
sh adb-split.sh backup.ab [password if needed]
Resulting files go in app-ab folder
# ADB Backup Splitter:
#
# This bash script utility generates an adb backup for each item
# on apps/ folder. That equals to generating an adb backup for
# each app.
#
# shared/0 and shared/1 (sdcards) are ignored, so that means that
# external data and OBB files, although may be backed up, may not
# be restored. For doing so backup the folder Android of the sdcards.
# Creating a backup with -shared flag has know issues (corruption)
# so is not recommended. Future releases of this script may add
# the option to automatically detect things from sdcards inside
# the adb backup file.
#
# This script works on Cygwin also, and in general where star does.
#
# There may be apps in the backup that don't include the apk. In that
# particular case, the data won't be restored, so the individual backup
# is generated for testing purposes only. Installing the apk afterwards won't work.
# There is generated an html file called apk-missing.html so you can open it and
# install all applications from Play Store on any of your devices.
Patch apps to allow adb backup
Some apps specify in the AndroidManifest.xml android:allowBackup="false" so it won't be backed up by adb backup. If you have root and use Titanium Backup or CWM or TWRP they will do the job, but if you don't want root the only solution is to modify the apk to edit that string. Read here how to do that and the pros and cons:
[GUIDE] How to enable adb backup for any app changing android:allowBackup
Links
Android Backup Splitter (@ Sourceforge)
Android Backup Extractor (@ Sourceforge)
Android Backup Extractor (@ GitHub)
Documentation at Blogspot
Full local backup infrastructure
[Guide] Types of Android backups
Backing Up Android Games
Titanium Backup Decrypter
Perl scripts to encrypt/decrypt adb backup files
[GUIDE] Full Phone Backup without Unlock or Root
[GUIDE] Internal Memory Data Recovery - Yes We Can!
[Q] [adb] backup not working for Temple Run
Related adb backup and restore bugs:
Issue 55860: adb backup skips the apk for certain apps and doesn't prompt for download when restoring
Issue 48126: device adb connecting to localhost adbd cannot execute commands anymore since 4.2.2 (error device offline)
Issue 3254: adb shell doesn't return exit code of program
Issue 53065: Debugging via ADB stopping the onClick action on ImageView widget
Issue 53668: ADB : An existing connection was forcibly closed by the remote host
Issue 54781: adb install: single quotes in filenames cause quoting issues
Issue 55173: (Game) Offroad Legends does not detect obb when restored via adb restore
Issue 55178: (Game) Carmageddon Promo does not detect obb when restored via adb restore
Issue 28303: adb backup doesn't respect -noshared flag
Issue 32830: adb restore errors not displayed on device
Issue 34311: Galaxy Nexus gets stuck when restoring adb backup
Issue 25780: BackupManager causes reboot when BackupAgent missing
Issue 40691: ab backup and restore Ski Safari does not restore saved data
Issue 16286: Restoration of phone not working properly
Issue 39014: Nexus 7 adb restore freezes when restoring udk.android.reader
Warning about Helium (Carbon): helium backups are just android adb backups that are not compressed with Java Deflater. It means they were created with the option static final boolean COMPRESS_FULL_BACKUPS = false.
A normal adb restore should work.3
Read the guide. 7z is not compatible, you have to use star or equivalent and meet the requirements.2Great guide, and can confirm that it does work!
I managed to use this method to restore my angry birds data from a Froyo phone (HTC Desire), to a Jelly Bean phone (HTC One X+). This is no easy task since you cannot use adb backup/restore on Froyo, but you can use adb pull, and you can't use adb push on Jelly Bean, but you can use adb backup/restore. Just in case anyone is looking to do the same, here is the process I used (requires having adb up and running but does not require root)...
As the adb backup function doesn't work on Froyo, I had to use adb pull to grab the angry birds data files (highscores.lua and settings.lua). Then, making sure that you have angry birds installed on the jelly bean phone and have played the first level, use the adb backup command to backup just the angry birds app. You can then use scandiuns guide to extract a tar file from the backup file, and then extract the "app" folder from the tar file.
Once this is done, you will find the highscores.lua and settings.lua files in the apps/com.rovio.angrybirds/f directory. Replace these two files with your original ones that you pulled from the old device. Then package up the app folder using the guide. Then just use adb restore to restore the modified backup file to your new device, and you should have all your old scores restored!
Hope that helps and thanks to scandiun for the great guide! :good:2
Probably java is not in your path, try
Code:which java
It should result something like
Code:$ which java /cygdrive/c/Windows/system32/java
Add Java to the Windows Path
Then restart cygwin2
Yes, from windows you can use pax. You can download it from cygwin's setup.exe. I've updated the guide, see the step 5. - Oracle Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files 7 if you are going to work with password encrypted backups.
