Perl scripts to encrypt/decrypt adb backup files

Search This thread

puterboy

Senior Member
Jun 24, 2012
59
24
I wrote the following attached PERL routines for reading/decrypting/decompressing and writing/encrypting/compressing adb backup format backup files.

The routines are:
backupdecrypt.pl: Decrypt (and decompress) android backup file
backupencrypt.pl: Encrypt (and decompress) android backup file
tarfix.pl: Fix broken tar files produced by android backup when using -shared flag

The first two routines allow for reading and writing to the standard ".ab" adb backup format.

Backupdecrypt.pl takes an '.ab' file as input and outputs a standard format tar file (which may be optionally gzip'd).

Backupencrypt.pl takes an arbitrary file (though typically it should be tar file) as input and outputs a standard ".ab" format backup file. Options include the ability to encrypt (or not) and deflate (or not) the backup. Also, one can automatically decompress most standard input formats before encrypting.

For encryption, passwords can be queried for or passed on the command line or read from a file.

NOTE: unfortunately the standard 'adb backup' routine seem to have a SEVERE *BUG* in it when using the '--shared' option in combination with certain other options.

First, the backup is not compressed even though the header claims it is. To get around this, backupdecrypt.pl has a --nocompress option to override the header.

Second, the encapsulated tar file is corrupted by the insertion of 4 extra bytes before every file header and before every group of 64 512-byte blocks of data.

The third routine tarfix.pl fixes this corruption and outputs a normal readable tar file. So, if you are not able to recover a valid tar backup file using backupdecrypt.pl, try doing the following:

backupdecrypt.pl --nocompress <backup.ab> <backupdata>
tarfix.pl backupdata | tar xv


Enjoy!

NOTE: I am incredibly grateful to Nikolay Elenkov for providing sample java routines and for help in understanding the encryption formats
 

Attachments

  • abbackuproutines.tar
    40 KB · Views: 8,134
T

trogdan

Guest
Problem decrypting

just what i was looking for, unfortunately, the tarfix.pl doesn't seem to like my backup.

Code:
user@machine:~/Sandbox/transformerprime$ ~/bin/adbbackup/backupdecrypt.pl --nocompress backup.ab decrypted

the following is where things get funky. not recognized as a tar archive
Code:
user@machine:~/Sandbox/transformerprime$ ~/bin/adbbackup/tarfix.pl decrypted | tar xv
Illegal binary digit ']' ignored at /home/user/bin/adbbackup/tarfix.pl line 107.
Wide character in oct at /home/user/bin/adbbackup/tarfix.pl line 107.
Illegal binary digit '�������������' ignored at /home/user/bin/adbbackup/tarfix.pl line 107.
tar: This does not look like a tar archive
tar: Skipping to next header
Illegal octal digit '8' ignored at /home/user/bin/adbbackup/tarfix.pl line 107.
Wide character in oct at /home/user/bin/adbbackup/tarfix.pl line 107.
Illegal binary digit '�������������' ignored at /home/user/bin/adbbackup/tarfix.pl line 107.
Illegal hexadecimal digit 'X' ignored at /home/user/bin/adbbackup/tarfix.pl line 107.
Illegal hexadecimal digit '' ignored at /home/user/bin/adbbackup/tarfix.pl line 107.
Illegal octal digit '9' ignored at /home/user/bin/adbbackup/tarfix.pl line 107.
Wide character in oct at /home/user/bin/adbbackup/tarfix.pl line 107.
Illegal binary digit '�������������' ignored at /home/user/bin/adbbackup/tarfix.pl line 107.
Wide character in oct at /home/user/bin/adbbackup/tarfix.pl line 107.
 

binaryhero

New member
Jul 20, 2012
2
2
Fix for tarfix.pl issue

I had ran into issues with ADB backups performed under Android 4.0.4 before the JB upgrade (on a Samsung Galaxy Nexus). I did include the shared storage (accidentally or intentionally I don't remember) and I ran into this bug (Android issue 28303; sorry, I am new here and not allowed to post outside links). Some investigation revealed that while the backup was supposedly AES encrypted and "Deflate" compressed, this was only true for the first "part" of it. At around 150 MB into the file, a simple tar archive of the SD card content was appended. ADB was unable to restore any SD card content per the bug linked to above.

What I did to resolve:

  • Use a hex editor (HxD) to get to the start of the appended tar archive
  • Copy this part to a new ".tar" file
  • Experiment with tarfix.pl and run into the same issues as the previous poster
  • Look at the code and TAR file content and find out that 00 00 02 00 needs to be prepended to the tar file for tarfix.pl to do its job

Key learning was that the ADB backup tool will create plaintext, corrupted tar format backups that it cannot restore. It is problematic that while the user will believe they have an encrypted backup they can restore, they actually have a plaintext backup that they cannot restore...
 
Jun 29, 2010
10
0
Hi,

where did you get PBKDF2.pm from?

I couldnt find it in Fedora repos and the only one I got from the internat has an issue:


Undefined subroutine &Crypt::eek:penSSL::pBKDF2::derive called at ./backupdecrypt.pl line 266, <STDIN> line 1.

Thanks,
Klement
 

binaryhero

New member
Jul 20, 2012
2
2
CPAN

All modules were pulled from CPAN directly (I had to use Cygwin as I was on the road), e.g. "perl -M CPAN -e shell" and then issuing "install Crypt::eek:penSSL::pBKDF2".
 

KentigernEnnis

New member
Dec 23, 2009
1
1
I had ran into issues with ADB backups performed under Android 4.0.4 before the JB upgrade (on a Samsung Galaxy Nexus). I did include the shared storage (accidentally or intentionally I don't remember) and I ran into this bug (Android issue 28303; sorry, I am new here and not allowed to post outside links). Some investigation revealed that while the backup was supposedly AES encrypted and "Deflate" compressed, this was only true for the first "part" of it. At around 150 MB into the file, a simple tar archive of the SD card content was appended. ADB was unable to restore any SD card content per the bug linked to above.

What I did to resolve:

  • Use a hex editor (HxD) to get to the start of the appended tar archive
  • Copy this part to a new ".tar" file
  • Experiment with tarfix.pl and run into the same issues as the previous poster
  • Look at the code and TAR file content and find out that 00 00 02 00 needs to be prepended to the tar file for tarfix.pl to do its job

Key learning was that the ADB backup tool will create plaintext, corrupted tar format backups that it cannot restore. It is problematic that while the user will believe they have an encrypted backup they can restore, they actually have a plaintext backup that they cannot restore...

I ran afoul of this (foolishly didn't finish the 30 page forum post on it before diving in :( ). I'm trying your perl solution now, but I'm afraid I'm unfamiliar with tar headers in a hex viewer. Could I trouble you for some pointers on how best to determine where the TAR starts? I understand there is some sort of header, but I can't figure out what to look for.

Thanks, though, the perl runs well and I'm learning alot (far more than I ever wanted, tbh) about tars,encrypted backups, adb, etc.
:good:
 
  • Like
Reactions: MAFLO321

munjeni

Senior Member
Jun 2, 2011
9,720
22,374
Thanks for your great tools! Here is some better way for unpacking?
dd if=mybackup.ab bs=1 skip=24 | openssl zlib -d > mybackup.tar

For packing (just need to add 24 butes to header (41 4E 44 52 4F 49 44 20 42 41 43 4B 55 50 0A 31 0A 31 0A 6E 6F 6E 65 0A)?
openssl zlib -in mybackup.tar -out gg.ab

Hope it helps! :good:
 
Last edited:

scandiun

Senior Member
Jul 12, 2010
1,903
983
Thanks for your great tools! Here is some better way for unpacking?
dd if=mybackup.ab bs=1 skip=24 | openssl zlib -d > mybackup.tar

For packing (just need to add 24 butes to header (41 4E 44 52 4F 49 44 20 42 41 43 4B 55 50 0A 31 0A 31 0A 6E 6F 6E 65 0A)?
openssl zlib -in mybackup.tar -out gg.ab

Hope it helps! :good:

Of course. You have to concatenate the created ab backup to the first 24 bytes.

Get the first 24 bytes of an unencrypted backup
Code:
dd if=mybackup.ab bs=24 count=1 of=first24

Concatenate
Code:
cp first24 backup.ab
openssl zlib -in mybackup.tar >> backup.ab


---------- Post added at 10:25 AM ---------- Previous post was at 10:21 AM ----------

All modules were pulled from CPAN directly (I had to use Cygwin as I was on the road), e.g. "perl -M CPAN -e shell" and then issuing "install Crypt::eek:openSSL::pBKDF2".

Says "Missing argument to -M."
 
  • Like
Reactions: munjeni

munjeni

Senior Member
Jun 2, 2011
9,720
22,374
You concatenation is wrong! Your command "cp" will overwrite backup.ab! You can concetate 2 files using "cat" for example "cat first24 backup.ab > new.backup.ab" :)

Do you have idea how I can generate timestamp of these file in "13 number" format?

<?xml version="1.0" encoding="UTF-8"?>
<recordset version="1" timestamp="1344764788434" size="61667">
<record name="back.ab" type="1" size="12662" order="1" catagory="1" id="back.ab"><packagelist><package>com.android.settings</package></packagelist></record>
</recordset>

Edit:
found a way for generating timestamt with 13 numbers :)
stat -c '%Y000' backup.ab
 
Last edited:

marchard

Senior Member
Feb 3, 2009
71
2
I had ran into issues with ADB backups performed under Android 4.0.4 before the JB upgrade (on a Samsung Galaxy Nexus). I did include the shared storage (accidentally or intentionally I don't remember) and I ran into this bug (Android issue 28303; sorry, I am new here and not allowed to post outside links). Some investigation revealed that while the backup was supposedly AES encrypted and "Deflate" compressed, this was only true for the first "part" of it. At around 150 MB into the file, a simple tar archive of the SD card content was appended. ADB was unable to restore any SD card content per the bug linked to above.

What I did to resolve:

  • Use a hex editor (HxD) to get to the start of the appended tar archive
  • Copy this part to a new ".tar" file
  • Experiment with tarfix.pl and run into the same issues as the previous poster
  • Look at the code and TAR file content and find out that 00 00 02 00 needs to be prepended to the tar file for tarfix.pl to do its job

Key learning was that the ADB backup tool will create plaintext, corrupted tar format backups that it cannot restore. It is problematic that while the user will believe they have an encrypted backup they can restore, they actually have a plaintext backup that they cannot restore...

Hi al!
I also ran into the problem, that my .ab backup created with -shared option can't be restored.
Running backupdecrypt.pl seems to work fine, but tar archive is corrpted. But not the whole file: e.g. Ark sees some files and folders both from apps and storage. When using tarfix.pl i get same errors like trogdan in 3rd post.
To me it seems, that not every file in the tar archive is corrupted. I tried with cpio to rescue at least something and it runs quite a while. It also can read files from storage folder, but fails then somewhere in the middle (to be specific: i have a TitaniumBackup folder in storage. cpio can copy some of the files from this folder, but then fails at others. i don't know, why the corrupted segment should start somewhere and not at the beginning of storage...)
binaryhero and others: could you be please more specific how you managed to repair the .ab file or at least rescued the containing files? I would like to know, how to get tarfix.pl working. Is it possible, that only segments of the .ab file are corrupted? how can i identify them?

Any help and hints are highly appreciated! Thanks in advance! LLAP _\V/, bye Marc.
 

scandiun

Senior Member
Jul 12, 2010
1,903
983
Unfortunately dataloss occurred. that means, it is essential for me, to read somehow the content of .ab file. :(

zlib has to be able to process the ab file (considering that has no password). If zlib doesn't work, I don't know how could you workaround that, but seems impossible to me.

You can concetate 2 files using "cat" for example "cat first24 backup.ab > new.backup.ab" :)
To overwrite is this operator: >
To binary append this one: >>

So in this case is being done right.
 

cn198

Member
Jun 30, 2012
8
3
Portsmouth
Hi all! I'm having a few problems, and not really sure where I'm going wrong so hoping you can help. I am coming from froyo (rooted) to jelly bean (not rooted), so cannot use adb backup on the froyo phone, and cannot use adb push on jelly bean (don't want to root my htc one x+). So at the moment I haven't got many options to transfer my app data from my froyo phone to jelly bean.

At the moment, my plan is to use adb pull on the froyo phone to grab my "app data" (specifically angry birds), and then use adb backup on the jelly bean phone to create a .ab file. I then use the OPs perl files to convert the .ab file to a .tar, and then extract that file (on Ubuntu 12.04), replace the app data in the extracted folder with my froyo app data, compress that back to a .tar, then use the perl file to convert back to .ab. I then use adb restore to restore the backup. However, when I then go into angry birds, its just reset and there is no save data. Can anybody see if there is anything wrong with what i'm doing? :confused:
 

scandiun

Senior Member
Jul 12, 2010
1,903
983
Hi all! I'm having a few problems, and not really sure where I'm going wrong so hoping you can help. I am coming from froyo (rooted) to jelly bean (not rooted), so cannot use adb backup on the froyo phone, and cannot use adb push on jelly bean (don't want to root my htc one x+). So at the moment I haven't got many options to transfer my app data from my froyo phone to jelly bean.

At the moment, my plan is to use adb pull on the froyo phone to grab my "app data" (specifically angry birds), and then use adb backup on the jelly bean phone to create a .ab file. I then use the OPs perl files to convert the .ab file to a .tar, and then extract that file (on Ubuntu 12.04), replace the app data in the extracted folder with my froyo app data, compress that back to a .tar, then use the perl file to convert back to .ab. I then use adb restore to restore the backup. However, when I then go into angry birds, its just reset and there is no save data. Can anybody see if there is anything wrong with what i'm doing? :confused:

I am almost 100% sure that you are generating an invalid tar file.
 

scandiun

Senior Member
Jul 12, 2010
1,903
983
Is there a unique attribute applied to the original tar file then? Any idea how to generate a valid tar file? :)

It's not that simple. I am writing a guide based on a third party software for the purpose of Android backups. I'll post here as soon as is published. I hope to have it completed in a day our do.
 
Last edited:
  • Like
Reactions: cn198

Top Liked Posts

  • There are no posts matching your filters.
  • 18
    I wrote the following attached PERL routines for reading/decrypting/decompressing and writing/encrypting/compressing adb backup format backup files.

    The routines are:
    backupdecrypt.pl: Decrypt (and decompress) android backup file
    backupencrypt.pl: Encrypt (and decompress) android backup file
    tarfix.pl: Fix broken tar files produced by android backup when using -shared flag

    The first two routines allow for reading and writing to the standard ".ab" adb backup format.

    Backupdecrypt.pl takes an '.ab' file as input and outputs a standard format tar file (which may be optionally gzip'd).

    Backupencrypt.pl takes an arbitrary file (though typically it should be tar file) as input and outputs a standard ".ab" format backup file. Options include the ability to encrypt (or not) and deflate (or not) the backup. Also, one can automatically decompress most standard input formats before encrypting.

    For encryption, passwords can be queried for or passed on the command line or read from a file.

    NOTE: unfortunately the standard 'adb backup' routine seem to have a SEVERE *BUG* in it when using the '--shared' option in combination with certain other options.

    First, the backup is not compressed even though the header claims it is. To get around this, backupdecrypt.pl has a --nocompress option to override the header.

    Second, the encapsulated tar file is corrupted by the insertion of 4 extra bytes before every file header and before every group of 64 512-byte blocks of data.

    The third routine tarfix.pl fixes this corruption and outputs a normal readable tar file. So, if you are not able to recover a valid tar backup file using backupdecrypt.pl, try doing the following:

    backupdecrypt.pl --nocompress <backup.ab> <backupdata>
    tarfix.pl backupdata | tar xv


    Enjoy!

    NOTE: I am incredibly grateful to Nikolay Elenkov for providing sample java routines and for help in understanding the encryption formats
    3
    That sounds great, looking forward to seeing your efforts! :good:
    I've finished it!

    [GUIDE] How to extract, create or edit android adb backups

    Try restoring the backup with with the angry birds data and see if you have the savegames there. Remember to install first the game prior to restoring the adb backup (such things explained in the guide)
    2
    Fix for tarfix.pl issue

    I had ran into issues with ADB backups performed under Android 4.0.4 before the JB upgrade (on a Samsung Galaxy Nexus). I did include the shared storage (accidentally or intentionally I don't remember) and I ran into this bug (Android issue 28303; sorry, I am new here and not allowed to post outside links). Some investigation revealed that while the backup was supposedly AES encrypted and "Deflate" compressed, this was only true for the first "part" of it. At around 150 MB into the file, a simple tar archive of the SD card content was appended. ADB was unable to restore any SD card content per the bug linked to above.

    What I did to resolve:

    • Use a hex editor (HxD) to get to the start of the appended tar archive
    • Copy this part to a new ".tar" file
    • Experiment with tarfix.pl and run into the same issues as the previous poster
    • Look at the code and TAR file content and find out that 00 00 02 00 needs to be prepended to the tar file for tarfix.pl to do its job

    Key learning was that the ADB backup tool will create plaintext, corrupted tar format backups that it cannot restore. It is problematic that while the user will believe they have an encrypted backup they can restore, they actually have a plaintext backup that they cannot restore...
    2
    Thanks for your great tools! Here is some better way for unpacking?
    dd if=mybackup.ab bs=1 skip=24 | openssl zlib -d > mybackup.tar

    For packing (just need to add 24 butes to header (41 4E 44 52 4F 49 44 20 42 41 43 4B 55 50 0A 31 0A 31 0A 6E 6F 6E 65 0A)?
    openssl zlib -in mybackup.tar -out gg.ab

    Hope it helps! :good:
    1
    Mine backup version was 5 and successfully decrypted after changing the following line in the perl script:
    Code:
    my $BACKUP_VERSION = 1;