[TOOLS][ZIPS][SCRIPTS]Android Backup and Restore Tools[Multiple Devices/Platforms]

Search This thread

AndDiSa

Senior Member
Dec 2, 2009
3,701
5,074
Heidelberg
HTC Desire
Nexus 7
android-backup-project
The android backup project provides a set of scripts and tools to backup and/or restore applications installed on Android devices.

This is not adb backup which didn't work for my requirements as
  • it doesn't backup applications if those are requesting not to be backed up
  • it does not support restoring to different devices quite well
  • Google warns that it might be discontinued in future versions of Android


Motivation
I never was really happy about the possibilities to backup / restore Android devices. Especially when you are developing with (different) Android devices, you wish you were able to transfer "configurations" from one device to a different one. Or you would like to revert to an older version of an app or ...

The same is valid when you change your "main" device. The Google mechanism to set up a new device from another one or from a backup works quite well, but has almost the same deficits as adb backup: it doesn't restore all of your apps and data.

Also TWRP or other custom recovery implementation do not really help to get out of this situation. It's fine for creating a backup and restoring it later on the same device but there is no support for switching devices. In addition Android often changes architecture so that TWRP support for a specific device is not guaranteed and at the time of writing there is not even TWRP support for Android 10 yet.

There is another very interesting project related to backup and migration on XDA (https://xdaforums.com/android/apps-games/app-migrate-custom-rom-migration-tool-t3862763). Unfortunately it relays on TWRP, too, and has still some issues to be solved.

Last but not least there is Titanium Backup which is available for a very long time now. It works quite well but the free version is very limited in functionality and provides only the possibility to store the backup on the device itself which is a kind of contradiction for a backup.

Some time ago I started already a similar project (https://github.com/AndDiSa/ART) to manage backup and restore of Android devices remotely. Unfortunately it never finished and due to architectural changes of Android it would need a complete workover so that I decided to start over from scratch :)


Precondition
  • adb installed and in execution path
  • Ubuntu Linux (other should work, too)
  • a rooted Android device (either by adbd unsecure or by e.g. Magisk / SuperSU)


Usage
Theses scripts kind of do that TiBackup is doing, but controlling the backup- and restore process remotely.


backup_apps.sh

./backup_apps.sh [--system-apps]

This script creates a backup of all user (and system) applications and their data. The backup files will be stored in a directory newly created named by the device and the current date.


restore_apps.sh

./restore_apps.sh [<directory_name>]

Ths script restores the apps and their data of a previous backup created by backup_apps.sh Either the directory is identified automatically by the device connected and the current date or you can pass a directory name as parameter. In that case all the apps and data found in the given directory will be restored.

If you want to restore only a part of them, please copy them into a different directory and give that directory as a parameter to the script.

Important

Keep in mind that restoring all apps may cause issues as some have unique IDs that end up causing problems if you use the same unique ID on different devices.

You also may want to consider only restoring apps that are missing from google play or apps that sadly decide to prevent backups of their (i.e. your) data.


full-backup.sh

./full-backup.sh [--data-backup][--no-data-backup][--media-backup][--no-media-backup][--image-backup][--no-image-backup]

This script does a full backup of (different parts of) the /data partition. There are several options to control the behaviour.

--data-backup will backup /data without /data/media and /data/mediadrm

--media-backup will backup /data/media and data/mediadrm

--image-backup will backup the whole partition which is mounted as /data as a 1:1 copy

As it cannot be guaranteed that during the backup process there will be no single modification on the /data partition it's quite common to get a checksum validation failure after the backup is finished.


full-restore.sh

./full-restore.sh [--data-backup][--no-data-backup][--media-backup][--no-media-backup][--image-backup][--no-image-backup] <directory_name>

This script is restoring data previously created by full-backup.sh It accepts the same parameters as full-backup.sh and takes in addition the name of the directory from where the backup shall be restored.


Important

Be very careful in unsing this method, especially when you are restoring /data It will overwrite everything on the /data partition and this may cause major issues up to the point that your device becomes no longer usable. In that case you probably need to do a full wipe to get it working again.


restore-single-partition.sh

./restore-single-partition.sh <path-to-image-to-be-restored> <name-of-the-partition-on-device>

This script is restoring a single partition image previously created by backup-partitions.sh back to the device. The image can be either in compressed or plain format. As parameters you need to give the path to the partition image and the name of the partition as is is configured on the device (see: /dev/block/by-name/<name>).

Important

This command is in experimental state and may be very dangerous as it can fully brick your device. Be very careful in using this method, you have been warned(!).


Downloads:

android-backup-project-v2.1.zip (md5: b0c71d602963696f7c372d3c2222ae75)
android-backup-project-v2.0.zip (md5: 14be02aa3e2a458ba7d7456dc152dad8)
android-backup-project-v1.6.zip (md5: 00f8dc8630b63fe1398ea61a436c2a0f)
android-backup-project-v1.5.zip (md5: 9cb1183d385f854c1f4b7858855cde58)
android-backup-project-v1.4.zip (md5: 9ad78f33a86e117a72058a336b34ab28)
android-backup-project-v1.3.1.zip (md5: 737a7d3483d98616f650d51eea033619)
android-backup-project-v1.3.zip (md5: 3bb11d4da02754a82f8614a53aacd4df)
android-backup-project-v1.2.zip (md5: 27f365663a97b3c7451938d24ffb7779)
android-backup-project-v1.1.zip (md5: c9806edb32b68109c2fc8e087927ac7a)
android-backup-project-v1.zip (md5: b754cf72fe10c89cdb8e4cb8e94f9e0a)


Special thanks
Credit goes to Raphael Moll who initiated a similar project some time ago and Marc Merlin who improved it to work with Android O. I took some ideas and inspiration from these projects and from my first trial I started years ago. The current implementation does not have much in common with neither of those versions.

Especially I would like to thank @topjohnwu for his great Magisk project and @osm0sis for his great collection of scripts and tools.

XDA:DevDB Information
Android Backup Project, Tool/Utility for all devices (see above for details)

Contributors
AndDiSa
Source Code: https://github.com/AndDiSa/android_backup_project/


Version Information
Status: Stable
Current Stable Version: V1.0
Stable Release Date: 2019-12-08

Created 2019-12-08
Last Updated 2020-01-19
 
Last edited:

AndDiSa

Senior Member
Dec 2, 2009
3,701
5,074
Heidelberg
HTC Desire
Nexus 7
Missing Permissions

Magisk Canary in combination with newer Android versions are restricting permissions even on rooted devices more and more. Due to that not all actions required to do a backup / restore are allowed any longer. To circumvent this restrictions there is a new Magisk Module available which enhances the permissions so that the backup / restore scripts are working.

If you are observing permission issues using the backup / restore scripts please install the AndroidBackupRestore Module in Magisk.

Edit: Latest Magisk Canary (>=20425) is working again without the need for additional changes
 
Last edited:

cocolol

Member
Dec 4, 2008
35
6
I there and thanks for the work.
I tried to use your script and I am stuck at "Checking root access"
I am using Magisk. Shell is autorized in #Superuser.
In command line "su" is working see #
Here is my terminal screen
flo@DESKTOP-~
$ bash full-backup.sh --data-backup --media-backup --image-backup
~/busybox-ndk ~
Déjà à jour.
~
Waiting for device...
Devices detected:
List of devices attached
da279704 device

Checking for root access...

flo@DESKTOP-~
$ adb shell
gemini:/ $ su
gemini:/ #
 

AndDiSa

Senior Member
Dec 2, 2009
3,701
5,074
Heidelberg
HTC Desire
Nexus 7
@cocolol What Android version / device are you using? I did my tests with e.g. a Magisk rooted Pixel 3a on Android 10 without any issues.

Can you try from your command line the following:

adb shell su root -c whoami

and if this is blocked the following:

adb shell su root whoami

probably for your device / configuration

adb shell su whoami

is working
 
Last edited:
  • Like
Reactions: cocolol

cocolol

Member
Dec 4, 2008
35
6
@andisa thank you for your prompt reply and sorry for my late answer
I updated script to 1.1 and still be blocked.

Thus "adb shell su root -c whoami" is working answer is "root"
But other commands didn't work (see log below)

My aim is to jump from PE 9 to PE 10 on xiaomi MI5 and MI8 (dipper) with all my apps.
I came here from Migrate app forum which is not working for me since it is TWRP dependend.

flo@DESKTOP-S8FMRD1 ~
$ bash full-backup.sh --data-backup --media-backup --image-backup
~/busybox-ndk ~
Déjà à jour.
~
Waiting for device...
Devices detected:
List of devices attached
da279704 device

Checking for root access...

flo@DESKTOP-S8FMRD1 ~
$ adb shell su root -c whoami
root

flo@DESKTOP-S8FMRD1 ~
$ adb shell su root whoami

flo@DESKTOP-S8FMRD1 ~
$ adb shell su whoami
 

AndDiSa

Senior Member
Dec 2, 2009
3,701
5,074
Heidelberg
HTC Desire
Nexus 7
@cocolol ok, that's just as it should be, i.e. if you are rooted with Magisk

adb shell su root -c

prefix should work. Question is: why it is blocked within the script.

Edit: I expect that the script is blocked while checking a sane /data partition. Can you try to remove that part from the script? It doesn't harm and we can prove that my expectations are correct. It's sufficient the remove / comment out the checkForCleanData line.
 
Last edited:

Keule-Tm

Senior Member
Oct 3, 2016
373
150
Berlin
Samsung Galaxy S10
I got an error: "Unrecognised architecture armv7l".
After adding that to line 82 in functions.sh (and switching flaky usb port) it went through.

Edit:
Device gohan (BQ Aquaris X5 plus)
OS LineageOS 14.1-20191115 (unofficial)
Host Manjaro, bash v5.0.11 (ah well, I had to install PV command)
Edit: fixed device name
 
Last edited:
  • Like
Reactions: AndDiSa

AndDiSa

Senior Member
Dec 2, 2009
3,701
5,074
Heidelberg
HTC Desire
Nexus 7
I got an error: "Unrecognised architecture armv7l".
After adding that to line 82 in functions.sh (and switching flaky usb port) it went through.

Edit:
Device gohan (BQ Aquaris X5 Pro)
OS LineageOS 14.1-20191115 (unofficial)
Host Manjaro, bash v5.0.11 (ah well, I had to install PV command)
@Keule-T Thank you very much for the info. I'll add that this change into the next version.
Would be great if others could report issues / success, too, so we can improve the scripts to be able to run on (almost) every device.
 

Oblias

Senior Member
Sep 3, 2016
498
261
Would be great if others could report issues / success, too, so we can improve the scripts to be able to run on (almost) every device.

Well, that was a disturbing experience. I downloaded and un-zipped the v1.1 on Arch Linux, opened a terminal from within that extracted folder, and ran the command ./backup_apps.sh with no additional arguments.

The terminal reported this:

Checking for root access...
Determining architecture...
Pushing busybox to device...
busybox-ndk/busybox-arm64: 1 file pushed. 26.0 MB/s (1478208 bytes in 0.054s)
### Creating dir qcom_2019-12-21_QQ1C.191205.016.A1
~/Downloads/android-backup-project-v1.1/qcom_2019-12-21_QQ1C.191205.016.A1 ~/Downloads/android-backup-project-v1.1


Then the terminal listed about 70 lines of .apk files, and then disaster happened. These are the last few lines of what the terminal reported:


## Stop Runtime
## Pull apps
package:/data/app/com.touchtype.swiftkey-80xpz-vz5Aii7U6vdFVHnQ==/base.apk=com.touchtype.swiftkey
/data/app/com.touchtype.swiftkey-80xpz-vz5Aii7U6vdFVHnQ==/base.apk
/data/app/com.touchtype.swiftkey-80xpz-vz5Aii7U6vdFVHnQ==
com.touchtype.swiftkey
./backup_apps.sh: line 55: pv: command not found


Then after that, complete silence. I mean COMPLETE. My phone did a Force Close and powered off. Wouldn't reboot back into system, just looped. I had to do a a Nandroid restore to get back into my phone again. Not good....


OnePlus 3T rooted with Magisk 20.1 and running MSM Xtended XQ v4.0 Android Q rom.

I like the concept, but I think I'll wait until you get it a bit more developed before I do a beta test again... Maybe you can make use of the above information to further your project.

---------- Post added at 12:12 PM ---------- Previous post was at 12:11 PM ----------

Forgot to add that the only thing that made it into the directory your script created on my PC was a file named

app_com.touchtype.swiftkey.tar.gz with a filesize of 0 . That's all that was there.
 
  • Like
Reactions: AndDiSa

AndDiSa

Senior Member
Dec 2, 2009
3,701
5,074
Heidelberg
HTC Desire
Nexus 7
@Oblias thank you very much for your report. I am very sorry for the mess but it's exactly those reports which help really a lot.

pv is a program which helps logging the communication between the device and the pc and I did not have in mind that it probably is not installed on a Unix system. I'll add a check for the presence of the important programs so that there is a stable basis before starting the scripts.

The normal behaviour of the script is as follows:
- shutdown runtime (to prevent modifications during the backup process)
- backup apps and data
- restart runtime

So a "warm" restart of the device is normal after the backup script finishes. I am only wondering why you had to restore a nandroid ...:confused:
 
  • Like
Reactions: ze7zez and Oblias

Oblias

Senior Member
Sep 3, 2016
498
261
@Oblias thank you very much for your report. I am very sorry for the mess but it's exactly those reports which help really a lot.

pv is a program which helps logging the communication between the device and the pc and I did not have in mind that it probably is not installed on a Unix system. I'll add a check for the presence of the important programs so that there is a stable basis before starting the scripts.

The normal behaviour of the script is as follows:
- shutdown runtime (to prevent modifications during the backup process)
- backup apps and data
- restart runtime

So a "warm" restart of the device is normal after the backup script finishes. I am only wondering why you had to restore a nandroid ...:confused:

Well, you said in the OP that one of the things needed for these scripts to run was to have "Ubuntu Linux (other should work, too)" installed. Arch Linux and Ubuntu Linux both use the Linux kernel, so I thought that would be alright. Which leaves me a bit confused as to why you said that PV is not installed on a Unix system??? But anyways.......

As far as the normal warm restart is concerned, perhaps something got messed up in the command when the script halted?? I'm not all that good with scripts, so I don't know. Also, perhaps whatever made the phone do a poweroff instead of a reboot, also corrupted some system file? Just guessing, of course... But it's not biggie really. Anyone who tries anything like this had better have a backup on hand, just in case any sort of unpredictable event happens.

So, let me know about the PV command issue, and I'll give it another try, if you'd like.

---------- Post added at 01:27 PM ---------- Previous post was at 01:24 PM ----------

Also, as an added thought, when you get a chance, perhaps a small FAQ in the OP as to what the normal behavior is when running these scripts. For example, letting folks know that their phone will do a reboot on the script completion, and things of that nature.... Just a thought....

---------- Post added at 01:31 PM ---------- Previous post was at 01:27 PM ----------

UPDATE: I also just checked my repos, and pv is available to be installed, it just wasn't installed during a 'normal' install of Arch Linux (if there is such a thing). :)

So, if I install pv, are there any other commands I should check for to see if I have installed before trying the script again?
 
  • Like
Reactions: Keule-Tm

AndDiSa

Senior Member
Dec 2, 2009
3,701
5,074
Heidelberg
HTC Desire
Nexus 7
@Oblias beside those "normal" programs like git, tar, gzip, ... which are normally always installed there is no additional program needed. But nevertheless I agree that probably some additional compatibility checks should be done especially as there are different versions of the programs existing which take probably also different parameters ...
 

Oblias

Senior Member
Sep 3, 2016
498
261
@Oblias beside those "normal" programs like git, tar, gzip, ... which are normally always installed there is no additional program needed. But nevertheless I agree that probably some additional compatibility checks should be done especially as there are different versions of the programs existing which take probably also different parameters ...

Makes sense. Alright then, I'll wait until you check things out, and once you have posted back, I'll give it another try. I don't mind some beta testing..
 
  • Like
Reactions: AndDiSa

cocolol

Member
Dec 4, 2008
35
6
@cocolol
Edit: I expect that the script is blocked while checking a sane /data partition. Can you try to remove that part from the script? It doesn't harm and we can prove that my expectations are correct. It's sufficient the remove / comment out the checkForCleanData line.

After some investigation script was blocked on line 29 of functions.sh file on this condition "if [[ $($AMAGISK whoami) == "root" ]]"
I dont know why since when I replace this line with "$AMAGISK whoami" I get "root" as answer :confused:
Thus I replace "checkRootType" directly with "AS=$AMAGISK"

But I get the same with next function
Determining architecture...
Unrecognized architecture aarch64
Seems to be does'nt get the answers same as problem above

I am using cygwin maybe it come from this.
I replaced == with =~ and it works :eek: but for the case condition I found nothing
 

AndDiSa

Senior Member
Dec 2, 2009
3,701
5,074
Heidelberg
HTC Desire
Nexus 7
@cocolol I suppose cygwin has some issues with evaluating the expressions as aarch64 is supported. Probably also here you need to change the comparison ... If you know the type, you can also execute the code for the matching architecture.
 

Top Liked Posts

  • There are no posts matching your filters.
  • 28
    android-backup-project
    The android backup project provides a set of scripts and tools to backup and/or restore applications installed on Android devices.

    This is not adb backup which didn't work for my requirements as
    • it doesn't backup applications if those are requesting not to be backed up
    • it does not support restoring to different devices quite well
    • Google warns that it might be discontinued in future versions of Android


    Motivation
    I never was really happy about the possibilities to backup / restore Android devices. Especially when you are developing with (different) Android devices, you wish you were able to transfer "configurations" from one device to a different one. Or you would like to revert to an older version of an app or ...

    The same is valid when you change your "main" device. The Google mechanism to set up a new device from another one or from a backup works quite well, but has almost the same deficits as adb backup: it doesn't restore all of your apps and data.

    Also TWRP or other custom recovery implementation do not really help to get out of this situation. It's fine for creating a backup and restoring it later on the same device but there is no support for switching devices. In addition Android often changes architecture so that TWRP support for a specific device is not guaranteed and at the time of writing there is not even TWRP support for Android 10 yet.

    There is another very interesting project related to backup and migration on XDA (https://xdaforums.com/android/apps-games/app-migrate-custom-rom-migration-tool-t3862763). Unfortunately it relays on TWRP, too, and has still some issues to be solved.

    Last but not least there is Titanium Backup which is available for a very long time now. It works quite well but the free version is very limited in functionality and provides only the possibility to store the backup on the device itself which is a kind of contradiction for a backup.

    Some time ago I started already a similar project (https://github.com/AndDiSa/ART) to manage backup and restore of Android devices remotely. Unfortunately it never finished and due to architectural changes of Android it would need a complete workover so that I decided to start over from scratch :)


    Precondition
    • adb installed and in execution path
    • Ubuntu Linux (other should work, too)
    • a rooted Android device (either by adbd unsecure or by e.g. Magisk / SuperSU)


    Usage
    Theses scripts kind of do that TiBackup is doing, but controlling the backup- and restore process remotely.


    backup_apps.sh

    ./backup_apps.sh [--system-apps]

    This script creates a backup of all user (and system) applications and their data. The backup files will be stored in a directory newly created named by the device and the current date.


    restore_apps.sh

    ./restore_apps.sh [<directory_name>]

    Ths script restores the apps and their data of a previous backup created by backup_apps.sh Either the directory is identified automatically by the device connected and the current date or you can pass a directory name as parameter. In that case all the apps and data found in the given directory will be restored.

    If you want to restore only a part of them, please copy them into a different directory and give that directory as a parameter to the script.

    Important

    Keep in mind that restoring all apps may cause issues as some have unique IDs that end up causing problems if you use the same unique ID on different devices.

    You also may want to consider only restoring apps that are missing from google play or apps that sadly decide to prevent backups of their (i.e. your) data.


    full-backup.sh

    ./full-backup.sh [--data-backup][--no-data-backup][--media-backup][--no-media-backup][--image-backup][--no-image-backup]

    This script does a full backup of (different parts of) the /data partition. There are several options to control the behaviour.

    --data-backup will backup /data without /data/media and /data/mediadrm

    --media-backup will backup /data/media and data/mediadrm

    --image-backup will backup the whole partition which is mounted as /data as a 1:1 copy

    As it cannot be guaranteed that during the backup process there will be no single modification on the /data partition it's quite common to get a checksum validation failure after the backup is finished.


    full-restore.sh

    ./full-restore.sh [--data-backup][--no-data-backup][--media-backup][--no-media-backup][--image-backup][--no-image-backup] <directory_name>

    This script is restoring data previously created by full-backup.sh It accepts the same parameters as full-backup.sh and takes in addition the name of the directory from where the backup shall be restored.


    Important

    Be very careful in unsing this method, especially when you are restoring /data It will overwrite everything on the /data partition and this may cause major issues up to the point that your device becomes no longer usable. In that case you probably need to do a full wipe to get it working again.


    restore-single-partition.sh

    ./restore-single-partition.sh <path-to-image-to-be-restored> <name-of-the-partition-on-device>

    This script is restoring a single partition image previously created by backup-partitions.sh back to the device. The image can be either in compressed or plain format. As parameters you need to give the path to the partition image and the name of the partition as is is configured on the device (see: /dev/block/by-name/<name>).

    Important

    This command is in experimental state and may be very dangerous as it can fully brick your device. Be very careful in using this method, you have been warned(!).


    Downloads:

    android-backup-project-v2.1.zip (md5: b0c71d602963696f7c372d3c2222ae75)
    android-backup-project-v2.0.zip (md5: 14be02aa3e2a458ba7d7456dc152dad8)
    android-backup-project-v1.6.zip (md5: 00f8dc8630b63fe1398ea61a436c2a0f)
    android-backup-project-v1.5.zip (md5: 9cb1183d385f854c1f4b7858855cde58)
    android-backup-project-v1.4.zip (md5: 9ad78f33a86e117a72058a336b34ab28)
    android-backup-project-v1.3.1.zip (md5: 737a7d3483d98616f650d51eea033619)
    android-backup-project-v1.3.zip (md5: 3bb11d4da02754a82f8614a53aacd4df)
    android-backup-project-v1.2.zip (md5: 27f365663a97b3c7451938d24ffb7779)
    android-backup-project-v1.1.zip (md5: c9806edb32b68109c2fc8e087927ac7a)
    android-backup-project-v1.zip (md5: b754cf72fe10c89cdb8e4cb8e94f9e0a)


    Special thanks
    Credit goes to Raphael Moll who initiated a similar project some time ago and Marc Merlin who improved it to work with Android O. I took some ideas and inspiration from these projects and from my first trial I started years ago. The current implementation does not have much in common with neither of those versions.

    Especially I would like to thank @topjohnwu for his great Magisk project and @osm0sis for his great collection of scripts and tools.

    XDA:DevDB Information
    Android Backup Project, Tool/Utility for all devices (see above for details)

    Contributors
    AndDiSa
    Source Code: https://github.com/AndDiSa/android_backup_project/


    Version Information
    Status: Stable
    Current Stable Version: V1.0
    Stable Release Date: 2019-12-08

    Created 2019-12-08
    Last Updated 2020-01-19
    4
    Uploaded version 1.2 of the tool with the following improvements:
    - support for armv7l architecture
    - checking prerequisites needed by the tool, e.g. adb, pv, ...
    4
    I got an error: "Unrecognised architecture armv7l".
    After adding that to line 82 in functions.sh (and switching flaky usb port) it went through.

    Edit:
    Device gohan (BQ Aquaris X5 Pro)
    OS LineageOS 14.1-20191115 (unofficial)
    Host Manjaro, bash v5.0.11 (ah well, I had to install PV command)
    @Keule-T Thank you very much for the info. I'll add that this change into the next version.
    Would be great if others could report issues / success, too, so we can improve the scripts to be able to run on (almost) every device.
    2
    Missing Permissions

    Magisk Canary in combination with newer Android versions are restricting permissions even on rooted devices more and more. Due to that not all actions required to do a backup / restore are allowed any longer. To circumvent this restrictions there is a new Magisk Module available which enhances the permissions so that the backup / restore scripts are working.

    If you are observing permission issues using the backup / restore scripts please install the AndroidBackupRestore Module in Magisk.

    Edit: Latest Magisk Canary (>=20425) is working again without the need for additional changes
    2
    @Oblias thank you very much for your report. I am very sorry for the mess but it's exactly those reports which help really a lot.

    pv is a program which helps logging the communication between the device and the pc and I did not have in mind that it probably is not installed on a Unix system. I'll add a check for the presence of the important programs so that there is a stable basis before starting the scripts.

    The normal behaviour of the script is as follows:
    - shutdown runtime (to prevent modifications during the backup process)
    - backup apps and data
    - restart runtime

    So a "warm" restart of the device is normal after the backup script finishes. I am only wondering why you had to restore a nandroid ...:confused: