FORUMS

[TOOLS][ZIPS][SCRIPTS]Android Backup and Restore Tools[Multiple Devices/Platforms]

3,228 posts
Thanks Meter: 4,107
 
By AndDiSa, Senior Member on 8th December 2019, 03:11 PM
Post Reply Email Thread
9th February 2020, 06:41 PM |#41  
AndDiSa's Avatar
OP Senior Member
Flag Heidelberg
Thanks Meter: 4,107
 
More
@pixiebob one question regarding your slow / (quasi blocked) transfer speed: What kind of USB connection do you use? USB 2.0 or USB 3.x? I heared from some people that they had connection issues and after changing the USB port the connection was much better.
9th February 2020, 10:47 PM |#42  
Senior Member
Thanks Meter: 119
 
More
Quote:
Originally Posted by AndDiSa

@pixiebob one question regarding your slow / (quasi blocked) transfer speed: What kind of USB connection do you use? USB 2.0 or USB 3.x? I heared from some people that they had connection issues and after changing the USB port the connection was much better.

incredible you are right, first time i use usb3, and after your response i tried with usb2 and this time that's working(no more stuck) get high transfer speed.

thank you, i can now backup all my app

edit: is there a way using backup.app.sh to backup only user app+data and not system app?
The Following User Says Thank You to pixiebob For This Useful Post: [ View ] Gift pixiebob Ad-Free
10th February 2020, 10:00 AM |#43  
AndDiSa's Avatar
OP Senior Member
Flag Heidelberg
Thanks Meter: 4,107
 
More
@pixiebob if you omit the --system-app flag only user app+data are regarded ... but if you e.g. update a system app like Chrome / Maps / ... which is pre-installed on system/app but due to the update is installed on /data/app it's regarded, too. I'll need to see whether there is a chance to filter out "updated system apps".
11th February 2020, 07:24 PM |#44  
Junior Member
Thanks Meter: 2
 
More
Quote:
Originally Posted by AndDiSa

strange, looks like the data transfer between the device and your PC is blocked for whatever reason. You have an adb connection via USB or WLAN?

First of all, let me express my deepest gratitude for your tool and effort
After about a week trying - your toolkit was the first one to finally backup the /data directory into .tar
That's just great really
it works, can't really believe

Why am I so pessimistic?
Due to Android 10 (Q) - it's a harsh challenge to do anything 'extra' in this system

But, [was to be expected]
I failed to restore that backup
here's the terminal output:

[email protected]:~/Downloads/bac/android-backup-project-v1.3.1$ ./full-restore.sh --data-backup mt6771_2020-02-11_QP1A.190711.020
parm: mt6771_2020-02-11_QP1A.190711.020
~/Downloads/bac/android-backup-project-v1.3.1/busybox-ndk ~/Downloads/bac/android-backup-project-v1.3.1
Already up to date.
~/Downloads/bac/android-backup-project-v1.3.1
Waiting for device...
Devices detected:
List of devices attached
VFW7HV87WGVO37GO87X device

Checking for root access...
Determining architecture...
Pushing busybox to device...
busybox-ndk/busybox-arm64: 1 file pushed. 21.0 MB/s (1478208 bytes in 0.067s)
restoring from mt6771_2020-02-11_QP1A.190711.020
## Stop Runtime
~/Downloads/bac/android-backup-project-v1.3.1/mt6771_2020-02-11_QP1A.190711.020 ~/Downloads/bac/android-backup-project-v1.3.1
Restoring full tar backup of /data excluding /data/media ...
3.12MiB 0:00:00 [49.9MiB/s] [49.9MiB/s]
[email protected]:~/Downloads/bac/android-backup-project-v1.3.1$



So, as we can see the phone refused to accept the data back
Btw phone's reaction was: going to black screen, but still illuminated
So I pressed the power button for some time and it successfully rebooted back to system without any bricks/bootloops, but without any backup restored as well [obviously]
I understand it's the phone (ie mine) problem, not the tool's

But maybe you can help me (and, suppose, 1000's of others) somehow with the restoration process?

I saw something about avd manager in this thread, but it seems to concern emulating, so it doesn't help to restore the backup physically in the device, right?

As I see it [if you let me, of course]: would be just great if you could adapt
./full-restore.sh --data-backup
output to a twrp-backup archive (which is data.ext4.win for /data partition)
I'm saying that, because you are just in a touching distance to make the only working solution in the world for 'backup-restore /data partition' process for Android 10 (Q)
Just imagine

As I heard twrp-backups are also .tar archives (like yours)
But they are kinda tricky .tar archives, and I didn't manage to mount/unpack them properly [doesn't mean it's impossible, of course]

Maybe you can dig somehow in this way? I mean twrp-style backup? So that we are able to flash it via TWRP's Restore button. Or maybe flashable .zip.. I don't know
Or
Maybe you can advice another working method of restoring your backup physically to phone?

I've tried to execute
./full-restore.sh --data-backup
while phone in TWRP (adb recovery), but received
/sbin/sh: su: not found
and same for all 3 check commands (adb shell su root -c whoami / etc.)

So, maybe you can adapt not the output, but add extra command for execution full-restore while in TWRP (adb recovery)?

Anyway, would appreciate any help or advice, because as for now - there's no working solution to backup-restore /data partition for Android 10 (or maybe I don't know something)
And as I can see - yours the only managed to backup it at least (in a normal .tar archive with kept (saved) permissions)

So I'll keep my fingers crossed concerning your project, because it looks very promising

Android 10 (Q) and backup problems

There's no more possibility in Android 10 to mount rw vendor and system partitions, therefore no possibility to make any changes in this partitions thru phone while rooted.
More than that /system, /vendor and newly added /product (which is also unmountable rw) are now making up one big super.img (no separate .img's for system and vendor anymore)
At least in my phone (specs below)
So, all that means that I can't edit the /vendor/etc/fstab.*processor_type to turn the auto-encryption off, even if I'm rooted
All magisk options (modules) aiming to disable encryption (which patching the same fstab file, but in boot partition) are useless, ie the phone is still encrypted, although modules are working and the boot's fstab is patched
Obviously, need to patch vendor's fstab, but there's no possibility for that from inside the phone

Neither from outside
As I couldn't find any kitchen, which could be able to operate with Android 10 img's
I managed to unpack super.img to 3 separate img's: system, vendor and product
Managed to mount/unpack each of them
But failed to (a) make any editions while mounted or (b) repack edited directory (even untouched directory won't pack back, saying there's lack of memory all the time, even if I manually increase the size - same problem while mounted, even under root in Ubuntu - not enough memory, though it's pretty enough there)

So, as there's no option to turn off the auto-encryption - there's no option to backup /data using TWRP, because most of the partition stays encrypted for TWRP (ie some dBCDsycb7sbc87BsDCS instead of files), although /sdcard is readable which is quite surprising

All that means: no nandroid backup for Android 10 (at least for my device)

Until I found this thread, thank you again, @AndDiSa, you did a great job, and now there is such an option - nandroid for Q
The only thing left is to restore it


My input parameters (models/versions/etc):

Phone:
Umidigi f2 (stock rom + magisk 20.3 + twrp 3.3.1-0 (beta, as no official twrp support for Q))
PC:
Ubuntu 19.10 (everything needed installed, including git pv etc.)
The Following User Says Thank You to bennibe For This Useful Post: [ View ] Gift bennibe Ad-Free
11th February 2020, 08:30 PM |#45  
AndDiSa's Avatar
OP Senior Member
Flag Heidelberg
Thanks Meter: 4,107
 
More
@bennibe hugh ... what a post I'll need some time to answer all your questions / remarks / suggestions, but let me give a short reply now:
- I would like to renounce on TWRP completely (even TWRP backup formats, etc.) as TWRP is not supported for Android 10 (yet) and I expect that TWRP on Android 10 (if there will be one in future) will not provide the same functionality as we are used to
- restoring the full tar should work but I will test it again. Up to now I was more concentrated on backing up apps+data as the rest of the /data partition can be copied easily on other ways
- regarding editing or changing /system, /vendor, ... : I suppose there will be no chance in future. They are signed and any modification will be detected and this will lead to a boot failure. That's also the reason why Magisk is bind-mounting the partitions / directories where modifications are needed.
- backing-up /data, /system, ... partitions can be done by a 1:1 image which then can be restored only in the same way. Modifications of those images will (most likely) lead to a boot failure
- regarding backup-restore of single apps and their data the situation is a bit different: As the system is up and running while doing the backup, the data can be accessed in an unencrypted way and so extracting apps and data is done in an unencrypted way, too (which might be a security issue ... but on the other side you already have the device under your control)

Nevertheless there are still many issues to be resolved. Naming only some of them: different partitioning and partition naming for different devices, dynamic partitions in Android 10, compatibility with Windows, ...
The Following User Says Thank You to AndDiSa For This Useful Post: [ View ] Gift AndDiSa Ad-Free
12th February 2020, 02:06 PM |#46  
Junior Member
Thanks Meter: 2
 
More
Thanks for fast reply

Quote:
Originally Posted by AndDiSa

- I would like to renounce on TWRP completely

Ok, got you, and it makes sense, I must admit
But, in case you and anybody else is interested, here's what I personally figured out about your tool and TWRP backup restoration:
your
./full-backup.sh --data-backup
output is basically twrp backup archive already
All your need is to rename the archive from data.tar.gz to data.ext4.win, and put that archive in /sdcard/TWRP/BACKUPS/*anyname*/data.ext4.win [case sensitive]
Then, after tapping 'Restore' button in TWRP, this archive will be recognized as a data partition backup under *anyname*, and can be successfully restored
But not for me
Under 'successfully' I mean, that all files and folders are at their places in /data directory (and pretty sure with right set permissions, didn't check)
But my phone (umi f2) won't boot after such a restoration, throwing me back to TWRP..
Guess my plans became quite impossible finally, don't know whether it depends on Q or especially on my device
Guess I just need to close that old-twrp-chapter and try to open the new-Q one, where, there's no nandroid backup-restore.. at least by now

Quote:
Originally Posted by AndDiSa

rest of the /data partition can be copied easily on other ways

Could you please be more specific?
I haven't yet tried your
./backup_apps.sh
will try it today
Does it also keep magisk with all it modules and settings inside? [something tells me 'not']
Frankly speaking, that's is all I really need - magisk set with all modules and specified settings inside that modules
So just to make one (any kind of) installation instead of 5-10 each time
Maybe you can advice me such a solution?
Or, maybe all the modules (and their settings, at least some modules allow to pre-configure them via some script editing) can be compiled in 1 big zip, and flashed at once?
I do understand you're not a magisk developer, sorry for being annoying on that, but you know and understand so much, couple (or may be 10?) levels deeper than mine, that's why I can't miss such a chance, sorry


Quote:
Originally Posted by AndDiSa

be no chance in future

yeah.. seems so

Quote:
Originally Posted by AndDiSa

They are signed and any modification will be detected and this will lead to a boot failure

But what about custom ROMs? They exist for Q, all that treble/gsi stuff
that means anybody also can build a one, doesn't it?
I just don't have enough competence, suppose kitchen for Q is not soon to come, and all is left - build thru sdk, android studio and all that..
that's already couple steps ahead of me at least today

Quote:
Originally Posted by AndDiSa

bind-mounting

that should mean something like virtual/parallel mounting, right?

Quote:
Originally Posted by AndDiSa

1:1 image

so that should be approx 120gb image for userdata.img, right? (the phone is 128GB memory)
yeah, that also could be done thru SP Flash tool for MTK-devices, but that's something not that convenient, you know


So, anyway, thanks again
really appreciate your detailed answer
any advice regarding magisk restoration/unification will be also very nice
The Following User Says Thank You to bennibe For This Useful Post: [ View ] Gift bennibe Ad-Free
13th February 2020, 07:08 PM |#47  
AndDiSa's Avatar
OP Senior Member
Flag Heidelberg
Thanks Meter: 4,107
 
More
@bennibe I just checked the full-backup.sh and full-restore.sh and could not find an issue for the --data-backup yet. To be clear, I did the backup in the following way:

./full-backup.sh --data-backup

and then the restore with

./full-restore.sh --data-backup <dir>

while <dir> is the directory where the backup was created in the step before. Giving the following log:

Code:
parm: ranchu_2020-02-13_QSR1.191030.002
/media/ds/7d13d1f7-be33-4667-9b16-839f3c0f49e4/src/android-backup-project/busybox-ndk /media/ds/7d13d1f7-be33-4667-9b16-839f3c0f49e4/src/android-backup-project
Bereits aktuell.
/media/ds/7d13d1f7-be33-4667-9b16-839f3c0f49e4/src/android-backup-project
Waiting for device...
Devices detected:
List of devices attached
emulator-5554	device

Checking for root access...
su: invalid uid/gid '-c'
Determining architecture...
Pushing busybox to device...
busybox-ndk/busybox-x86_64: 1 file pushed. 136.1 MB/s (1670936 bytes in 0.012s)
restoring from ranchu_2020-02-13_QSR1.191030.002
## Stop Runtime
/media/ds/7d13d1f7-be33-4667-9b16-839f3c0f49e4/src/android-backup-project/ranchu_2020-02-13_QSR1.191030.002 /media/ds/7d13d1f7-be33-4667-9b16-839f3c0f49e4/src/android-backup-project
Restoring full tar backup of /data excluding /data/media ... 
 134MiB 0:00:03 [39,2MiB/s] [39,2MiB/s]
SELinux: Loaded file_contexts
SELinux:  Relabeling /data/data/com.google.android.youtube/cache/cronet_metadata_cache/version from u:object_r:app_data_file:s0 to u:object_r:app_data_file:s0:c120,c256,c512,c768.
SELinux:  Relabeling /data/data/com.google.android.youtube/cache/cronet_metadata_cache/prefs/local_prefs.json from u:object_r:app_data_file:s0 to u:object_r:app_data_file:s0:c120,c256,c512,c768.
...
## Restart Runtime
16th February 2020, 05:57 PM |#48  
AndDiSa's Avatar
OP Senior Member
Flag Heidelberg
Thanks Meter: 4,107
 
More
I've just updated the project on GitHub and I've added a link to version 1.4 in the OP.
Most important changes:
- support for restoring an image of the data partition (i.e. data.img.gz) which was not supported yet

Please note: this feature is still experimental. I've successfully restored the full image on a HTC one (m8) running LineageOS 17.1 but other devices / OS versions need to be done. Please report your experience!

Syntax is for creating an image:

./full-backup.sh --image-backup

and

./full-restore.sh --image-backup <dir>

for restoring while <dir> is the directory where the backup file was created in the step before.

Please note:you should restore an image only on the same device running the same OS as you have done the backup.
The Following User Says Thank You to AndDiSa For This Useful Post: [ View ] Gift AndDiSa Ad-Free
23rd February 2020, 12:07 PM |#49  
AndDiSa's Avatar
OP Senior Member
Flag Heidelberg
Thanks Meter: 4,107
 
More
I've just updated the project on GitHub and I've added a link to version 1.5 in the OP.

Most important changes:
- support for restoring a single partition image back to the device

Please note: this feature is still experimental. I've successfully restored several images back to my devices but as there is the risk for bricking be very careful.

Syntax is for restoring an image:

./restore-single-partition.sh <path-to-image> <name-of-the-partition>

Please note:you should restore an image only on the same device running the same OS as you have done the backup.
The Following User Says Thank You to AndDiSa For This Useful Post: [ View ] Gift AndDiSa Ad-Free
24th February 2020, 04:22 PM |#50  
Senior Member
Thanks Meter: 119
 
More
Quote:
Originally Posted by AndDiSa

I've just updated the project on GitHub and I've added a link to version 1.5 in the OP.

Most important changes:
- support for restoring a single partition image back to the device

Please note: this feature is still experimental. I've successfully restored several images back to my devices but as there is the risk for bricking be very careful.

Syntax is for restoring an image:

./restore-single-partition.sh <path-to-image> <name-of-the-partition>

Please note:you should restore an image only on the same device running the same OS as you have done the backup.

thanks you for the update especially for the ability to backup image data partition, will try soon and report.

edit here the result of backup image data partition, there are some error about gzip is it problematic, and checksum don't match?

./full-backup.sh --image-backup
~/Téléchargements/android-backup-project-v1.5/busybox-ndk ~/Téléchargements/android-backup-project-v1.5
Déjà à jour.
~/Téléchargements/android-backup-project-v1.5
Waiting for device...
Devices detected:
List of devices attached
R28M2270H9M device

Checking for root access...
Determining architecture...
Pushing busybox to device...
busybox-ndk/busybox-arm64: 1 file pushed. 19.1 MB/s (1478208 bytes in 0.074s)
### Creating dir qcom_2020-02-24_QP1A.190711.020
~/Téléchargements/android-backup-project-v1.5/qcom_2020-02-24_QP1A.190711.020 ~/Téléchargements/android-backup-project-v1.5
## Stop Runtime
Creating full tar backup of /data excluding /data/media
3,12GiO 0:06:44 [7,90MiB/s] [7,90MiB/s]
Creating image backup...
gzip: read: Permission denied
gzip: gzclose: Permission denied

gzip: stdin: unexpected end of file
0,00 O 0:00:00 [0,00 B/s] [0,00 B/s]
Verifying image backup...
Calculate checksum on device:
feebeccf5447a849271db8f78101d6b5dffd3087c6d409651e f5586091edae5e
Calculate checksum locally: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca4 95991b7852b855
Checksums don't match! e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca4 95991b7852b855 != feebeccf5447a849271db8f78101d6b5dffd3087c6d409651e f5586091edae5e
## Restart Runtime
24th February 2020, 05:33 PM |#51  
AndDiSa's Avatar
OP Senior Member
Flag Heidelberg
Thanks Meter: 4,107
 
More
@pixiebob that doesn't look very good. Creating the tar should have been successful but the image was not extracted. For whatever reason it seems as if the permissions are missing to read the /data partition. You can try to use the script backup-partitions.sh to see whether this works. There is a chance as the partitions are identified differently. full-backup.sh searches the mount point for /data while backup-partitions.sh looks at /dev/block/by-name/...
But there is a disadvantage currently: there is no filter currently, i.e. *all* partitions are read and copied to the PC ... which may take some time.

Edit: if you use the following full-backup.sh you will see the partition name the script is trying to read ...
Code:
#!/bin/bash
# License; Apache-2
# [email protected] 2019/12

curr_dir="$(dirname "$0")"
. "$curr_dir/functions.sh"

set -e   # fail early

use_adb_root=false
data_backup=true
media_backup=false
image_backup=false
extra_backup=false

if [[ $# -gt 0 ]]; then
    for param in [email protected]; do
        case "$param" in
            help|-h|--help)
                echo "Makes a full backup over ADB"
                echo "tar /data, binary img /data block"
                exit 0
                ;;
            --data-backup)
                data_backup=true
                ;;
            --no-data-backup)
                data_backup=false
                ;;
            --media-backup)
                media_backup=true
                ;;
            --no-media-backup)
                media_backup=false
                ;;
            --image-backup)
                image_backup=true
                ;;
            --no-image-backup)
                image_backup=false
                ;;
            --extra-backup)
                extra_backup=true
                ;;
            --no-extra-backup)
                extra_backup=false
                ;;
            *)
                echo "Unknown argument $1"
                exit 1
                ;;
        esac
    done
fi

checkPrerequisites

updateBusybox

lookForAdbDevice

checkRootType

checkForCleanData

pushBusybox

mkBackupDir
pushd "$DIR"

stopRuntime

if $data_backup; then
    echo "Creating full tar backup of /data excluding /data/media"
    if [[ "$AS" == "$AMAGISK" ]]; then
    	$AS '"cd /data && /dev/busybox tar -czf - --exclude=media --exclude=mediadrm ./ 2>/dev/null"' | pv -trabi 1 > data.tar.gz
    else
    	$AS '/dev/busybox tar -cv -C /data --exclude="/data/media" --exclude="/data/mediadrm" . | gzip' | gzip -d | pv -trabi 1 | gzip -c9 > data.tar.gz
    fi
fi

if $media_backup; then
    echo "Creating full tar backup of /data/media"
    if [[ "$AS" == "$AMAGISK" ]]; then
    	$AS '"cd /data/media && /dev/busybox tar -czf - ./ 2>/dev/null"' | pv -trabi 1 > data_media.tar.gz
    	$AS '"cd /data/mediadrm && /dev/busybox tar -czf - ./ 2>/dev/null"' | pv -trabi 1 > data_mediadrm.tar.gz
    else
    	$AS '/dev/busybox tar -cv -C /data/media . 2>/dev/null | gzip' | gzip -d | pv -trabi 1 | gzip -c9 > data_media.tar.gz
    	$AS '/dev/busybox tar -cv -C /data/mediadrm . 2>/dev/null | gzip' | gzip -d | pv -trabi 1 | gzip -c9 > data_mediadrm.tar.gz
    fi
fi

if $image_backup; then
    echo "Creating image backup..."
    #get data image location
    PARTITION=$($AS mount | grep " /data " | cut -d ' ' -f1)
    echo "trying to get $PARTITION as data.img.gz"
    $AS "/dev/busybox dd if=$PARTITION bs=16777216 2>/dev/null | gzip" | gzip -d | pv -trabi 1 | gzip -c9 > data.img.gz

    echo "Verifying image backup..."
    echo -n "  Calculate checksum on device: "
    device_checksum="$($AS /dev/busybox sha256sum $PARTITION | cut -d ' ' -f1)"
    echo "$device_checksum"
    echo -n "  Calculate checksum locally: "
    local_checksum="$(gzip -d < data.img.gz | sha256sum | cut -d ' ' -f1)"
    echo "$local_checksum"

    if [ "$local_checksum" == "$device_checksum" ]; then
        echo "Checksums match."
    else
        echo -e "\033[1mChecksums don't match! $local_checksum != $device_checksum\033[0m"
    fi
fi

cleanup

startRuntime

popd # $DIR
Post Reply Subscribe to Thread

Guest Quick Reply (no urls or BBcode)
Message:
Previous Thread Next Thread
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes