[DEPRECATED] Migrator - Android Backup Solution and Data Migration Utility

[totalnoob34]

It doesn't wipe /system, only /data. That's done while preserving most of the useful data you would want to keep (i.e., contacts, WI-FI networks, call log, Magisk modules, adoptable storage keys, and much more.).

There's one thing I need to know before enabling the 'wipe' feature by default: "what's left in /data after a TWRP factory reset (when data is encrypted)".

Someone who uses encryption would have to make a nandroid backup, save it somewhere else (OUT of /data), then wipe (not format) /data (with TWRP) and copy everything left (except /data/media) to /data/media/somefolder, so that we can check whether any encryption key is there. If there's one, you can already imagine what would happen if it's not preserved... everything in /data (inc. /data/media) would be lost!

I understand.

I just tried doing a ROM flash using adk with wipe enabled. The order of flashing was:

• adk (latest, I see it outputs what it wipes)
• ROM (which happens to include Magisk 16.6, is that a problem?)
• Gapps

I then wiped /cache and dalvik, and rebooted. First off, I didn't go through the usual setup wizard, is that normal? My settings were preserved (it's RR, so all its tweaks, background, etc), as well as Magisk modules.
Being noauto disabled, I expected applications to be automatically restored, but nothing was happening. So after a couple minutes I opened adk from the shell, and tried restoring packages that way. $e/adk/backups had not been renamed to backups.old, if that is relevant. To avoid accidental overwrites, I renamed it myself.

Anyway, trying to restore app/data (option 3) resulted in the following error:

(i) Input pattern(s) to match (i.e., sp.*fy or duk|faceb|whats. A dot matches all)... wiki

Restoring apps+data...
- org.wikipediaError: Can't open non-file: /mnt/media_rw/8580-1BEA/adk/backups.old/apk/org.wikipedia.apk
Error: java.lang.IllegalArgumentException: Unknown package: org.wikipedia

So I thought, fine, I enabled wipe and something must've survived that it shouldn't have. I then proceeded to perform a TWRP factory reset, ran through the setup wizard, installed adk via Magisk Manager, and tried restoring apps this way. No luck, still.

I'm not sure what I did wrong, I assume something went wrong with the restore process. Fear not, I pulled main.log from /data/media/adk/logs, it's attached to this post.

EDIT: I'm available to perform additional tests, feel free to PM me.

 

[VR25]

@totalnoob34, the built-in Magisk is fine.

Your settings, inc. setup wizard were preserved. That's good.

Now, looking at the log, the first thing which stands out is, your folders in /data/app are suffixed by -2OsxMlVF1Ckjp370OlVsjg==, so that the standard "/data/app/com.google.android.gms-NUMBER" is instead "/data/app/com.google.android.gms-2OsxMlVF1Ckjp370OlVsjg==". That's a major issue!

Is this a Resurrection Remix thing?

Also, did you check whether your apps+data were actually backed up to <external storages>/adk/backups?

BTW, I appreciate your willingness to perform further testing. Thanks.

 

[goldencore]

I had an old install of adk which wasn't working for me, so I uninstalled it. There is still a folder /media/adk with some files in it.
I can't delete these files even using a root explorer or the terminal with su. Anyone knows what's causing this?

 

[totalnoob34]

@totalnoob34, the built-in Magisk is fine.

Your settings, inc. setup wizard were preserved. That's good.

That's good to know.

Now, looking at the log, the first thing which stands out is, your folders in /data/app are sufixed by -2OsxMlVF1Ckjp370OlVsjg==, so that the standard "/data/app/com.google.android.gms-NUMBER" is instead "/data/app/com.google.android.gms-2OsxMlVF1Ckjp370OlVsjg==". That's a major issue!

Is this a Resurrection Remix thing?

I honestly have no idea. I can type stuff in the terminal, but I don't dig deeper than that. I suspect it's more an issue with my install, but being my G5 very poorly mantained it could be the ROM's fault as well. I just reinstalled the same ROM, and sure enough the folders in /data/app are all padded with the same -*== string. I'm moving to LOS, I'll see if the situation changes.

What you're saying is that it's installed ROM's fault, right?

Also, did you check whether your apps+data were actually backed up to <external storages>/adk/backups?

Yes, all the apks + appdata were backed up in $e/adk/backups. Do you happen to know why adk didn't rename the folder to backup.old on its own? When is it supposed to do that again?

BTW, I appreciate your willingness to perform further testing. Thanks.

That's no problem. It's also in my interest to have this module as stable as possible. The concept is an ablsolute game changer.

 

[VR25]

I had an old install of adk which wasn't working for me, so I uninstalled it. There is still a folder /media/adk with some files in it.
I can't delete these files even using a root explorer or the terminal with su. Anyone knows what's causing this?

Is the path really "/media/adk" instead of "/data/media/adk"? If so, that's crazy! I wonder how that folder would get there...
Anyway, if that's the case, run su -c mount -o remount,rw / && rm -rf /media/adk && mount -o remount,ro /.

What you're saying is that it's installed ROM's fault, right?

Yes, and that string is the issue.

Yes, all the apks + appdata were backed up in $e/adk/backups. Do you happen to know why adk didn't rename the folder to backup.old on its own? When is it supposed to do that again?

The renaming from "backups" to "backups.old" happens right after apps+data migration and before "wipe" kicks in. Why that failed is a mad question mark. The apps+data migration subshells likely crashed due to that odd suffix. However that shouldn't have interfered with "backups" migration at all. A TWRP log would reveal the missing pieces of this little puzzle.

That's no problem. It's also in my interest to have this module as stable as possible. The concept is an ablsolute game changer.

That's good to know.

P.S., I suppose your APKs were backed up with that strange suffix. This could be the reason why the manual restore failed as well.

 

[totalnoob34]

Is the path really "/media/adk" instead of "/data/media/adk"? If so, that's crazy! I wonder how that folder would get there...
Anyway, if that's the case, run su -c mount -o remount,rw / && rm -rf /media/adk && mount -o remount,ro /.

Yes, and that string is the issue.

The renaming from "backups" to "backups.old" happens right after apps+data migration and before "wipe" kicks in. Why that failed is a mad question mark. The apps+data subshell has likely crashed due to that odd suffix. However that shouldn't interfere with backups migration at all. A TWRP log would reveal the missing pieces of this little puzzle.

That's good to know.

P.S., I suppose your APKs were backed up with that strange suffix. This could be the reason why the manual restore failed as well.

I just booted a freshly installed LOS, and a ls in /data/app yields

com.topjohnwu.magisk-DnuoiSwDBMDIeIAmI2gAiw==

I'm not sure what to make of all of this, but I'm starting to wonder if all the builds for my device (LG G5, variant H850) have that suffix. I suppose you could account for this variation, it's entirely possible some other device has that funky business going on.

Can I still pull the TWRP log? I've done other wipes in the meanwhile. Where would I find it?

 

[pedritopl]

I'm not sure what to make of all of this, but I'm starting to wonder if all the builds for my device (LG G5, variant H850) have that suffix.

It's the same for me on OnePlus 3T with official Lineage 15.1.
My guess is that the suffixes might be related to the data encryption.

 

[goldencore]

Is the path really "/media/adk" instead of "/data/media/adk"? If so, that's crazy! I wonder how that folder would get there...

Sorry, I am stupid! Was trying to delete in some /emulated folder.

 

[VR25]

Sorry, I am stupid! Was trying to delete in some /emulated folder.

No worries!

It's the same for me on OnePlus 3T with official Lineage 15.1.
My guess is that the suffixes might be related to the data encryption.

The encryption, of course... that suffix could be a side effect of it. ?

I'm not sure what to make of all of this, but I'm starting to wonder if all the builds for my device (LG G5, variant H850) have that suffix. I suppose you could account for this variation, it's entirely possible some other device has that funky business going on.

Currently working on that...

Can I still pull the TWRP log? I've done other wipes in the meanwhile. Where would I find it?

You would have to go through the whole process over again...
Install an app or two, then perform the adk migration/wipe.
Before rebooting into system, grab /cache/recovery/*log* and/or /tmp/recovery.log or just go to TWRP "advanced" section > "copy logs to sdcard". See of you can also get a fresh main.log for comparison.

EDIT: @totalnoob34, install the attached zip from TWRP and use it instead of the latest on the repo. Run touch /data/.adk before the first flashing (forced upgrade). I changed some stuff to handle that odd suffix and stop right away, with a more descriptive error message if apps+data migration fails.

 

[totalnoob34]

Currently working on that...

You would have to go through the whole process over again...
Install an app or two, then perform the adk migration/wipe.
Before rebooting into system, grab /cache/recovery/*log* and/or /tmp/recovery.log or just go to TWRP "advanced" section > "copy logs to sdcard". See of you can also get a fresh main.log for comparison.

Will do. I'll do a standard TWRP factory reset. In the meanwhile, I also posted in my ROM's thread to see what the developer says.

EDIT: @VR25, the recovery.log file is attached. I'm reinstalling adk right now, main.log will follow shortly.
EDIT2: main.log is attached as well.

EDIT3: @VR25, please make a post, this editing thing is complicated)

I used the DEBUG build you posted, this is what I did:

• Installed adk DEBUG (I'll refer to it as adk from now on) in magisk, after forcing the update.
• Rebooted to TWRP, flashed adk again, no error prompts. I see that apks in $e/adk/backups/apk no longer have the infamous trailing string.
• Standard factory reset.
• Reboot to system, installed adk, and tried restoring the apps I had.

Now apps restore fine, and no additional prompts are printed. However, appdata doesn't seem to be restored. I did some additional testing with the ROM booted, and adk will restore an app fine if its data is wiped first, but it ignores (Solid Explorer) or corrupts (Titanium Backup, crashing) its data.

Definitely a step forward, I pulled main.log and attached it as mainNew.log.

 

[VR25]

EDIT3: @VR25, please make a post, this editing thing is complicated)

Indeed...

Now apps restore fine, and no additional prompts are printed. However, appdata doesn't seem to be restored. I did some additional testing with the ROM booted, and adk will restore an app fine if its data is wiped first, but it ignores (Solid Explorer) or corrupts (Titanium Backup, crashing) its data.

Definitely a step forward, I pulled main.log and attached it as mainNew.log.

Nice! We're getting there. Now I need to take a look at your /data/system/packages.xml and packages.list files, respectively. Could you upload these? Thanks.

BTW, you're still using encryption, right?

In order to speed up the debugging process and fully support encrypted data as soon as possible, I may have to encrypt my own data. Unfortunately this takes a while and I only have a single device in hand. Will see once it's "safe" to disconnect from the word for a few hours...

 

[totalnoob34]

Indeed...

Nice! We're getting there. Now I need to take a look at your /data/system/packages.xml and packages.list files, respectively. Could you upload these? Thanks.

They're attached. I renamed *.list to *.list.zip so that xda would upload it.

BTW, you're still using encryption, right?

No, I never was.

In order to speed up the debugging process and fully support encrypted data as soon as possible, I may have to encrypt my own data. Unfortunately this takes a while and I only have a single device in hand. Will see once it's "safe" to disconnect from the word for a few hours...

I get that, I'm doing this troubleshooting on my daily driver /: I don't unfortunately have any other device at hand, otherwise I'd be delighted to do that on it.

 

[VR25]

pkgName[-suffix]
@totalnoob34, could you confirm there are no such trailing strings anywhere in /data/data?

Also, where's the wizard.log file?

 

[totalnoob34]

pkgName[-suffix]
@totalnoob34, could you confirm there are no such trailing strings anywhere in /data/data?

Also, where's the wizard.log file?

Nope, no trailing edges:

h1:/ # cd /data/data/
h1:/data/data # ls
android                                com.android.inputdevices                com.android.vpndialogs                   com.vzw.apnservice
com.android.apps.tag                   com.android.inputmethod.latin           com.android.vzwomatrigger                lineageos.platform
com.android.backupconfirm              com.android.keychain                    com.android.wallpaper.livepicker         org.codeaurora.ims
com.android.bips                       com.android.location.fused              com.android.wallpaperbackup              org.lineageos.audiofx
com.android.bluetooth                  com.android.managedprovisioning         com.android.wallpapercropper             org.lineageos.eleven
com.android.bluetoothmidiservice       com.android.messaging                   com.android.wallpaperpicker              org.lineageos.jelly
com.android.calculator2                com.android.mms.service                 com.android.webview                      org.lineageos.lineageparts
com.android.calendar                   com.android.mtp                         com.google.android.apps.turbo            org.lineageos.lineagesettings
com.android.calllogbackup              com.android.nfc                         com.google.android.backuptransport       org.lineageos.lockclock
com.android.captiveportallogin         com.android.pacprocessor                com.google.android.configupdater         org.lineageos.overlay.accent.black
com.android.carrierconfig              com.android.phone                       com.google.android.ext.services          org.lineageos.overlay.accent.blue
com.android.carrierdefaultapp          com.android.printservice.recommendation com.google.android.ext.shared            org.lineageos.overlay.accent.brown
com.android.cellbroadcastreceiver      com.android.printspooler                com.google.android.feedback              org.lineageos.overlay.accent.cyan
com.android.certinstaller              com.android.providers.blockednumber     com.google.android.gms                   org.lineageos.overlay.accent.green
com.android.companiondevicemanager     com.android.providers.calendar          com.google.android.gms.setup             org.lineageos.overlay.accent.orange
com.android.contacts                   com.android.providers.contacts          com.google.android.googlequicksearchbox  org.lineageos.overlay.accent.pink
com.android.cts.ctsshim                com.android.providers.downloads         com.google.android.gsf                   org.lineageos.overlay.accent.purple
com.android.cts.priv.ctsshim           com.android.providers.downloads.ui      com.google.android.hiddenmenu            org.lineageos.overlay.accent.red
com.android.defcontainer               com.android.providers.media             com.google.android.onetimeinitializer    org.lineageos.overlay.accent.yellow
com.android.deskclock                  com.android.providers.settings          com.google.android.packageinstaller      org.lineageos.overlay.black
com.android.development                com.android.providers.telephony         com.google.android.partnersetup          org.lineageos.overlay.dark
com.android.dialer                     com.android.providers.userdictionary    com.google.android.setupwizard           org.lineageos.profiles
com.android.documentsui                com.android.proxyhandler                com.google.android.syncadapters.calendar org.lineageos.recorder
com.android.dreams.basic               com.android.sdm.plugins.sprintdm        com.google.android.syncadapters.contacts org.lineageos.settings.device
com.android.dreams.phototable          com.android.server.telecom              com.google.android.tts                   org.lineageos.settings.doze
com.android.egg                        com.android.settings                    com.keramidas.TitaniumBackup             org.lineageos.setupwizard
com.android.email                      com.android.sharedstoragebackup         com.motricity.verizon.ssodownloadable    org.lineageos.snap
com.android.emergency                  com.android.shell                       com.qualcomm.qcrilmsgtunnel              org.lineageos.trebuchet
com.android.externalstorage            com.android.smspush                     com.qualcomm.qti.telephonyservice        org.lineageos.updater
com.android.facelock                   com.android.statementservice            com.qualcomm.timeservice                 org.lineageos.weather.provider
com.android.gallery3d                  com.android.stk                         com.quicinc.cne.CNEService               pl.solidexplorer2
com.android.hotwordenrollment.okgoogle com.android.storagemanager              com.svox.pico                            qualcomm.com.vzw_msdc_api
com.android.hotwordenrollment.tgoogle  com.android.systemui                    com.topjohnwu.magisk
com.android.hotwordenrollment.xgoogle  com.android.terminal                    com.verizon.vzwavs
com.android.htmlviewer                 com.android.vending                     com.vzw.apnlib
h1:/data/data #

I attached wizard.log. You're needy, though!

 

[VR25]

Nope, no trailing edges
I attached wizard.log. You're needy, though!

Yeah, got no choice but to dive deeper
Need supplies to survive down there and you got plenty
Gotta gimme some
Don't be evil ?

 

[totalnoob34]

Yeah, got no choice but to dive deeper
Need supplies to survive down there and you got plenty
Gotta gimme some
Don't be evil

I won't, for this one time :highfive:

 

[VR25]

I won't, for this one time :highfive:

Found out all I needed to know about the sneaky trailing strings, and exactly how to handle these.

I can make the necessary changes to the manual restore part fast.

The automatic mechanism, on the other hand, will take some time.

Within the next 24 hours or so, a new version should be up.

 

[totalnoob34]

Found out all I needed to know about the sneaky trailing strings, and exactly how to handle these.

I can make the necessary changes to the manual restore part fast.

The automatic mechanism, on the other hand, will take some time.

Within the next 24 hours or so, a new version should be up.

That's very impressive. I'll check out the updated version when it hits the repo.

Thanks a lot for your dedication, this community needs people like you. Kudos, man.

EDIT: If you have the manual-only version on hand, please post it. I've set up a script and renamed the apks as their data folders, and I'm curious to see if the apps finally restore correctly before restoring the old TBK way.

 

[Sander9847]

Wow this seems like a great project!!
While reading the least pages found out I have the same problem; the weird strings on RR even though my rom was never encrypted.
Really looking forward to using/testing this once new build is out ?

 

[totalnoob34]

Wow this seems like a great project!!
While reading the least pages found out I have the same problem; the weird strings on RR even though my rom was never encrypted.
Really looking forward to using/testing this once new build is out

That's a relief, I was worrying I was the only one. Which device/ROM are you on?