[APP][ROOT][5.0+][1st Nov 2020] Migrate - custom ROM migration tool

Is Migrate backup and restore working without any errors?


  • Total voters
    971
Search This thread

twu2

Senior Member
Oct 25, 2009
177
281
Taipei
blog.teatime.com.tw
@SayantanRC, I got TWRP error 6 for some backup.
When I check updater-script, it include some non-English word in app name (wrong encoding) in ui_print().
So.. it's better just use English here, this will avoid error 6 in TWRP.

I remove the wrong encoding word from update-script, the backup zip file work fine for TWRP now.
 
  • Like
Reactions: naren12

twu2

Senior Member
Oct 25, 2009
177
281
Taipei
blog.teatime.com.tw
@SayantanRC, I got TWRP error 6 for some backup.
When I check updater-script, it include some non-English word in app name (wrong encoding) in ui_print().
So.. it's better just use English here, this will avoid error 6 in TWRP.

I remove the wrong encoding word from update-script, the backup zip file work fine for TWRP now.
 
  • Like
Reactions: naren12

Wishmasterflo

Senior Member
Nov 21, 2018
1,788
1,435
Stockholm
OnePlus 8T
Google Pixel 7a
When I try to flash a Migrate_Backup.zip file in TWRP (on a Android 10 custom ROM Oneplus6). I get error 6 as well.
No luck in flashing back a Backup. I tried with several different Backup files created with different Versions of Migrate.

What is that AWK ? Something with/for Busybox?

Here some extract from the recovery.log file:

Code:
[...]
/tmp/prep.sh[311]: can't create /proc/self/fd//dev/null: No such file or directory
/tmp/prep.sh[311]: awk: not found
DEBUG:: --- df output /data ---
linker: error reading config file "/system/etc/ld.config.txt" for "/sbin/toybox" (will use default configuration): error reading file "/system/etc/ld.config.txt": Too many symbolic links encountered
WARNING: linker: error reading config file "/system/etc/ld.config.txt" for "/sbin/toybox" (will use default configuration): error reading file "/system/etc/ld.config.txt": Too many symbolic links encountered
run_program: child exited with status 1
script aborted: Exiting...
Exiting...
Updater process ended with ERROR: 7
I:Legacy property environment disabled.
I:Install took 5 second(s).
Error installing zip file '/sdcard/Install/Migrate_Backup_20-11-2019/Part_1_of_2.zip'
Updating partition details...
[...]

EDIT: OK I removed now the part with the free space checking and AWK from the script and after that I could successfully flash my Backup.zip file(es) in TWRP without any error 7 but then when restarting my Phone the Migrate Helper app has not found anything to restrore. I dont know why.
I took a look in the following folders and all of them were empty:

SDCARD/Android/Data/balti.migrate
SDCARD/Android/Data/balti.migrate.helper

also the subfolder CACHE within these folders is empty.
Are these the folders where the flashed Migrate_Backup.zip data is put or where is that located?
 
Last edited:
  • Like
Reactions: takobaba

Yeusepe

Member
Dec 2, 2016
42
5
Problems with missing files in Migrate Helper

So I'm trying to install a backup, and everything goes fine, no problems with TWRP or the installation. But, when I go and try to restore the backup, the app prompts me that there are missing files in data/user/baiti.migratehelper/files/ being those initSu.sh (1), initSu.sh (2), and initSu.sh (3).

Help
 

BillGoss

Senior Member
Sep 2, 2010
5,910
5,242
Sydney
OnePlus 8T
Google Pixel 8 Pro
When I try to flash a Migrate_Backup.zip file in TWRP (on a Android 10 custom ROM Oneplus6). I get error 6 as well.
No luck in flashing back a Backup. I tried with several different Backup files created with different Versions of Migrate.

What is that AWK ? Something with/for Busybox?

Here some extract from the recovery.log file:



EDIT: OK I removed now the part with the free space checking and AWK from the script and after that I could successfully flash my Backup.zip file(es) in TWRP without any error 7 but then when restarting my Phone the Migrate Helper app has not found anything to restrore. I dont know why.
I took a look in the following folders and all of them were empty:

SDCARD/Android/Data/balti.migrate
SDCARD/Android/Data/balti.migrate.helper

also the subfolder CACHE within these folders is empty.
Are these the folders where the flashed Migrate_Backup.zip data is put or where is that located?
Rather than remove the free space check (it'll assume there's none and not extract the backed up data), replace every instance of "awk" in the script files (*.sh) with "/tmp/busybox awk". Then it should work.

It seems that not every version of TWRP includes awk.

Busybox is a collection of Unix utilities. Awk is one of the utilities and is a file editor, similar to sed, but more versatile.

Sent from my OnePlus3T using XDA Labs
 

Wishmasterflo

Senior Member
Nov 21, 2018
1,788
1,435
Stockholm
OnePlus 8T
Google Pixel 7a
Rather than remove the free space check (it'll assume there's none and not extract the backed up data), replace every instance of "awk" in the script files (*.sh) with "/tmp/busybox awk". Then it should work.

It seems that not every version of TWRP includes awk.

Busybox is a collection of Unix utilities. Awk is one of the utilities and is a file editor, similar to sed, but more versatile.

OK I did that now but it fails when trying to move the files to the migrate\cache folder:

mv: bad '/data/migrateTemp/org.thoughtcrime.securesms.app': Operation not permitted

so that error Message I see then for all my apps and later on the error "unpack failed" I guess due to the above error.

What is causing that the mv command fails?
 
Last edited:

BillGoss

Senior Member
Sep 2, 2010
5,910
5,242
Sydney
OnePlus 8T
Google Pixel 8 Pro
OK I did that now but it fails when trying to move the files to the migrate\cache folder:

mv: bad '/data/migrateTemp/org.thoughtcrime.securesms.app': Operation not permitted

so that error Message I see then for all my apps and later on the error "unpack failed" I guess due to the above error.

What is causing that the mv command fails?
Zip up the recovery log and the modified shell scripts and I'll have a look.

Sent from my OnePlus3T using XDA Labs
 

BillGoss

Senior Member
Sep 2, 2010
5,910
5,242
Sydney
OnePlus 8T
Google Pixel 8 Pro
Here the recovery log.
Hmm. I see two issues which may or may not be related.

1. The command "getprop ro.product.cpu.abi" in the prep.sh script doesn't return anything as seen in the following output (nothing showing after Found: )
ui_print Original CPU ABI was arm64-v8a
ui_print Found:
ui_print Restoration of some apps MAY FAIL!
2. The many lines from the mover.sh like:
mv: bad '/data/migrateTemp/balti.migrate.app': Operation not permitted
are related, from my limited Google search, to a permission problem.

Unfortunately I can't help with this.

It seems to me that the context in which the scripts run might not have root access. But I could be totally wrong.

The endless lines of ' error reading config file "/system/etc/ld.config.txt" ' show up for me also in TWRP when an Android 10 system is mounted. But they don't appear to cause problems.

Anybody else have any suggesting as to why a "mv" command gets "Operation not permitted"?

Sent from my OnePlus3T using XDA Labs
 
  • Like
Reactions: Wishmasterflo
G

GuestK00376

Guest
Hmm. I see two issues which may or may not be related.

1. The command "getprop ro.product.cpu.abi" in the prep.sh script doesn't return anything as seen in the following output (nothing showing after Found: )
ui_print Original CPU ABI was arm64-v8a
ui_print Found:
ui_print Restoration of some apps MAY FAIL!
2. The many lines from the mover.sh like:
mv: bad '/data/migrateTemp/balti.migrate.app': Operation not permitted
are related, from my limited Google search, to a permission problem.

Unfortunately I can't help with this.

It seems to me that the context in which the scripts run might not have root access. But I could be totally wrong.

The endless lines of ' error reading config file "/system/etc/ld.config.txt" ' show up for me also in TWRP when an Android 10 system is mounted. But they don't appear to cause problems.

Anybody else have any suggesting as to why a "mv" command gets "Operation not permitted"?
Likely due to the dynamic partitions that were introduced as part of Android 10.
Depending on the device (the 7T experiences this badly), much of the internal storage is not converted until init.

For this reason, a lot of TWRP based modules are now moving to Magisk, due to completing it's tasks around init.
 

Wishmasterflo

Senior Member
Nov 21, 2018
1,788
1,435
Stockholm
OnePlus 8T
Google Pixel 7a
Likely due to the dynamic partitions that were introduced as part of Android 10.
Depending on the device (the 7T experiences this badly), much of the internal storage is not converted until init.

For this reason, a lot of TWRP based modules are now moving to Magisk, due to completing it's tasks around init.

OK I see. Thanks. I have a Oneplus 6.
But at least the manual Backup extract works here and I can restore my Migrate Backup this way.
 

str8str

Senior Member
Apr 1, 2013
3,921
1,046
Kitchener Ontario
Excellent, thanks!

A missing awk or a version of awk that behaves differently will cause the script to exit with an error and TWRP gives an error 7. I'm glad the BB one worked.

@SayantanRC I have a couple of suggestions to make:
1. Use the include busybox for the awk calls (/tmp/busybox awk ...) to ensure consistency across versions of TWRP.
2. Use a shell script update-binary (and a dummy updater-script) instead of the edify binary version. This allows OUTFD to be exported in update-binary and used in the other shell scripts that are called. Then the echoIt function will work if it's defined as:


These two changes would eliminate most(?) of the error 7 and error 1 problems and produce good output to the TWRP display and logs. I've attached a sample update-binary for your consideration.
So where do I put this or how to use it
 

BillGoss

Senior Member
Sep 2, 2010
5,910
5,242
Sydney
OnePlus 8T
Google Pixel 8 Pro
So where do I put this or how to use it
I'm not providing a solution - that's up to the developer to do. I'm just identifying the cause of the errors and possible solutions.
If you're not familiar with shell scripts and how to modify them, then you're best waiting for an updated version of Migrate.
Also, to fix an individual problem depends on the errors you're getting and then modifying the scripts in your backup zips.

Sent from my OnePlus3T using XDA Labs
 

1aladdin1

Senior Member
Nov 11, 2008
145
37
Austria
OnePlus 7T
so prep.sh calls /system/bin/awk and ... it's not there.

Reason is the specific recovery version I used.
At the moment official TWRP recovery is not ready for Android 10 - as described by Dees_Troy here
So I took a special system-as-root ready version (needed for Android 10), developed by XDA Senior Member Captain Throwback for my device HTC M8 - see here

In this recovery system partition is mounted at /system_root and not at /system
/system is a link to /system_root/system and the latter doesn't even exist

Excellent, thanks!

A missing awk or a version of awk that behaves differently will cause the script to exit with an error and TWRP gives an error 7. I'm glad the BB one worked.

@SayantanRC I have a couple of suggestions to make:
1. Use the include busybox for the awk calls (/tmp/busybox awk ...) to ensure consistency across versions of TWRP.
2. Use a shell script update-binary (and a dummy updater-script) instead of the edify binary version. This allows OUTFD to be exported in update-binary and used in the other shell scripts that are called. Then the echoIt function will work if it's defined as:
Code:
echoIt() {
    echo "ui_print $1" > ${OUTFD}
}
These two changes would eliminate most(?) of the error 7 and error 1 problems and produce good output to the TWRP display and logs. I've attached a sample update-binary for your consideration.

Sent from my OnePlus3T using XDA Labs
Finally I got all mtest*.zip running in the specific recovery version I use.
I placed a symlink
Code:
/system -> /system_root/
in the root directory. Then all awk calls were redirected accordingly and installation of zips finished without errors.


I also asked the dev of the particular recovery I use @Captain_Throwback and got this answer:
/system_root is the correct mount point for SAR devices. There is a symlink from /system_root/system to /system by default, but in order for zip installations to succeed, it has to be deleted. The actual location of the binary is /system_root/system/bin. There is no adjustment to be made here. The proper way to determine where the binaries should be located is to check the path of the ANDROID_ROOT variable, and if it's system, use system/bin, but if it's system_root, use the longer path mentioned above. That's pretty basic logic which any app should be able to accommodate
 
Last edited:

Top Liked Posts

  • There are no posts matching your filters.
  • 616
    Jumping from one custom ROM to another can be a challenging task. So I decided to make an app to simplify the process.

    Introducing

    Migrate!

    Migrate-GPE (v4.0) is on Google Play
    It was temporarily suspended. Read more here: https://xdaforums.com/showpost.php?p=82230451&postcount=1273

    Download from Google Play Store
    Download from AndroidFileHost
    Download from XDA labs (removed as XDA Labs has shutdown)

    CODEBASE ON GITHUB


    migrate_1.png


    With Migrate, you can back up:
    1. App apk
    2. App data
    3. App runtime permissions
    4. Contacts (as vcf file)
    5. SMS messages
    6. Call logs
    7. Current ROM screen DPI
    8. Current default keyboard option
    9. App installer (Migrate 3.0)
    10. Adb state (Migrate 3.0)
    11. Font scale (Migrate 3.0)

    ... everything in one app.

    migrate_3.png

    migrate_4.png


    Ok. So its just another of those backup and restore apps, right?

    Sort of... but with a twist.
    Restoring everything requires flashing the backup via TWRP.

    ALTERNATELY
    You can use the Migrate Flasher app (if your phone does not have a compatible recovery, or your recovery is broken).

    If you are flashing via TWRP, see the below screenshot.

    migrate_recovery.png


    All your apps and data are restored to a directory under /data/balti.migrate, /data/local/tmp/migrate_cache (from v2.0+) and a helper app is injected into the system.. Once you boot up your new ROM, the app will spring up, ask for root permission and do all the job while you can relax.

    migrate_helper.png


    IF YOU ARE USING MIGRATE FLASHER
    The process is similar. Only difference is Migrate Helper is not injected into system, but installed as a user app. There is no use of recovery environment.
    Once Migrate Helper is installed, just press RESTORE APPS AND DATA and proceed as usual.

    Please see the HOW TO FLASH section on Migrate Flasher for complete illustration.

    IF you have already flashed using TWRP (with no errors) then the above sections about Migrate Flasher is not applicable.

    Donations:
    Well, the app already has ads, so you need not explicitly donate anything :p
    But if you are feeling reaalllyyy grateful for the app, you can donate here: paypal.me/sayantanrc
    Thanks.

    Migrate Gemini (v4.0) release

    Hello everyone.
    Migrate 4.0 is now released.

    There are 2 editions of Migrate.

    Migrate NG: This is the Non-Google and Beta edition. It is not published on Google Play store. It will continue having in-app updates. Moreover new features will be tested on these builds first. People who did not like firebase analytics, this edition is for you as it does not have firebase SDK anymore. However, Google admob ads are still present.

    Download link (NG): https://www.androidfilehost.com/?fid=10763459528675575696

    Migrate GPE: This version is Google Play Edition. It is the stable version. It does not have in-app updates anymore as it violated Google Play policies.
    Download on Google Play: https://play.google.com/store/apps/details?id=balti.migrate

    Coming from alpha 3.1, if you update via the in-app updater, you will be switched to the Non-Google Edition.

    Thank you.

    Want to translate the app?
    The strings.xml file is available here: https://github.com/SayantanRC/translations

    Telegram group!
    We now have a Telegram group for migrate:
    http://t.me/migrateApp
    This is not to rival XDA but only another form of communication between users and me :)
    Thanks.
    146
    FAQ:

    Click here for Troubleshooting options


    If you are facing issues while flashing, refer to this video: https://youtu.be/0Cp6TNayXnM
    Full manual extract guide: https://xdaforums.com/showpost.php?p=82198185&postcount=1269

    How is this app different from Titanium backup?
    This app is a ROM migration tool. It not only backs up app and data, but also contacts, SMS, call logs etc. Moreover, the restoration process also involves flashing via TWRP. Titanium backup is an excellent app. Migrate just views the restoration process differently.

    In recent times however, Migrate Flasher app can be used to flash a zip even if you dont have a compatible TWRP recovery. Everything is almost the same other than just the flashing part, which is done from the Flasher app.

    Do I need to download the Migrate app to restore the apps?
    No. Only your ROM needs to be rooted. The restoration process is handled by a helper app, which deletes itself after restoring everything.

    Is it necessary to use Magisk?
    The app works best with Magisk. Other options like Lineage OS su addon causes weird errors like incomplete file copy, etc.

    All apps are restored from recovery?
    No. That is not possible. Because Recovery is a completely different environment from Android. Think of it as installing a Windows game from a Linux distro. Instead from recovery, an app is injected into system which takes up the restore process when android boots up. Most things are automated, so you need not worry much.

    A quick word on SD card support...
    SD card support in v2.0 requires a Magisk module "ExSDCard Access Enabler", made by developer Romain. According to him, this module requires FUSE support on ROMs. FUSE is supported on some ROMs as a legacy feature as Android has moved away from FUSE to SDCARDFS. As such, the current implementation may or may not work based on your device and ROM.
    If it does not work for you, please wait until either the module developer rewrites the module :p or I find a different implementation :) Thanks for your patience.


    From version 5.0+ (Not yet released, available on GitHub)
    Users will be able to natively store backups on SD Cards, especially on Android 11+.
    For lower Android versions, Storage Access Framework is used (along with root access) to store backups on SD Cards as well as on USB-OTG.
    57
    Just posting this for everyone to know.

    An alpha build of Migrate 3.0 should be ready by the end of this month. The version should support Android 10, and will come with new features.

    In the main app, backup and all new features are fully operational. But the helper app is not ready yet. I am working on it. That should be ready by this month.

    Please hold on a little. Thank you for your support. ?
    56
    Changelog

    Version 4.0 [1st November 2020]

    # Compatibility fixes for Android 11
    # Add Message center to display important announcements.
    # Migrate helper now allows to retry failed apk installs
    # Add a "Migrate flasher only" backup option
    # Save app icons as png files
    # Show number of apps selected for backup in Extra backups screen
    # Keep screen on when calculating backup size
    # Save last state of selection in extras
    # Don't remount /data partition in system test
    # Many under the hood changes

    Version 3.1.1 [25thApril 2020]

    # Fix issue in v3.1 where restore would not be possible for apps with unsupported characters in the name.

    Version 3.1 [24thApril 2020]

    # Probable fix to data restore problems
    # Probable fix to helper crashing during loading apps
    # Record raw list of files in a backup
    # Many other minor improvements

    Use v3.1.1 above

    Version 3.0.3 [18thApril 2020]

    #. Slightly tweaked logo.
    #. Probable fix to No metadata found after flashing.

    Version 3.0.1 [9thApril 2020]

    #. Bug fixes
    #. Added translations

    Version 3.0 [20th March 2020]

    #. The app is rewritten in Kotlin
    #. Many new preferences have been added.
    #. Helper app is now available on Google Play.
    #. Many fixes in recovery. Support for SAR and Android 10.
    #. Added options for backup of app installer, font scale, adb state
    #. Set max zip size in preferences.
    #. Verification of zips after backup is complete.
    #. Many more changes and improvements…

    Version 2.1 [15th Jan]

    # Minor errors are now suppressed, but recorded in log.
    # Fix icon retention form previous backup
    # Fix wrong backup duration display.
    # Fixed a bug in error correction mechanism.
    # Probable fix for some apps not being backed up.
    # Fix for error 6 in TWRP for some locales.
    # Fixed a long standing wrong /data space calculation in TWRP.
    # Migrate helper no longer uninstalls app before installing.
    # Added Telegram group link!

    Version 2.0.1 [30th Dec] (christmas_beta)

    # Fixed a probable crash in Migrate helper (needs a magisk module)

    Version 2.0 [26th Dec] (christmas_beta)

    # Experimental SD card support (needs a magisk module)
    (please read the previous post)
    # Support for split apks
    # "Abort" option in Migrate helper
    # "Temporary disable" in Migrate helper
    # Apks are now unpacked under /data/local/tmp/migrate_cache
    (probable fix to some devices unable to install apks)
    # Each backup zip now has a busybox file as fallback

    Version 1.2 [12th Nov] (beta)

    # Completely avoid "dumpsys" to backup app permissions
    # Now app data in tar.gz is unpacked in /data/data
    (probable fix to "Operation not permitted" issues in Google Pixel and some other devices)

    Version 1.1.1 [10th Nov] (beta)

    v1.1 (skipping upload)

    # [New] Backup verification before zipping
    # Major code refactoring and under-the-hood changes
    # CANCEL button now kills the backup process
    # Fix to "set_progress()" in recovery
    # Probable fix to "Failed to change locale for db"
    # Probable fix to missing updater-script
    # Probable fix to some "Operation not permitted" issues
    # Remove illegal characters from backup name
    # "Share scripts" option should now work properly.

    v1.1.1

    # Fix to a no-closing-quote issue in helper

    Version 1.0.5 [7th Nov] (beta)
    # Probable fix to "android.os.TransactionTooLargeException"
    # Better in app protection to prevent zips from not being made
    # App version is now written in logs
    # App now targets Oreo instead of Pie to potentially avoid some crashes
    # Fixed unexpected ( in restore process

    Version 1.0.4 [5th Nov] (beta)
    # Improvements to logging
    # Ignore tar socket warnings
    # New update-binary to prevent TWRP error 3
    # Should fix a null String issue while reading backup data

    Version 1.0.3 (beta)
    # Fixed log sharing
    # Share script file in log
    # Blacklist Android system from system app backup list (prevents tar socket error)

    Version 1.0 (beta)
    # Initial release
    32
    Some plaguing bugs on some devices :(


    "dumpsys not found"
    (Severity: mild. Only permissions are not backed up)
    Mainly in ASUS devices. Probable solution is to not use dumpsys at all.

    Probably fixed in v1.2

    "tar: can't execute 'gzip': Permission denied"
    (Severity: severe. Do not restore this backup)
    I still have no idea why this is happening. Probable solution is use something other than tar.gz


    Data files cannot be moved during restore. "Operation not permitted"
    (Severity: severe)
    No idea yet. Prominent in Google Pixel devices and some Xiaomi devices

    Probably partially or completely fixed in v1.2

    Error 7 (or 6) in TWRP
    (Severity: severe)

    This issue which was supposedly fixed in v1.1.1 seems to occur again on few devices
    Since I cannot reproduce the issue and the number of error reports are pretty small, please send the recovery.log from TWRP:
    - Goto Advanced -> Terminal in TWRP
    OR
    open an adb shell inside TWRP
    - Type the following and press enter: cp /tmp/recovery.log /sdcard/
    - Reboot and upload the recovery.log file (from internal sd card) here.



    Stuck at calculating size.
    (Severity: severe)

    This is being reported multiple times. I am still unable to reproduce the issue on my device.
    That being said, I will be adding new controls to skip size checking of its stuck on any device, in the next release. Also, I will add more fine logging to see what is actually happening behind the scenes, which cannot be detected by a logcat.

    Probably fixed in v2.0

    Pocophone TWRP recovery has issues
    (Severity: severe)

    I need someone to do the following and then upload the file:/sdcard/test_log.log
    - TWRP main menu -> Mount -> Data and System should be mounted
    - TWRP main menu -> Advanced -> Terminal
    OR
    open an adb shell inside TWRP
    - Type the following commands:

    getprop > /sdcard/test_log.log
    echo " " >> /sdcard/test_log.log
    df /data >> /sdcard/test_log.log
    echo " " >> /sdcard/test_log.log
    df -k /data >> /sdcard/test_log.log
    echo " " >> /sdcard/test_log.log
    df -h /data >> /sdcard/test_log.log
    echo " " >> /sdcard/test_log.log
    df /system >> /sdcard/test_log.log

    The following should work as an alternative:
    You have to manually do the following:
    1. Open the backup zip file
    2. Edit the "package-data.txt" file
    3. Put the "data_required_size" as "0"
    4. Save the file.
    5. Repack the zip file.

    "Segmentation fault"
    (Severity: unknown)
    Mainly in Xiaomi devices. I cannot reproduce it. Neither do I know how much this affects the backup. Please restore a backup and tell me if it works.