FORUMS
Remove All Ads from XDA

[TOOLS][ZIPS][SCRIPTS] osm0sis' Odds and Ends [Multiple Devices/Platforms]

13,611 posts
Thanks Meter: 29,957
 
By osm0sis, Recognized Developer / Recognized Contributor on 18th April 2013, 12:37 AM
Post Reply Email Thread
26th November 2015, 03:05 AM |#411  
skulldreamz's Avatar
Senior Member
Flag Baltimore, MD
Thanks Meter: 1,187
 
Donate to Me
More
Quote:
Originally Posted by osm0sis

Between this: http://blog.droidzone.in/2012/11/30/...d-arm-devices/ where @zeppelinrox reported success on Cygwin, and this: https://github.com/tias/android-busybox-ndk to weed out a few more erroring applets I think I'm getting somewhere.

If all goes well I'll have the new AIK-mobile out by next week some time.

Just curious what you think of this build and if there are too many applets for a "normal" busybox build

Sent from my Nexus 5X using Tapatalk
 
 
26th November 2015, 03:43 AM |#412  
osm0sis's Avatar
OP Recognized Developer / Recognized Contributor
Flag Halifax
Thanks Meter: 29,957
 
Donate to Me
More
Quote:
Originally Posted by skulldreamz

Just curious what you think of this build and if there are too many applets for a "normal" busybox build

Well so far a bunch of stuff that tias notes need to be disabled is still enabled in the blog post's .config, and judging by the errors it seems like tias is correct. I'm also seeing a few more I need to remove that have changed since then like I2C*, TELNET*, SYNC and MDEV. Once I chop out all the broken stuff I'll see what I'm left with and if I want more. Ideally, I'd like to have something on par with Stericson, and tias provides a number of patches I can apply which get us close to that.

@zeppelinrox I'm still having issues with linking and includes from the NDK standalone CCs, any pointers from when you got it working in Cygwin?

Edit: Nevermind, I think I figured it out. I was overcomplicating things with some ldflags pointing to sysroot that apparently aren't necessary/break things.
The Following User Says Thank You to osm0sis For This Useful Post: [ View ]
27th November 2015, 12:44 PM |#413  
Member
Thanks Meter: 9
 
More
hi and hank you a lot for all the scripts.

I'm having a problem with the gapps integrator on a AOSP ROM, 5.1.1:
init.d didn't work, so I put it in su.d. After that, on the first reboot messenger, maps and store were integrated.
I rebooted again and didn't get the "optimizing..." so I rebooted another time.
Now all my GApps besides the three mentioned above are just gone. Their folders in /data/apps are gone as well. In /system/apps there are folders with apks inside, but they are not installed (and can't be installed by opening them - parsing error).
The only outlier is Play Services which stayed in /data.
No log was created in /sdcard.

Any ideas?
27th November 2015, 04:37 PM |#414  
osm0sis's Avatar
OP Recognized Developer / Recognized Contributor
Flag Halifax
Thanks Meter: 29,957
 
Donate to Me
More
Quote:
Originally Posted by dasdreHmomenT

hi and hank you a lot for all the scripts.

I'm having a problem with the gapps integrator on a AOSP ROM, 5.1.1:
init.d didn't work, so I put it in su.d. After that, on the first reboot messenger, maps and store were integrated.
I rebooted again and didn't get the "optimizing..." so I rebooted another time.
Now all my GApps besides the three mentioned above are just gone. Their folders in /data/apps are gone as well. In /system/apps there are folders with apks inside, but they are not installed (and can't be installed by opening them - parsing error).
The only outlier is Play Services which stayed in /data.
No log was created in /sdcard.

Any ideas?

Hmm not sure why there was no log if it was clearly doing something.. Perhaps it was executing too early for /sdcard to be mounted.

Lollipop support is highly experimental at this point as the way the optimization and cleanup works, especially on AOSP (versus Nexus), is drastically different. CM seems to be a no-go, and as for AOSP, it works quite well on my trusty old Galaxy Nexus (minus a few stray /data/app directories that weren't being cleaned) and I've heard others on several devices have it working reasonably well, but I've also heard several reports like your own that it doesn't work on other devices.

There's no rhyme or reason to it I can see so far, honestly the script could use a fresh set of eyes if there's someone else interested in collaborating to bring it the rest of the way for Lollipop and Marshmallow.
27th November 2015, 05:34 PM |#415  
Member
Thanks Meter: 9
 
More
Quote:
Originally Posted by osm0sis

Hmm not sure why there was no log if it was clearly doing something.. Perhaps it was executing too early for /sdcard to be mounted.

Lollipop support is highly experimental at this point as the way the optimization and cleanup works, especially on AOSP (versus Nexus), is drastically different. CM seems to be a no-go, and as for AOSP, it works quite well on my trusty old Galaxy Nexus (minus a few stray /data/app directories that weren't being cleaned) and I've heard others on several devices have it working reasonably well, but I've also heard several reports like your own that doesn't work on other devices.

There's no rhyme or reason to it I can see so far, honestly the script could use a fresh set of eyes if there's someone else interested in collaborating to bring it the rest of the way for Lollipop and Marshmallow.

thx for the reply, will watch this thread as to any updates on the script because the idea is brilliant.
The Following User Says Thank You to dasdreHmomenT For This Useful Post: [ View ] Gift dasdreHmomenT Ad-Free
28th November 2015, 11:21 AM |#416  
smint86's Avatar
Senior Member
Flag Dortmund
Thanks Meter: 719
 
More
Quote:
Originally Posted by osm0sis

Hmm not sure why there was no log if it was clearly doing something.. Perhaps it was executing too early for /sdcard to be mounted.

Lollipop support is highly experimental at this point as the way the optimization and cleanup works, especially on AOSP (versus Nexus), is drastically different. CM seems to be a no-go, and as for AOSP, it works quite well on my trusty old Galaxy Nexus (minus a few stray /data/app directories that weren't being cleaned) and I've heard others on several devices have it working reasonably well, but I've also heard several reports like your own that it doesn't work on other devices.

There's no rhyme or reason to it I can see so far, honestly the script could use a fresh set of eyes if there's someone else interested in collaborating to bring it the rest of the way for Lollipop and Marshmallow.

I'm currently using it on LP with a cm caf ROM on OnePlus One, which works quite okay. Had to put it in su.d, but it's doing its job. Only apps which are not deleted from data/app are:

com.google.android.apps.docs.editors.sheets-2
com.google.android.apps.translate-1

They are mentioned in the log, though. I don't really know why they aren't moved, even after several reboots. Last log is from 2 days ago now, at least 5 reboots in between.

The rest (Maps, Play store, docs, street, YouTube) are moved correctly. I'm unfortunately not that good in scripting, but could ship in logs if you need.



Swyped via Tapatalk
The Following User Says Thank You to smint86 For This Useful Post: [ View ] Gift smint86 Ad-Free
28th November 2015, 10:06 PM |#417  
osm0sis's Avatar
OP Recognized Developer / Recognized Contributor
Flag Halifax
Thanks Meter: 29,957
 
Donate to Me
More
Quote:
Originally Posted by smint86

I'm currently using it on LP with a cm caf ROM on OnePlus One, which works quite okay. Had to put it in su.d, but it's doing its job. Only apps which are not deleted from data/app are:

com.google.android.apps.docs.editors.sheets-2
com.google.android.apps.translate-1

They are mentioned in the log, though. I don't really know why they aren't moved, even after several reboots. Last log is from 2 days ago now, at least 5 reboots in between.

The rest (Maps, Play store, docs, street, YouTube) are moved correctly. I'm unfortunately not that good in scripting, but could ship in logs if you need.

Oh cool! This is the first report I've heard that it works on a CM ROM. Thanks!

Check the versions in /data/app against the versions integrated into system. A good file explorer app like FX is good for that. If they match then just delete the /data/app version and you should be back to normal working order.

As for logs, nah, I don't think this is something I can figure out. I spent days making changes to /data/app, /system/*app and packages.xml and diffing the resulting changes to packages.xml from the optimization to figure out how to experimentally safely alter it how it might want it to be changed to avoid further breakage, and even now I guess it's not 100% on all ROMs. Like I said, I really need a fresh set of eyes to hopefully figure it out and take it the rest of the way.

Also, here's where I'm at with busybox if anyone's interested in following along:
https://github.com/osm0sis/android-busybox-ndk

So far I've fixed all the host signing warnings I got during build on Cygwin and it builds for arm. I'm still vetting a few more of the patches from the original tias repo I forked, and bringing some headers over from dorimanx's monster combined busybox source + patches + toolchain repo to see what else I can fix, then of course the other 5 target architectures, but it's coming along. I think it'll be good to have all the required Android patches split out from the main source again.
29th November 2015, 09:04 AM |#418  
Senior Member
Thanks Meter: 286
 
More
Quote:
Originally Posted by osm0sis

Oh cool! This is the first report I've heard that it works on a CM ROM. Thanks!

...

Just for the records. I've also got the 000gappsintegrator script working on CM12.1 (Nexus 4 with nightly from 26.11.).

- /system/etc/init.d did not work, /system/su.d did
- chmod 0770 for 000gappsintegrator

First reboot copied the apps and libs and created "integrated"-files in the corresponding folders. Apps an libs were not moved.

Second reboot removed the old apps and libs.

Got now log at all.
The Following User Says Thank You to rogers11 For This Useful Post: [ View ] Gift rogers11 Ad-Free
29th November 2015, 10:10 AM |#419  
osm0sis's Avatar
OP Recognized Developer / Recognized Contributor
Flag Halifax
Thanks Meter: 29,957
 
Donate to Me
More
Quote:
Originally Posted by rogers11

Just for the records. I've also got the 000gappsintegrator script working on CM12.1 (Nexus 4 with nightly from 26.11.).

- /system/etc/init.d did not work, /system/su.d did
- chmod 0770 for 000gappsintegrator

First reboot copied the apps and libs and created "integrated"-files in the corresponding folders. Apps an libs were not moved.

Second reboot removed the old apps and libs.

Got now log at all.

su.d must be extremely early in the boot. Thanks for the report! Maybe I can figure out a way to make it wait for sdcard, might solve some non-working devices, but could also break some working devices by having it happen too late into the boot... Hmm, or maybe it'd be better to save the log in a buffer until after boot completes then copy it to sdcard, to leave the process alone for those it works for...
The Following User Says Thank You to osm0sis For This Useful Post: [ View ]
9th December 2015, 04:30 PM |#420  
osm0sis's Avatar
OP Recognized Developer / Recognized Contributor
Flag Halifax
Thanks Meter: 29,957
 
Donate to Me
More
New addition to the OP.

Busybox Installer:
As a byproduct of building my own static busybox compiles in all supported Android architectures for my latest AIK-mobile v2.0 package I figured I might as well offer them up separately as well since there aren't any providers making Android x64 builds, at least not for all of them when I went hunting (to try and save myself some time). The installer detects what architecture (ARM/ARM64, x86/x86_64, MIPS/MIPS64) your device uses and installs the correct busybox binary to /system/xbin accordingly; it detects and supports "systemless" root installation as well, installing to /su/(x)bin instead. It then cleans up any symlinks from a possible previous installation at that location and cleverly generates new symlinks directly from the output of the installed binary, so they always match.

My busybox configs and patches are available here: https://github.com/osm0sis/android-busybox-ndk

I've got a patch submitted to the official busybox mailinglist to resolve some API 21 compatibility, so that will probably be added in a future version to save people time and make it easier for more providers. Anyway, hopefully I got all the applets most people want working.


Edit: Now the zip will also clean up an installation made to /su/bin and then install to /su/xbin.

Edit 2: New special zip features:
1) system may be added to the zip filename to force a system installation.
2) uninstall or Uninstall may be added to the zip filename to uninstall busybox from the zip's detected target location (/su/xbin > /magisk/busybox-ndk/system/xbin > /system/xbin, unless system forced per above).



For those curious, I automate bundling my busybox binaries directly with builds from my AIK Build Farm using the following script.

~/bbox-bundle.sh:
Code:
# also this

home=~;
magisk=$home/Magisk;

bold() { echo -e "\033[1m"[email protected]"\033[0m"; }

cd "$home";

case $1 in
     help) echo "busybox-bundle.sh [verbose] [nocopy]"; exit 1;;
  verbose) stdout=/dev/stdout; shift;;
        *) zipflag=-q; stdout=/dev/null;;
esac;

rm -f Busybox-*-ALL*.zip UPDATE-Busybox.Installer.*-ALL-signed.zip 2>/dev/null;

ver=$(ls | grep ^busybox- | cut -d- -f2);

if [ -f AIK-build/mobile/arm/busybox -a "$1" != "nocopy" ]; then
  bold "- Copying new Busybox $ver builds ...";
  cd AIK-build/mobile;
  for i in *; do
    cp -fp $i/busybox $magisk/UPDATE-Busybox.Installer.*-ALL/busybox-$i;
  done;
  modver=$(grep -o "^version=.*$" $magisk/UPDATE-Busybox.Installer.*-ALL/module.prop | cut -d= -f2);
  modcode=$(grep -o "^versionCode=.*$" $magisk/UPDATE-Busybox.Installer.*-ALL/module.prop | cut -d= -f2);
  if [ "$ver" == "$modver" ]; then
    newmodver="$modver";
    newmodcode="$((modcode + 1))";
    iteration=1;
  else
    newmodver="$ver";
    newmodcode="${ver//.}0";
  fi;
  if [ "$iteration" ]; then
    echo -ne "\033[1m\n*** Update Magisk module version code ($modcode -> $newmodcode) (y/n)? \033[0m";
    read n; echo;
    case $n in
      Y|y) ;;
      *) skipmodprop=1;;
     esac;
  fi;
  if [ ! "$skipmodprop" ]; then
    sed -i -e "s/version=.*/version=$newmodver/" -e "s/versionCode=.*/versionCode=$newmodcode/" $magisk/UPDATE-Busybox.Installer.*-ALL/module.prop;
  fi;
fi;

bold "- Packing \"Busybox-$ver-ALL.zip\" ...";
cd $magisk/UPDATE-Busybox.Installer.*-ALL;
zip -9 -x .git -r $zipflag "$home"/Busybox-$ver-ALL.zip *;
curdir=$(basename $PWD);
cd ..;
if [ "$curdir" != "UPDATE-Busybox.Installer.v$ver-ALL" ]; then
  mv UPDATE-Busybox.Installer.*-ALL UPDATE-Busybox.Installer.v$ver-ALL;
fi;

cd "$home";
bold "- Signing as \"UPDATE-Busybox.Installer.v$ver-ALL-signed.zip\" ...";
java -jar zipsigner-*.jar Busybox-$ver-ALL.zip Busybox-$ver-ALL-signed.zip;
mv -f Busybox-$ver-ALL-signed.zip UPDATE-Busybox.Installer.v$ver-ALL-signed.zip 2>/dev/null;
rm -f Busybox-*-ALL.zip /tmp/signAPK*.tmp 2>/dev/null;
The Following 92 Users Say Thank You to osm0sis For This Useful Post: [ View ]
10th December 2015, 04:20 AM |#421  
osm0sis's Avatar
OP Recognized Developer / Recognized Contributor
Flag Halifax
Thanks Meter: 29,957
 
Donate to Me
More
New version of the GApps Auto-Integration script up in the OPs!

/system/su.d/000gappsintegrator:
Code:
#!/system/bin/sh
# chmod -R 755 /system/etc/init.d /system/su.d
#
# ROM GApps Auto-Integration
# osm0sis @ xda-developers

logbuff() { logbuff+="$($*)"; }
writable() { touch $1/tmpfile 2>/dev/null; ret=$?; rm $1/tmpfile 2>/dev/null; echo $ret; }

# wait for /system to become remountable/writable
until [ "$(mount -o remount,rw /system >/dev/null; writable /system; mount -o remount,ro /system >/dev/null)" == 0 ]; do
  sleep 1;
done;

logbuff echo -n `date`;
gtmp=/data/local/tmp/gapp;

# get SDK version to perform different actions due to /data/app layout changes 
sdkver=`getprop ro.build.version.sdk`;

# find new unintegrated Google Apps APKs in /data
for i in $(ls /data/app/ | grep -E 'com.android|com.google.android'); do

  # find equivalent /system APK name and only process if it exists
  xml=/data/system/packages.xml;
  package=`echo $i | cut -d- -f1`;
  sysapk=`grep "updated-package name=\"$package" $xml | grep -o 'codePath=.*$' | cut -d\" -f2`;

  logbuff echo -ne "\n/data/app/$i $sysapk";

  if [ "$sysapk" ]; then

    # compare /data and /system APK versions and only integrate if /data is newer (necessary on Lollipop and above)
    datver=$(grep "codePath=\"/data/app/$i" $xml | grep -o 'version=.*$' | cut -d\" -f2);
    sysver=$(grep "codePath=\"$sysapk" $xml | grep -o 'version=.*$' | cut -d\" -f2);

    if [ "$datver" -gt "$sysver" ]; then

      logbuff echo -ne "\t ($datver > $sysver)";
      mkdir -p $gtmp;

      # KitKat (and below) support
      if [ "$sdkver" -le 20 ]; then

        # remove libraries from copied APK and zipalign if the binaries exist
        datapk=/data/app/$i;
        if [ "$(zip --help)" -a "$(zipalign --help 2>&1)" ]; then
          cp -fp $datapk $gtmp/preopt-$i;
          zip -d $gtmp/preopt-$i lib/*;
          zipalign -v 4 $gtmp/preopt-$i $gtmp/$i;
          datapk=$gtmp/$i;
        fi;

        # extract and force copy libraries to /system
        unzip /data/app/$i -d $gtmp;
        chmod 644 $gtmp/lib/arm*/*;
        mount -o remount,rw /system;
        cp -fp $gtmp/lib/arm*/* /system/lib/;

        # overwrite /system APK with new /data APK then fix permissions
        cp -f $datapk $sysapk;
        chown root.root $sysapk;
        chmod 644 $sysapk;

      # Lollipop support
      elif [ "$sdkver" -le 22 ]; then

        # save time on boots after APK work has been completed but apps are awaiting optimzation
        if [ ! -f /data/app/$i/integrated ]; then

          # remove libraries from copied APK and zipalign if the binaries exist
          datapk=/data/app/$i/base.apk;
          if [ "$(zip --help)" -a "$(zipalign --help 2>&1)" ]; then

            # workaround for Chrome not playing by the usual rules (per usual)
            if [ "$(basename $sysapk)" == "Chrome" ]; then
              mount -o remount,rw /system;
              rm -f $sysapk/lib/arm*/libchrome.so;
            else

              cp -fp $datapk $gtmp/preopt-$i.apk;
              zip -d $gtmp/preopt-$i.apk lib/*;
              zipalign -v 4 $gtmp/preopt-$i.apk $gtmp/$i.apk;
              datapk=$gtmp/$i.apk;

            fi;
          fi;

          # force copy libraries to /system respecting symlinks then clean up empty files
          mount -o remount,rw /system;
          cp -RLf /data/app/$i/lib $sysapk;
          for j in `ls $sysapk/lib/arm*/*`; do
            test ! -s $j && rm -f $j;
          done;

          # overwrite /system APK with new /data APK then fix permissions
          cp -fp $datapk $sysapk/`basename $sysapk`.apk;
          chown -R root.root $sysapk;
          chmod 644 $sysapk/*.apk $sysapk/lib/arm*/*;

          # flag for cleanup on reboot following optimization
          touch /data/app/$i/integrated;

        fi;
        # remove packages.xml entry for /data APK
        sed -i -e "/<updated-package name=\"${package}/,/<\/updated-package>/d" $xml;

      # Marshmallow (and above) support
      elif [ "$sdkver" -ge 23 ]; then

        # save time on boots after APK work has been completed but apps are awaiting optimzation
        if [ ! -f /data/app/$i/integrated ]; then

          # if necessary force copy libraries to /system respecting symlinks then clean up empty files
          mount -o remount,rw /system;
          if [ -d $sysapk/lib ]; then
            cp -RLf /data/app/$i/lib $sysapk;
            for j in `ls $sysapk/lib/arm*/*`; do
              test ! -s $j && rm -f $j;
            done;
          fi;

          # if necessary force copy APK odex file to /system
          if [ -d $sysapk/oat ]; then
            cp -fp /data/app/$i/oat/arm*/base.odex $sysapk/oat/arm*/`basename $sysapk`.odex;
          fi;

          # overwrite /system APK with new /data APK then fix permissions
          cp -fp /data/app/$i/base.apk $sysapk/`basename $sysapk`.apk;
          chown -R root.root $sysapk;
          chmod 644 $sysapk/lib/arm*/* $sysapk/oat/arm*/*;

          # flag for cleanup on reboot following optimization
          touch /data/app/$i/integrated;

        fi;
        # remove packages.xml entry for /data APK
        sed -i -e "/<updated-package name=\"${package}/,/<\/updated-package>/d" $xml;

      fi;
      mount -o remount,ro /system;
      rm -rf $gtmp;

    fi;
  elif [ -f /data/app/$i/integrated ]; then

    # clean up to mimic pre-Lollipop (AOSP) behavior
    rm -rf /data/app/$i;

  fi;
done;

# global cleanups required on Lollipop (and above)
if [ "$sdkver" -ge 21 ]; then

  # fix /system/lib permissions to ensure libs copied via symlink are correct
  mount -o remount,rw /system;
  chown root.root /system/lib/*.so;
  chmod 644 /system/lib/*.so;
  mount -o remount,ro /system;

fi;

logbuff echo -e "\n---";

# write buffered log once /sdcard is available
until [ "$(writable /sdcard)" == 0 ]; do
  sleep 1;
done;
log=/sdcard/gapps-integrator.log;
test ! -f $log && echo -e "## GApps Auto-Integration Script Log\n" > $log;
echo -e "$logbuff\n" >> $log;
Changelog:
- added a check-and-wait until /system is writable to hopefully resolve the issues from the script possibly running too early to make changes.
- buffered the script output to a variable and then output it to sdcard at the end after a check-and-wait until /sdcard is writable to hopefully resolve reported logless running.
- skip processing APKs once they've been integrated on Lollipop+, allowing significantly more (usually the rest of the) apps to be optimized on the reboot after integration; finally fewer reboots to fully integrate a long list of updated APKs!
- added lib removal and zipalignment for Lollipop and below if the zip and zipalign binaries are present on the system. Many custom ROMs have zipalign but not many will have zip, so I've done the Google-fu for any who would like to test it out and hunted down static builds of both and uploaded them to my BasketBuild so people may add whichever they need to /system/xbin. Remember to make them root and executable.
- added a lib removal workaround for Chrome on Lollipop, which apparently cannot function without its libraries being present inside the APK. I even attempted to extract crazy.libchrome.so (not present in /data/app) to replace libchrome.so but it doesn't like it, so now we just delete the external (/system/app) libchrome.so to save the space instead and that seems to work!

Working well on my Jelly Bean TV Box (though with lib removal and zipalignment the boot takes significantly longer), and also on my aging Galaxy Nexus running AOSP Lollipop. So far no issues that another reboot hasn't resolved. Enjoy!
The Following 7 Users Say Thank You to osm0sis For This Useful Post: [ View ]
Post Reply Subscribe to Thread

Tags
automation, batch, flashable zip, script, tool

Guest Quick Reply (no urls or BBcode)
Message:
Previous Thread Next Thread
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes