Welcome to XDA

Search to go directly to your device's forum

Register an account

Unlock full posting privileges

Ask a question

No registration required
Post Reply

[SCRIPT] Repack multiple APKs with proper compression settings (and other stuff!)

OP omniwolf

14th June 2011, 01:40 PM   |  #1  
omniwolf's Avatar
OP Senior Member
Flag sydney
Thanks Meter: 462
 
995 posts
Join Date:Joined: Jan 2005
More
Hi Guys,

I was speaking to baadnwz a couple of weeks ago and i asked him which script he used to properly pack all the APKs in his ROM. He told me he used no script, just repacked the APKs he thought were most relevant.
I figured it couldn't be too hard to write something that took an APK, extracted it somewhere, then repacked it with the proper compression, so i wrote one. baadnwz then challenged me to make it work for multiple APKs, so he can just run it against a folder full of them like a lazy asshole. i went one better, and made a script that you could run against a ROM folder structure, it will search out all APKs, repack them, and copy them back into place. HOT!

because this uses recursion and functions to make it easier to write (and because i'm generally comfortable in it), i have written it in VBScript. So this means it won't work on linux as far as i know, sorry. Feel free to convert it or whatevs

WHAT DOES IT DO??????//
1. finds an APK
2. unpacks it in a folder
3. optimises the PNGs (if you want it to)
4. repacks all files as deflate except a blacklist of filetypes*
5. repacks the blacklisted filetypes* as STORE
6. zipaligns the new APK

7. copies it back to the original location, overwriting the original APK
8. finds the next APK, and repeat from step 2

* the blacklisted file types are easily editable. just find the STOREFileNames variable. by default the filetypes are:

"*.arsc", "*.m10", "*.png", "*.wav", "*.ogg", "*.mp4", "*.jpl", "*.jpeg", "*.gif","*.mp2", "*.mp3", _
"*.aac","*.mpg", "*.mpeg", "*.mid", "*.midi", "*.smf", "*.jet","*.rtttl", "*.imy", "*.xmf", _
"*.m4a","*.m4v", "*.3gp", "*.3gpp", "*.3g2", "*.3gpp2", "*.amr", "*.awb", "*.wma", "*.wmv", _
"*.so", "*.dat", "*.bin"

WHY DO YOU WANT THIS??
If certain filetypes in the APK are compressed as STORE (which means they aren't compressed at all), then Android can use them directly. If they are stored as DEFLATE (like all other file types should be), then Android must first decompress the APK somewhere. This means slower apps, more cache, and it's just not how things are supposed to be done. This script makes sure that all files inside all APKs are stored as they should be, and it will optimise PNG files and zipalign the final APK as it goes.

This script is really meant for ROM chefs to run against their ROMs before releasing them, but if you (a mere user) have a ROM that your chef hasn't run this script on, then you can extract the ROM, run the script on it, then recompress the ROM and flash it.

HOW DOES IT DO IT?????///////
  • It uses the command line version of 7-Zip. Download it from here:
    http://7-zip.org/download.html
  • It uses zipalign.exe from the android SDK. if you don't know where this comes from.. maybe you shouldn't be using this script
  • It uses a cool OptiPNG utility to optimise the PNGs if you want it to. This slows the processing down a lot.. but it's up to you if you wanna do it. You'll need to download the utility from here:
    http://optipng.sourceforge.net/


WHAT DO I HAVE TO DO TO MAKE IT WORK//////?
  1. Grab the script, put it in the same folder as the root of your ROM (the script will repack all APKs in the subfolders from where it's sitting, including the APKs in the folder the script is in).
  2. open up the script, there are 5 variables you MUST verify, most likely you will have to change them.. unless you put things on a D and E drive and your name is Nick..

    tempAPKextract = "E:\Temp\tempAPKExtract"
    tempAPKBuild = "E:\Temp\tempAPKBuild"
    sevenZipLoc = """D:\nick\android\7za\7za.exe"""
    zipAlignLoc = """D:\nick\android\android-sdk-windows\tools\zipalign.exe"""
    optiPNGLoc = """D:\android\7-zipA\ROM\optipng.exe"""


    the first two temp ones can be anywhere.. just make sure at least the sub folders up to "tempAPK..." exist (eg in this case the E:\Temp folder must already be there)

    the next three are the locations of the tools required for the script to work, edit to please!
  3. Run the script. If you have set verboseOutput to true (it's false by default), then I would highly recommend you run the script from the commandline with the command:

    cscript fixAllAPKs.vbs

anyways, hope it's useful to you.

DOWNLOAD

CHANGELOG????/
v1.5.1 - cleaned up some code, added in verboseOutput so you can watch the progress as it's fixing (must run as cscript!)
v1.5 - added OptiPNG to the script, enabled by default. Quite slow. Added some more file types.
v1.4.3 - added more file types to STORE, thanks gtg465x
v1.4.1 - minor code clean up, fixed some comments.
v1.4 - fixed the problem where the script wouldn't fix any APKs in the root folder that the script was in
v1.3 - almost doubled the speed of the script. fixed some naughty bugs
Last edited by omniwolf; 24th January 2012 at 01:26 PM. Reason: v1.5.1
The Following 15 Users Say Thank You to omniwolf For This Useful Post: [ View ]
14th June 2011, 10:25 PM   |  #2  
Recognized Developer
Thanks Meter: 2,937
 
3,913 posts
Join Date:Joined: Jun 2008
More
edit: nevermind
Last edited by gtg465x; 21st June 2011 at 10:23 PM.
15th June 2011, 11:02 AM   |  #3  
omniwolf's Avatar
OP Senior Member
Flag sydney
Thanks Meter: 462
 
995 posts
Join Date:Joined: Jan 2005
More
not from what i've been told. if it's not compressed, the android system doesn't have to explicitly decompress it somewhere when it wants to use it, it can just pick it straight out of the APK.
15th June 2011, 01:08 PM   |  #4  
Recognized Developer
Thanks Meter: 2,937
 
3,913 posts
Join Date:Joined: Jun 2008
More
Quote:
Originally Posted by omniwolf

not from what i've been told. if it's not compressed, the android system doesn't have to explicitly decompress it somewhere when it wants to use it, it can just pick it straight out of the APK.

Good man. Here are all of the files blacklisted by the default Android Asset Packaging Tool (aapt) if you want to include them.

{
".jpg", ".jpeg", ".png", ".gif",
".wav", ".mp2", ".mp3", ".ogg", ".aac",
".mpg", ".mpeg", ".mid", ".midi", ".smf", ".jet",
".rtttl", ".imy", ".xmf", ".mp4", ".m4a",
".m4v", ".3gp", ".3gpp", ".3g2", ".3gpp2",
".amr", ".awb", ".wma", ".wmv"
};
15th June 2011, 04:20 PM   |  #5  
omniwolf's Avatar
OP Senior Member
Flag sydney
Thanks Meter: 462
 
995 posts
Join Date:Joined: Jan 2005
More
Thumbs up
cool, thanks for the info. i've added the missing ones in and uploaded a new version.
cheers!
17th June 2011, 02:43 PM   |  #6  
myn's Avatar
Senior Recognized Developer
Thanks Meter: 4,002
 
2,658 posts
Join Date:Joined: Nov 2007
Donate to Me
Nice work.

Suggestion or maybe a challenge for Ver 2:

Given a definition of what languages the user defines, can you repack including only those languages. This would include both drawables and layout language folders.

Removing languages not only speeds up the rom but reduces the apk size which for Sense 3.0 is becoming challenging for many older (> 1 year) devices.

Thanks for your contributions to XDA.
17th June 2011, 04:36 PM   |  #7  
Recognized Developer
Thanks Meter: 2,937
 
3,913 posts
Join Date:Joined: Jun 2008
More
Also need to add .so to the blacklist. Ran into a problem with that the other day.
17th June 2011, 04:38 PM   |  #8  
Here's mine.

It synchronizes with our SVN repository and builds the rom. Any commits which were made are automatically turned into a new ClockWorkMod flashable package.

Code:
#! /bin/sh  

#This is the repository Directory, change to match your setup 
Repo="~/TeamKomin/trunk" 
#This is the Final folder, this is where the file will end up
Output="~/Desktop"   

#synchronize SVN repository
svn update "$Repo" 
#remove temp folder if exist
test -e "$Output/tmp/" && rm -Rf "$Output/tmp/" 
#make new temp folder
mkdir "$Output/tmp/" 
#copy repository to temp folder
cp -rf "$Repo/data" "$Output/tmp/" 
cp -rf "$Repo/META-INF" "$Output/tmp/" 
cp -rf "$Repo/system" "$Output/tmp/" 
cp -rf "$Repo/updates" "$Output/tmp/" 

#clear SVN folders to reduce size
find "$Output/tmp/" -name ".svn" -type d -exec rm -rf {} \; 
cd "$Output/tmp" 
#remove old files if there
test -e "$Output/AndromedaRepo.zip" && rm -f "$Output/AndromedaRepo.zip" 

#zip up a new Andromeda3 version
zip -r "$Output"/AndromedaRepo.zip ./* 

#done
I've also got a version which uses the Expect command to log into our SFTP server and update the files.teamkomin.com website... I'm not posting that here though as it contains security sensitive information.

Using this method, all members work on the ROM as though it were on the phone... This script packages it. No need to package or unpackage the entire thing.
Last edited by AdamOutler; 21st June 2011 at 09:14 PM.
19th June 2011, 03:04 PM   |  #9  
omniwolf's Avatar
OP Senior Member
Flag sydney
Thanks Meter: 462
 
995 posts
Join Date:Joined: Jan 2005
More
Quote:
Originally Posted by myn

Nice work.

Suggestion or maybe a challenge for Ver 2:

Given a definition of what languages the user defines, can you repack including only those languages. This would include both drawables and layout language folders.

Removing languages not only speeds up the rom but reduces the apk size which for Sense 3.0 is becoming challenging for many older (> 1 year) devices.

ok, sounds like a good idea. i'm not actually a ROM developer, so i don't know the ins and outs of the various files. any hints on which files i exclude for various languages? just having a vague look around i see in framework-res.apk there's some raw-ar, raw-cs, raw-en-GB folders, are these the ones you're saying i can strip out?
this will obviously be a lot easier if i can do it programmatically, do you know the rules?

Quote:

Also need to add .so to the blacklist. Ran into a problem with that the other day.

ok cool, added it in and uploaded 1.4.3. thanks dude.

Quote:

Here's mine.

It synchronizes with our SVN repository and builds the rom. Any commits which were made are automatically turned into a new ClockWorkMod flashable package.

I've also got a version which uses the Expect command to log into our SFTP server and update the files.teamkomin.com website... I'm not posting that here though as it contains security sensitive information.

Using this method, all members work on the ROM as though it were on the phone... This script packages it. No need to package or unpackage the entire thing.

ok cool. does this actually repack and zip align the APKs though? this looks like it just takes care of the overall ROM zip structure, not the APKs within..? maybe your script could call mine.. if there was a vbs parser for linux
19th June 2011, 10:02 PM   |  #10  
Quote:
Originally Posted by omniwolf

ok, sounds like a good idea. i'm not actually a ROM developer, so i don't know the ins and outs of the various files. any hints on which files i exclude for various languages? just having a vague look around i see in framework-res.apk there's some raw-ar, raw-cs, raw-en-GB folders, are these the ones you're saying i can strip out?
this will obviously be a lot easier if i can do it programmatically, do you know the rules?


ok cool, added it in and uploaded 1.4.3. thanks dude.


ok cool. does this actually repack and zip align the APKs though? this looks like it just takes care of the overall ROM zip structure, not the APKs within..? maybe your script could call mine.. if there was a vbs parser for linux

We keep the APKs on the server for ease of use. It makes it easier to reference a single file for testing that way.

Post Reply Subscribe to Thread
Previous Thread Next Thread
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes