Making modifications & mergesmali

Renate · Jun 27, 2012

This is a new thread on the subject of modifying the system or
applications on the Nook. I'd like to see a separate thread on
modifying the kernel, but let's keep that off this.

The Basics

A bit of the system or an app is usually written in Java. Different
tools are used to compile and process it to the final product.

The Java Runtime Environment (JRE) is what allows you to run various tools and program on your host computer.
The Java Development Kit (JDK), version 6, update 33 contains the tools you need to work with Java
- javac - the Java compiler
- jarsigner - a tool for signing products
Android Software Development Kit (SDK), with downloads for Level 7 API.
- Android Asset Packaging Tool (AAPT) - processes resources like images and layouts
- Android Debug Bridge (ADB) - allows connection and debugging to your device

The final product is either an APK file or a JAR file, both of which
are basically ZIP files. Both of these contain a file called
classes.dex which contain the executable code. An APK file also
contains resources in a directory hierarchy. Some of these items,
like regular PNG graphics are exactly as you'd find them anywhere.
Other items, like "9 patch files" (9.png), are modified and others,
like XML files, are compressed. An APK file also contains a file
called AndroidManifest.xml that describes the product. Both APK
and JAR files can also contain signatures in the META-INF
directory.

Reverse Engineering

In the best of all worlds, you would have the original Java code that
the developers used to make the product. This is seldom available.
To work around this you need to backwards step the entire
process to get back to the original Java code. There is a problem,
the backwards process is not unambiguous. Yes, you can
backtrack to some Java code, that if compiled would work the
same as the original, but it would not look the same. Often the
intent of a piece of software is apparent from its layout. Also, you
will have none of the comments in the original code.

We can easily backtrack to an intermediate place between source
Java code and the final product. This is a place that does not really
exist in the original product generation process. We can take the
raw executable code out of a product and display it in a human
readable (and editable) form. "Smali" is the name of this
representation. It is analogous to assembly code. As stated, it
really does not exist as a language in the original compilation.

There is a software tool for taking apart a product and dissecting
it into the Smali code and the resources (if any). This tool also can
be used for compiling the Smali code back into the modified
product.

apktool, a tool for reverse engineering Android files

One of the main actions of the apktool is to take apart classes.dex
and generate a whole tree of files that end in the extension ".smali".
These files will have names like:
\NeatoApp\smali\com\bigcompany\neatoapp\MainView$23.smali

Modifying a Product

So, we can take a finished product, use apktool on it to take it apart
to pieces, modify some piece, then put it back together with
apktool. Modifying a resource like a graphic is easy, just modify
it (except 9.png, more later). Changing the wording of a popup
message is also easy. Changing the language of the interface
takes a bit more care to do it correctly. Modifying the Smali code
takes a bit of knowledge and done incorrectly can even brick your
device (repairable with a backup). If you have a chunk of Smali
code that someone modified for some reason, it's not too difficult
to open up your extracted Smali file, edit in the chunk, save it and
run apktool to put everything back together.

The Problem

We see this problem with kernels all the time, that some users want
a kernel with A, B, C and others want it with C, D, E. The number
of competing configuration gets out of hand. Moreover if you
come up with feature F, then you have to find a way to package
it up with A, B, C, for the first user and D, E for the second user.

A Solution

One possible solution is to let the user decide. You can distribute
options A-F independently and let the user install them themselves.
In principle, this means the user takes whatever version of
something they have on their device, apply a specific patch, then
reinstall it on their device. This would also open things up to
modifying different versions (for example Nook Touch vs. Nook
Glow) with the same mod.

mergesmali

mergesmali is a new tool for managing modifications to Smali
code. There is no particular magic behind it. It simply can replace
sections of Smali in a text file. It is agile enough to not rely on line
numbers or exact specifics of the Smali file. Here is a simple
example how you would use it to modify something:

Code:

adb pull /system/framework/android.policy.jar
apktool d android.policy.jar \Policy
mergesmali /v \Policy\smali\com\android\internal\policy\impl\LockScreen.smali landscapemod.smali
apktool b \Policy android.policy.jar
adb push android.policy.jar /system/framework

Mod Developers

We'll get to how to make the modification files for Smali soon...

ApokrifX · Jun 27, 2012

Renate,
Great post! :good:
Could you add few words on logcat/ddms to help with troubleshooting, if a mod doesn’t work as expected, please?

Renate · Jun 30, 2012

Yes, I'm still working on this.
I've been patching the stock reader for dictionary and other things.

I was doing the patches on the 1.1.2 Reader.apk and it worked fine.
I just switched over to the 1.1.5 Reader.apk and it patched it fine without any changes.

Renate · Jul 4, 2012

This post addresses the developer side of things.

It is possible to patch Smali by hand.
For quick and dirty, this is not a bad solution.
The problem is that one must be aware of how many registers are used and for what.
In the first instance, if I need another register for my mod, that can push register references on other instructions out of the 4 bit range.
Also, if the stock code is revised later that can also change register references.
Even the simple addition of a debug print statement (Log.d) can affect things.
The solution is to always replace entire methods.
That way you are not fighting with conflicting code.

The logical place to keep your patches is in Java.
Obviously you can't have a standalone method, there has to be a class.

For instance, the stock Reader.apk has a place where it validates EAN (ISBN).
It checks for all digits, which bombs on sequences with dashes.
We want to replace that, easy enough, but we have to wrap it in the class.

Code:

package com.bn.nook.model.product;

public class Products
{
   public static boolean isValidEan(String ean)
   {
      return(true);
   }
}

Obviously this does not fully implement the class, but it has enough for our little method.
We do a normal compile of this, then apktool d it to the smali.
mergesmali can then scan through this smali and edit the stock smali from the apktool d'ed stock app.
A final apktool b, jarsigning and zipaligning puts it all back together.

But what if the method that we want to replace called some other method?
We would have to put something in our class to prevent compilation errors and to ensure that the smali code for our method did the correct invocation.
We end up writing stubs for these methods that do nothing.
If it's a void method, the body of the method is simply {}, for boolean types {return(false);}, etc.
But how do we make sure that mergesmali does not substitute these stub methods for the good methods already in stock?
Annotations.

By putting Annotations before each method, we can tell mergesmali what to do.
These annotations are preserved in the decoded smali.
They are subsequently deleted when mergesmali merges the smali.
Expanding our original example (this is just an example, not realistic):

Code:

package com.bn.nook.model.product;

import com.whoever.MergeSmali.*;

public class Products
{
   @Ignore
   public Products() {}

   @Ignore
   public static double complicatedOtherMethod(String ean, String msg)
   { return(0.0); }

   @Replace
   public static boolean isValidEan(String ean)
   {
      if (complicatedOtherMethod(ean, "Hello")<2.3) return(false);
      return(true);
   }
}

The @Replace tells mergesmali to replace this method.
The @Ignore tells mergesmali not to replace this method.
We need to have the stub for complicatedOtherMethod in the source.
Well, why is the constructor in there?
The compiler will generate a default constructor if we don't write one and it would have no annotation.
mergesmali prints an error if any method does not have an annotation;
There is also @Append for new methods that are not in the stock.
There is also @Delete to delete methods (that either have a super or not being used at all with the rewritten methods).

What's in com.whoever.MergeSmali.* ?

Code:

package com.whoever;

public class MergeSmali
{
   public @interface Ignore {}
   public @interface Replace {}
   public @interface Append {}
   public @interface Delete {}
}

The package is not important as long as it's an Annotation and the class/subclasses are as indicated.

mrWax · Jul 9, 2012

collaboration?

Maybe we could start some sort of work distribution (and knowledge) towards fullfiiling some common goal for the NST.

Renate · Jul 9, 2012

Here's the Windows (command line) executable for mergesmali.
The basic usage is:

Code:

mergesmali /v [i]stock.smali[/i] [i]patch.smali[/i]

And here's the commands to integrate the new dictionary app to the stock Reader.

Code:

apktool.bat d Reader.apk \Reader
mergesmali /v \Reader\smali\com\bn\nook\reader\ui\ReaderMainView.smali ReaderMainView.smali
apktool.bat b \Reader NewReader.apk

It should generate the following message:

Code:

Replaced: public showLookupView()V

At this point you only have the question of how you are going to sign the modified Reader.
If you have resigned your whole system already, you can just resign with that and install.
You could also edit AndroidManifest.xml before the apktool b and delete the sharedid.
Then you could install it as a regular app after deleting the system app.

ApokrifX · Jul 9, 2012

Renate NST said:
At this point you only have the question of how you are going to sign the modified Reader.
If you have resigned your whole system already

I guess no one did it... yet.

Renate NST said:
You could also edit AndroidManifest.xml before the apktool b and delete the sharedid.
Then you could install it as a regular app after deleting the system app.

I don’t understand Android security well enough.
There a discrepancy between documentation and how it works or maybe I don’t understand it at all.
Anyway – if Reader.apk is resigned, it should run under different user, thus loose access to shared databases (internal, annotations, bookmarks, etc.)
Do we need to fix permission to make them RW for everyone?

Could you check (ps) what user(s) "home", "library" and "patched reader" use on you nook now, please?

Renate · Jul 9, 2012

Annotations, bookmarks, last reading point are all providers implemented and used in Reader.apk.
As far as I can tell, there is no particular reason why Reader.apk uses a sharedId.

I don't have stock Home or Library running on my Nook.
I forget where I have Reader.apk installed right now.
It's been in /system/app or /data/app and worked fine in either place.

Resigning does not change user.

ApokrifX · Jul 9, 2012

Renate NST said:
Annotations, bookmarks, last reading point are all providers implemented and used in Reader.apk.

I would expect the provider for Annotations, bookmarks is Reader.apk
They are not used anywhere else.
last reading point is used in Home.apk too.
Out of curiosity, where Annotations, bookmarks and last reading point providers are implemented?
In framework itself?

Renate NST said:
Resigning does not change user.

Ouch.
Stock reader runs as shared user.
If we resign it – it has to run as different one, according to security docs...
And it doesn’t matter if shared id is in manifest still or not.
I guess, I need to do some testing…

Renate · Jul 9, 2012

Ok, I meant resigning it with the system signature it can keep the same sharedId.
If you don't sign it with the same signature then you have to get rid of the sharedId.
It will then have a normal application user ID.
As far as I can tell, this should make no difference in anything.
But I have gutted most of the B&N stuff so I can't guarantee that in-store browsing or something else won't be affected.

ApokrifX · Jul 10, 2012

Renate,
Could you answer this one:

ApokrifX said:
Out of curiosity, where Annotations, bookmarks and last reading point providers are implemented?
In framework itself?

And one more:
Is it difficult to mod and stock side-by-side?
Change app name + package for every class?

Renate · Jul 10, 2012

In Reader.apk

com.bn.nook.reader.providers.AnnotationsProvider
com.bn.nook.reader.providers.BaseDictionaryProvider
com.bn.nook.reader.providers.BookmarksProvider
com.bn.nook.reader.providers.LastReadingPointProvider
com.bn.nook.reader.providers.ReaderLocalProvider

(I'd like to know some time why the forums breaks words up.)

On your second question, that's an interesting one.
You'd have to change the package name all over the place,
not just in the manifest, but every smali file and every function call.
But sure, it could be done.

ApokrifX · Jul 10, 2012

Renate NST said:
In Reader.apk

com.bn.nook.reader.providers.AnnotationsProvider

com.bn.nook.reader.providers.BaseDictionaryProvider

com.bn.nook.reader.providers.BookmarksProvider

com.bn.nook.reader.providers.LastReadingPointProvider

com.bn.nook.reader.providers.ReaderLocalProvider

Thank you!

Renate NST said:
(I'd like to know some time why the forums breaks words up.)

No idea…

Renate NST said:
Annotations, bookmarks, last reading point are all providers implemented and used in Reader.apk.
As far as I can tell, there is no particular reason why Reader.apk uses a sharedId.

Looking at B&N coding style, I won’t be surprised if Home.apk uses its own provider to read the "last reading point".
If this is the case sharedId in a must.

Renate NST said:
On your second question, that's an interesting one.
You'd have to change the package name all over the place,
not just in the manifest, but every smali file and every function call.
But sure, it could be done.

and every function call

But it can be automated easily, right?

Renate · Jul 10, 2012

ApokrifX said:
If this is the case sharedId in a must.

No.
The "Last read" icon in the upper left corner broadcasts com.bn.nook.launch.LAST_BOOK
This is usually received by Home.apk, which uses the LRP provider in Reader.apk to get the EAN of the last book.
Home.apk then sends the intent android.intent.action.VIEW to Reader.apk with the path of the book.
Reader.apk then uses it's own LRP provider to get the actual LRP in the book.

None of this requires sharedIDs.

#2 Sure, that could be automated.

ApokrifX · Jul 10, 2012

Renate NST said:
No.
The "Last read" icon in the upper left corner broadcasts com.bn.nook.launch.LAST_BOOK
This is usually received by Home.apk, which uses the LRP provider in Reader.apk to get the EAN of the last book.
Home.apk then sends the intent android.intent.action.VIEW to Reader.apk with the path of the book.
Reader.apk then uses it's own LRP provider to get the actual LRP in the book.

None of this requires sharedIDs.

Home shows "last book thumbnail" + on page #XYZ of #ABC also.
I recall you have deleted Home.apk.

Renate NST said:
#2 Sure, that could be automated.

Ok. I gotta take a look.Will try to do later today.

Renate · Jul 10, 2012

ApokrifX said:
Home shows "last book thumbnail" + on page #XYZ of #ABC also.
I recall you have deleted Home.apk.

Yes, my Library.apk replacement has a receiver for the com.bn.nook.launch.LAST_BOOK intent.

Even though Home.apk uses the Reader.apk to find LB & LRP for the preview,
it only uses LB to tell the Reader.apk what to open.
Reader.apk does its own homework checking LRP for the desired book.
This can easily be verified by opening a few books with a file manager.

ApokrifX · Jul 11, 2012

Renate NST said:
Yes, my Library.apk replacement has a receiver for the com.bn.nook.launch.LAST_BOOK intent.

Even though Home.apk uses the Reader.apk to find LB & LRP for the preview,
it only uses LB to tell the Reader.apk what to open.
Reader.apk does its own homework checking LRP for the desired book.
This can easily be verified by opening a few books with a file manager.

Ok. I.e. it shouldn't be a problem to run side-by-side readers, right?
They both modify LRP, and Home will be able to pull it via stock reader.
Same idiotic question: do we need to need to change any permissions to let both readers access media databases?

Renate · Jul 11, 2012

As it stands, if I understand what you want to do,
you'd have to disable the providers in one of the Readers.
You can't have two providers responding to the same intents.

Is there some overwhelming reason why you want to do all this?

ApokrifX · Jul 11, 2012

Renate NST said:
As it stands, if I understand what you want to do,
you'd have to disable the providers in one of the Readers.
You can't have two providers responding to the same intents.

I didn't realize until now, provider responds to URI, so better have one only to avoid problems...

Renate NST said:
Is there some overwhelming reason why you want to do all this?

To "not break existing functionality"

I see, it'll be too difficult to have both, too many changes are needed...

Renate · Jul 13, 2012

Just a bump, old posts don't have signatures.

mergesmali & other stuff can be downloaded from my signature blob.

Making modifications & mergesmali

Recognized Contributor / Inactive Recognized Dev

Senior Member

Recognized Contributor / Inactive Recognized Dev

Recognized Contributor / Inactive Recognized Dev

Senior Member

Recognized Contributor / Inactive Recognized Dev

Senior Member

Recognized Contributor / Inactive Recognized Dev

Senior Member

Recognized Contributor / Inactive Recognized Dev

Senior Member

Recognized Contributor / Inactive Recognized Dev

Senior Member

Recognized Contributor / Inactive Recognized Dev

Senior Member

Recognized Contributor / Inactive Recognized Dev

Senior Member

Recognized Contributor / Inactive Recognized Dev

Senior Member

Recognized Contributor / Inactive Recognized Dev

Similar threads

Top Liked Posts