Guide: Compile /system/bin binaries for your device from AOSP source code

[droidvoider]

Now tested up to downloading AOSP and make toolbox you should be all set
Please give thanks to this thread: https://forum.xda-developers.com/newreply.php?do=newreply&p=43622764
Warning: I hacked my way through this stuff a few weeks ago I am not an expert!

How to compile Android Open Source Code modules​

I don't compile C code on Windows machines I have no idea about that.

Notice
This guide is a quick and dirty how to make a module. It will not cover finalizing setting up the source codes for your device. It is only my goal to enable you to compile binaries such as grep, toolbox, dumpstate, dalvikvm, jack and etc.

===>] Setup Ubuntu 64bit [<===​

Unplug that Windows drive, plug in a fresh hard drive and install Ubuntu latest/greatest. Ignore the recommendation to downgrade gnu make!, for now.

Open a terminal and issue these commands (warning ppa repository for OpenJDK 7 is said to have a security issue?, isn't being updated?.. whatevs it works)

sudo apt-get update
sudo apt-get upgrade
sudo add-apt-repository ppa:openjdk-r/ppa
sudo apt-get install openjdk-7-jdk
sudo apt-get install openjdk-8-jdk
sudo apt-get install libc6:i386 libncurses5:i386 libstdc++6:i386 lib32z1 libbz2-1.0:i386
sudo apt-get install git ccache automake lzop bison gperf build-essential zip curl zlib1g-dev zlib1g-dev:i386 g++-multilib python-networkx libxml2-utils bzip2 libbz2-dev libbz2-1.0 libghc-bzlib-dev squashfs-tools pngcrush schedtool dpkg-dev liblz4-tool make optipng

(choose Java 1.7 in the following way)

sudo update-alternatives --config java

(let me know if I missed anything please)

"Tried the Android SDK only it is missing too many things we need as a developer"

===>] Setup Android Studio SDK & NDK [<===​

Installation Paths:
*** I install to /home/username/Android and /home/username/Android/Sdk and /home/username/Android/Sdk/ndk-bundle ***
NOTE: from here forward username will == droidvoider

Note: Android Studio IDE isn't necessary only the SDK & NDK are needed to compile AOSP.

Install Android Studio Proper: (don't worry about setting up paths we will cover that, just install it)
https://developer.android.com/studio/install.html

or
SDK Only:
Typically we install these things manually by creating the directory then just unzipping the files there.
https://developer.android.com/studio/index.html#linux-bundle (scroll down for sdk only)

mkdir /home/droidvoider/Android
mkdir /home/droidvoider/Android/Sdk

(then unzip the sdk zip to that directory. I recommend the file explorer copy/paste right click uncompress and done.)
https://dl.google.com/android/repository/tools_r25.2.3-linux.zip

Install NDK through the SDK Manger:
(from terminal 'studio.sh' and then configure, and then sdk manger --- if this is hard to figure out tell me I will elaborate)

or
Manually Install Native Development Kit -- 'c programming support'
Download the Native Development Kit from Google: https://developer.android.com/ndk/downloads/index.html

mkdir /home/droidvoider/Android/Sdk/ndk-bundle

Then just unzip the ndk files into the directory we created above.

===>] Setup your toolchain [<===​

** This example is arm64-v8a aarch64 **
1. Navigate to /home/droidvoider/Android/Sdk/ndk-bundle/build/tools and then open a terminal "right click open area"
2. mkdir /home/droidvoider/toolchains
3. ./make_standalone_toolchain.py --arch arm64 --api 23 --stl=libc++ --install-dir /home/mm/toolchains/aarch64-linux-android-4.9
4. cd /home/droidvoider
5. gedit .bashrc and morph this in at the bottom.. (AND edit or replace the existing PATH variable)
DON'T just PASTE IN *my* $PATH export!! I included my entire path statements to show you.

export PATH=$PATH:/usr/local/android-studio/bin:/home/droidvoider/Android/Sdk/platform-tools:/home/droidvoider/Android/Sdk/ndk-bundle:/home/droidvoider/Android/Sdk/tools

I feel this is human readable, for example change Android_Build_Out to be on your desktop instead if you want.

export PATH=$PATH:/home/droidvoider/toolchains/aarch64-linux-android-4.9
export NDK=/home/droidvoider/Android/Sdk/ndk-bundle
export SYSROOT=$NDK/platforms/android-23/arch-arm64
export TARGET=aarch64-linux-android
export HOST=$TARGET
export BUILD=x86_64-linux
export ANDROID_NDK_BIN=/home/droidvoider/toolchains/aarch64-linux-android-4.9/bin
export CC=$ANDROID_NDK_BIN/aarch64-linux-android-gcc-4.9
export CPP=$ANDROID_NDK_BIN/aarch64-linux-android-g++
export AR=$ANDROID_NDK_BIN/aarch64-linux-android-ar
export OUT_DIR_COMMON_BASE=/home/droidvoider/Android_Build_Out

Note: You might want to setup an alternate toolchain also but this is all of the puzzle pieces.

===>] Google's version of this How To -- Just for reference [<===​

https://source.android.com/source/requirements.html
https://source.android.com/source/initializing.html

===>] Install the repo tool [<===​

https://source.android.com/source/downloading.html
(don't type repo init or repo sync --- I will be taking back over from there on the next page)

Added Repair Notes -- Not part of the install!
Have you accidentally installed or removed something you shouldn't have? (welcome to development, here try this before reinstall)

sudo apt-get clean
sudo apt-get update
sudo apt-get install -f
sudo dpkg -a --configure
sudo apt-get dist-upgrade
sudo apt-get install -f
sudo dpkg -a --configure

 

[droidvoider]

Selecting the correct AOSP branch and downloading it.

Tested up to downloading AOSP and make toolbox -- you should be all set

===>] Match your build number to it's AOSP sources [<===​

preface: You can get this from your device if you're on the same build id as your the available source code from your vendor for your device. Otherwise you need to open the AP file from the firmware that matches those available sources to extract the system.img, to extract build.prop. I explain how to open a system.img file below under retrieving your hardware drivers. build.prop is in the main directory of system.img
(Many times the build number is the same. For me I believe all of MM builds are using this number.)

Assumes sources match current device, worked out true in my case
1. Plug in your device and get it connected. (DEVELOPER OPTIONS|USB DEBUGGING) and select allow on device

2. Retrieve the build number that matches the available sources for your device.
From your ubuntu terminal retrieve the build id using this command:

adb shell getprop | grep 'ro.build.id'

Yields something similar to this: [ro.build.id]: [MMB29K]

3. Match it up to the Nexus build numbers (This info is for AT&T Note 5 Marshmallow MMB29K, get your specific build number!)
https://source.android.com/source/build-numbers.html#source-code-tags-and-builds
MMB29K android-6.0.1_r1 Marshmallow Nexus 5, Nexus 5X, Nexus 6, Nexus 7 (flo/deb), Nexus 9 (volantis/volantisg)

===>] Bring down a specific AOSP source branch [<===​

4. Make a directory for the source code.

mkdir /home/droidvoider/Desktop/AOSP_Android_6.01_r1

5.

cd /home/droidvoider/Desktop/AOSP_Android_6.01_r1

6. Bring down the sources, this one is approximately 13 gigabytes

repo init --depth=1 -u https://android.googlesource.com/platform/manifest -b android-6.0.1_r1
repo sync

===>] I'm not sure the rest of this is needed [<===​

For compiling toolbox the remainder wasn't needed.. But I have a large list of things to do so I can't test each item. If you can't compile a specific module continue reading.

===>] Merge Vendor sources & AOSP sources [<===​

7. Download the available sources for your device. In this example I downloaded PE6 Marshmallow sources for AT&T Note 5:
http://opensource.samsung.com/reception/receptionSub.do?method=sub&sub=F&searchValue=SM-N920A

8. Read the readme file from the sources platform zip to understand how to merge them into the AOSP sources. For the 2 Samsungs I've worked with the idea is to replace any directory that already exists. But if there is just one file such as core.mk only replace the one file. Then edit the .mk files as described in your readme. (and/or take info from cyanogen/lineagos) -- <I can help more, ask>
note: you probably don't need to take the configs from LineageOS and put them into your .mk files. However, if you do need to get more configs then you should get a big fat message when you type make 'modulename'. At first only edit .mk files as described by vendor device source readme file.

===>] Merge in Hardware drivers and etc [<===​

possibly unnecessary depends what you're doing
9. Obtain a copy of the firmware for your device that matches the version of the source code you are able to download from your vendor.
for me that was Build Number: MMB29K.N920AUCU2BPE6 but your mileage will almost certainly vary!

10. Download https://github.com/anestisb/android-simg2img

11. Unzip it right in your download folder, open the folder and then 'open in terminal'

12. Make it and then move it a directory in your path. Warning: My command puts in in the Ubuntu default /bin folder.

make
sudo mv append2simg img2simg simg2img simg2simg simg_dump.py /bin

13. Uncompress the AP file from the matching firmware and extract the system.img into it's own directory
then select that folder, right click, open in terminal

simg2img system.img sys.raw
mkdir sys
sudo mount -t ext4 -o loop sys.raw sys/

14. A drive mounted, look on your task bar it should've wiggled too. Copy the etc and vendor folders into the main folder of the sources we are merging

 

[droidvoider]

===>] Listing and building modules [<===​

Navigate to the folder where you download the sources "/home/droidvoider/Desktop/AOSP_Android_6.01_r1" and open in terminal.

make modules -- list the available modules
make <module name> -- builds a specific module
example: make dumpstate
description: Will build everything needed for dumpstate and place it in the folder we specified in our export (above step). The final build line will read install and detail the final output folder
Example successful output:
[CODE]
Install: /home/droidvoider/Android_Build_Out/Android_6.01_r1/target/product/generic/system/bin/dumpstate

===>] Android Build System, basic intro [<===​

Notice: I built this how to to answer the same question from 3 people regarding working with toolbox and the dirtycow exploit. So I decided to give a direct example of using toolbox.c from farm-root

#ifdef
Our makefile is Android.mk and that's where we link things together. If you look at the Android.mk file for farm-root you will notice bridge.c is used 3 different times called different 'module' names. bridge_pull, bridge_push, bridge_pull_boot. Each of these will be binaries of those names.
Inside bridge.c you will see #ifdef FARM_PULL and then you will see #else and further you will see #endif which you may have noticed matches inside the Android.mk file for the bridge modules -DFARM_PULL -DFARM_BOOT <== Notice the double define on bridge_pull_boot

toolbox.c
toolbox.c is going to be the same way. You will need to copy shared.h and shared.c into the directory where toolbox.c resides. Then edit the Android.mk, in our example:
1. Navigate to this directory and open: system/core/toolbox/Android.mk
2. CTRL + F and search for "LOCAL_MODULE := toolbox"
3. Add: LOCAL_CFLAGS += -DFARM_PULL -DFARM_BOOT (in this example add one, both or even new ones you created)
4. Navigate to the main directory of the sources, you should see a Makefile and a build_64bit.sh
5. from terminal: make toolbox

Note: I think from here you can Google it out in a few minutes if that is not the case please let me know.

 

[droidvoider]

Working with C cross platform​

Ubuntu is Linux based just like Android and this makes testing blocks of code extremely easy. You of course can't use Android headers and in some rare cases you can't test the code on Ubuntu at all but in most cases you can. When I want to design something for Android I open gedit and save it as a .c file. Then I compile it using gcc -o mycode mycode.c There's plenty of examples on using gcc with linux but just understand you can do it all. Then before too much work test it on Android. (helpful commands at end of post)

My advice really is to build out your small blocks of code on your linux box but then paste them into your Android program folder, edit your Android.mk, add it to your Makefile including your 'push' section so that you can simply type make push to test it.
I am in fact trying to encourage you to learn C and not so much trying to encourage you to hack things. But I know that interest/passion is what teaches, not my words and not someone else's curriculum. So in that spirit I will do my best to give examples to help you with 'whatever' it is you are passionate about. Let me know what's missing.

Don't forget to compile for Android first
Before you can test your code you will have compiled it using the cross compiler for Android. ndk-build, or the correct gcc cross compiler. (Personally I put the .c file into a directory with Android.mk and a Makefile then just type make to build it to Android)
see examples section I will add a couple examples.

Android Developer Bridge -- a developers tool
adb is included with the Android SDK along with some other tools. Some of those tools are fastboot for unlocking bootloaders and another way of flashing. There is monitor which is a cool tool for remotely viewing processes, logcat, memory dumps and etc.
But pointedly what we will use the most is simply adb.

Using adb to test your code on locked down Android systems
Shell has fairly high privileges, you may not be aware but you can execute binaries and bash scripts. We use /data/local/tmp/ for these things. You can create a directory, add or remove files, execute your binaries and even execute shell scripts using sh script.sh
ndk-build places the binary in libs/(arch type) .. For a quick test you can just open a terminal in that directory then:

adb push mybinary /data/local/tmp/
adb shell
cd data/local/tmp
chmod 777 mybinary
./mybinary

 

[droidvoider]

Added:
Examples of basic make files for Android.

happy coding

If you get an error​

Please reissue the command but pipe the output to a file.
make toolbox > /home/droidvoider/Desktop/build_toolbox-output.txt
zip that up with your source code, including your customized header files and attach it to this thread.
puzzles are fun but I like all the pieces

 

[DigitalDoraemon]

Tested up to downloading AOSP and make toolbox -- you should be all set

===>] Match your build number to it's AOSP sources [<===​

preface: You can get this from your device if you're on the same build id as your the available source code from your vendor for your device. Otherwise you need to open the AP file from the firmware that matches those available sources to extract the system.img, to extract build.prop. I explain how to open a system.img file below under retrieving your hardware drivers. build.prop is in the main directory of system.img
(Many times the build number is the same. For me I believe all of MM builds are using this number.)

Assumes sources match current device, worked out true in my case
1. Plug in your device and get it connected. (DEVELOPER OPTIONS|USB DEBUGGING) and select allow on device

2. Retrieve the build number that matches the available sources for your device.
From your ubuntu terminal retrieve the build id using this command:

adb shell getprop | grep 'ro.build.id'

Yields something similar to this: [ro.build.id]: [MMB29K]

3. Match it up to the Nexus build numbers (This info is for AT&T Note 5 Marshmallow MMB29K, get your specific build number!)
https://source.android.com/source/build-numbers.html#source-code-tags-and-builds
MMB29K android-6.0.1_r1 Marshmallow Nexus 5, Nexus 5X, Nexus 6, Nexus 7 (flo/deb), Nexus 9 (volantis/volantisg)

===>] Bring down a specific AOSP source branch [<===​

4. Make a directory for the source code.

mkdir /home/droidvoider/Desktop/AOSP_Android_6.01_r1

5.

cd /home/droidvoider/Desktop/AOSP_Android_6.01_r1

6. Bring down the sources, this one is approximately 13 gigabytes

repo init --depth=1 -u https://android.googlesource.com/platform/manifest -b android-6.0.1_r1
repo sync

===>] I'm not sure the rest of this is needed [<===​

For compiling toolbox the remainder wasn't needed.. But I have a large list of things to do so I can't test each item. If you can't compile a specific module continue reading.

===>] Merge Vendor sources & AOSP sources [<===​

7. Download the available sources for your device. In this example I downloaded PE6 Marshmallow sources for AT&T Note 5:
http://opensource.samsung.com/reception/receptionSub.do?method=sub&sub=F&searchValue=SM-N920A

8. Read the readme file from the sources platform zip to understand how to merge them into the AOSP sources. For the 2 Samsungs I've worked with the idea is to replace any directory that already exists. But if there is just one file such as core.mk only replace the one file. Then edit the .mk files as described in your readme. (and/or take info from cyanogen/lineagos) -- <I can help more, ask>
note: you probably don't need to take the configs from LineageOS and put them into your .mk files. However, if you do need to get more configs then you should get a big fat message when you type make 'modulename'. At first only edit .mk files as described by vendor device source readme file.

===>] Merge in Hardware drivers and etc [<===​

possibly unnecessary depends what you're doing
9. Obtain a copy of the firmware for your device that matches the version of the source code you are able to download from your vendor.
for me that was Build Number: MMB29K.N920AUCU2BPE6 but your mileage will almost certainly vary!

10. Download https://github.com/anestisb/android-simg2img

11. Unzip it right in your download folder, open the folder and then 'open in terminal'

12. Make it and then move it a directory in your path. Warning: My command puts in in the Ubuntu default /bin folder.

make
sudo mv append2simg img2simg simg2img simg2simg simg_dump.py /bin

13. Uncompress the AP file from the matching firmware and extract the system.img into it's own directory
then select that folder, right click, open in terminal

simg2img system.img sys.raw
mkdir sys
sudo mount -t ext4 -o loop sys.raw sys/

14. A drive mounted, look on your task bar it should've wiggled too. Copy the etc and vendor folders into the main folder of the sources we are merging

And where is exactly the main folder? Sorry, Im just confused

 

[droidvoider]

And where is exactly the main folder? Sorry, Im just confused

it's no problem this stuff isn't easy to just figure out on your own. remember to substitute droidvoider for your ubuntu user name

In this example my sources are on my desktop in a folder named Android_6.01_r1
Sources for toolbox for example:
/home/droidvoider/Desktop/Android_6.01_r1/system/core/toolbox/<sources will be here including Android.mk>
Script for modules, including toolbox
/home/droidvoider/Desktop/Android_6.01_r1/Makefile <--- this is our modules script, if you will
<open a terminal in the above folder then use that Makefile like so>
make toolbox <---- this will compile only what is needed to compile the module 'toolbox' (this takes a minute)
Out export folder we decided in ./home/droidvoider/bashrc
/home/droidvoider/Android_Build_Out/Android_6.01_r1/target/product/generic/system/bin

 

[FreakMurderer]

Anybody, please compile grep utility for arm and x86... Minimum Platform Version Android 4.0.3, API Level - 15
Thanks

 

[[email protected]]

Great & useful .