Welcome to XDA

Search to go directly to your device's forum

Register an account

Unlock full posting privileges

Ask a question

No registration required
Post Reply

[GUIDE] Learning about the Android Build Process

OP v_superuser

14th May 2014, 06:22 PM   |  #1  
v_superuser's Avatar
OP Senior Member
Flag Jammu
Thanks Meter: 1,346
 
484 posts
Join Date:Joined: Jan 2013
Donate to Me
More
Introduction

The Android open-source project (AOSP) is quite complex and it can be hard to find a good way to get more familiar with it. Im going to try a practical approach, by describing the relevant parts of the build process.

Note - I recommend reading my this guide before you through this one just in case you are actually gonna read and learn.

This guide explains the whole android build process right from envsetup.sh to the working and understanding of makefiles to Package complete!

The Beginning(envsetup.sh)

It all gets started with the
Code:
. build/envsetup.sh
command. What it basically does is that it adds a number of commands to your environment which you later use for building Android for your device.

The commands are listed below -
Code:
- lunch:   lunch <product_name>-<build_variant>
- tapas:   tapas [<App1> <App2> ...] [arm|x86|mips|armv5] [eng|userdebug|user]
- croot:   Changes directory to the top of the tree.
- cout:    Changes directory to out.
- m:       Makes from the top of the tree.
- mm:      Builds all of the modules in the current directory.
- mmp:     Builds all of the modules in the current directory and pushes them to the device.
- mmm:     Builds all of the modules in the supplied directories.
- mmmp:    Builds all of the modules in the supplied directories and pushes them to the device.
- mma:     Builds all of the modules in the current directory, and their dependencies.
- mmma:    Builds all of the modules in the supplied directories, and their dependencies.
- cgrep:   Greps on all local C/C++ files.
- jgrep:   Greps on all local Java files.
- resgrep: Greps on all local res/*.xml files.
- godir:   Go to the directory containing a file.
- cmremote: Add git remote for CM Gerrit Review.
- cmgerrit: A Git wrapper that fetches/pushes patch from/to CM Gerrit Review.
- cmrebase: Rebase a Gerrit change and push it again.
- aospremote: Add git remote for matching AOSP repository.
- cafremote: Add git remote for matching CodeAurora repository.
- mka:      Builds using SCHED_BATCH on all processors.
- mkap:     Builds the module(s) using mka and pushes them to the device.
- cmka:     Cleans and builds using mka.
- repolastsync: Prints date and time of last repo sync.
- reposync: Parallel repo sync using ionice and SCHED_BATCH.
- repopick: Utility to fetch changes from Gerrit.
- installboot: Installs a boot.img to the connected device.
- installrecovery: Installs a recovery.img to the connected device.
It also scans for the 'vendorsetup.sh' files in your source and display them in this form -
Code:
including device/generic/armv7-a-neon/vendorsetup.sh
including device/generic/goldfish/vendorsetup.sh
including device/generic/mips/vendorsetup.sh
including device/generic/x86/vendorsetup.sh
including vendor/cm/vendorsetup.sh
including sdk/bash_completion/adb.bash
including vendor/cm/bash_completion/git.bash
including vendor/cm/bash_completion/repo.bash
The file is responsible for -
  1. All the make device calling, make etc. commands are possible only after you execute the envsetup.sh file.
  2. It also scans and does some magic stuff with the build/core directory.
  3. It also displays choices of variant for the selected device, like 'userdebug', 'user', 'eng'.
  4. It is also responsible for performing the most important step, it sets up the android build paths and and sets up the toolchains for the build process.
  5. It also calls java and sets it parameters for the compilation.
  6. It displays the default device configurations that are -
    Code:
    add_lunch_combo aosp_arm-eng
    add_lunch_combo aosp_x86-eng
    add_lunch_combo aosp_mips-eng
    add_lunch_combo vbox_x86-eng
  7. It calls the 'roomservice.py' file in build/tools folder which is responsible for finding and downloading the device sources from CyanogenMod's github if they don't exist.
  8. The file also contains the CWM and TWRP commands to sideload the product files into the device.
  9. It also gets bug reports from the device to the machine for the developers to work upon.

The lunch/brunch Command

Next thing is to select target using the lunch menu, which was added to your bash shell environment after sourcing envsetup.sh. After making your selection, the chosen product and variant is verified and environment variables are set, including:
  • export TARGET_PRODUCT=$product The chosen product
  • export TARGET_BUILD_VARIANT=$variant The chosen variant
  • export TARGET_BUILD_TYPE=release Only release type is available. Use choosecombo if you want to select type.
  • export ANDROID_BUILD_TOP=$(gettop) The build root directory.
  • export ANDROID_TOOLCHAIN=... The toolchain directory for the prebuilt cross-compiler matching the target architecture
  • export PATH=... Among other stuff, the prebuilt toolchain is added to PATH.
  • export ANDROID_PRODUCT_OUT=... Absolute path to the target product out directory
  • export ANDROID_HOST_OUT=... Absolute path to the host out directory

The usage is pretty simply, you can just type 'lunch' and you'll be prompted with the list of devices that you can build for with the sources present or you can just type 'lunch cm_<device_name>-<variant_type>'.
The lunch command actually fixes the target for the build process and is responsible for the paths getting locked and the device configurationto be used and applied to the source.
It actually gets to use only after there is a vendorsetup.sh file in your device tree which is responsible for adding the lunch combo(Which shows up in the list of devices when you simply run lunch) of your device.

The MAKE Command

The purpose of the make utility is to determine automatically which pieces of a large program need to be recompiled, and issue the commands to recompile them. The manual describes the GNU implementation of make, which was written by Richard Stallman and Roland McGrath. The following examples show C programs, since they are most common, but you can use make with any programming language whose compiler can be run with a shell command. In fact, make is not limited to programs. You can use it to describe any task where some files must be updated automatically from others whenever the others change.

To prepare to use make, you must write a file called the makefile that describes the relationships among files in your program, and the states the commands for updating each file. In a program, typically the executable file is updated from object files, which are in turn made by compiling source files.

Once a suitable makefile exists, each time you change some source files, this simple shell command:

Code:
make
suffices to perform all necessary recompilations. The make program uses the makefile data base and the last-modification times of the files to decide which of the files need to be updated. For each of those files, it issues the commands recorded in the data base.

make executes commands in the makefile to update one or more target names, where name is typically a program. If no -f option is present, make will look for the makefiles GNUmakefile, makefile, and Makefile, in that order.

Normally you should call your makefile either makefile or Makefile. (I recommend Makefile because it appears prominently near the beginning of a directory listing, right near other important files such as README.) The first name checked, GNUmakefile, is not recommended for most makefiles. You should use this name if you have a makefile that is specific to GNU make, and will not be understood by other versions of make. If makefile is `-', the standard input is read.

make updates a target if it depends on prerequisite files that have been modified since the target was last modified, or if the target does not exist.

Makefile Execution




The make command executes the commands in the makefile line by line. As make executes each command, it writes the command to standard output (unless otherwise directed, for example, using the -s flag). A makefile must have a Tab in front of the commands on each line.

When a command is executed through the make command, it uses make's execution environment. This includes any macros from the command line to the make command and any environment variables specified in the MAKEFLAGS variable. The make command's environment variables overwrite any variables of the same name in the existing environment.

Note:
Quote:

When the make command encounters a line beginning with the word include followed by another word that is the name of a makefile (for example, include depend), the make command attempts to open that file and process its contents as if the contents were displayed where the include line occurs. This behavior occurs only if the first noncomment line of the first makefile read by the make command is not the .POSIX target; otherwise, a syntax error occurs.
Comments: Comments begin with a # character, anywhere but in a shell command line, and continue to the end of the line.

Environment: The make command uses the MAKEFLAGS environment variable, if it exists.

Target Rules


Target rules have the following format:

target[target...] : [prerequisite...] [;command]
<Tab>command
Multiple targets and prerequisites are separated by spaces. Any text that follows the ; (semicolon) and all of the subsequent lines that begin with a Tab character are considered commands to be used to update the target. A new target entry is started when a new line does not begin with a Tab or # character.

You can also look up at the Makefile Tutorial here http://mrbook.org/tutorials/make/ !

That was the most important part of the build process, where make files are resposible for compiling almost each of the files in the source and putting them together for the useful apps/binaries/libraries etc.

Build Tricks

Used from wiki.

Seeing the actual commands used to build the software
Use the "showcommands" target on your 'make' line:

$ make -j4 showcommands
This can be used in conjunction with another make target, to see the commands for that build. That is, 'showcommands' is not a target itself, but just a modifier for the specified build.

In the example above, the -j4 is unrelated to the showcommands option, and is used to execute 4 make sessions that run in parallel.

Make targets
Here is a list of different make targets you can use to build different parts of the system:
  • make sdk - build the tools that are part of an SDK (adb, fastboot, etc.)
  • make snod - build the system image from the current software binaries
  • make services
  • make runtime
  • make droid - make droid is the normal build.
  • make all - make everything, whether it is included in the product definition or not
  • make clean - remove all built files (prepare for a new build). Same as rm -rf out/<configuration>/
  • make modules - shows a list of submodules that can be built (List of all LOCAL_MODULE definitions)
  • make <local_module> - make a specific module (note that this is not the same as directory name. It is the LOCAL_MODULE definition in the Android.mk file)
  • make clean-<local_module> - clean a specific module

Helper macros and functions
There are some helper macros and functions that are installed when you source envsetup.sh. They are documented at the top of envesetup.sh, but here is information about a few of them:

croot - change directory to the top of the tree
Code:
m - execute 'make' from the top of the tree (even if your current directory is somewhere else)
mm - builds all of the modules in the current directory
mmm <dir1> ... - build all of the modules in the supplied directories
cgrep <pattern> - grep on all local C/C++ files
jgrep <pattern> - grep on all local Java files
resgrep <pattern> - grep on all local res/*.xml files
godir <filename> - go to the directory containing a file
Speeding up the build

You can use the '-j' option with make, to start multiple threads of make execution concurrently.

In my experience, you should specify about 2 more threads than you have processors on your machine. If you have 2 processors, use 'make -j4', If they are hyperthreaded (meaning you have 4 virtual processors), try 'make -j6.

You can also specify to use the 'ccache' compiler cache, which will speed up things once you have built things a first time. To do this, specify 'export USE_CCACHE=1' at your shell command line. (Note that ccache is included in the prebuilt section of the repository, and does not have to be installed on your host separately.)

Building only an individual program or module
If you use build/envsetup.sh, you can use some of the defined functions to build only a part of the tree. Use the 'mm' or 'mmm' commands to do this.

The 'mm' command makes stuff in the current directory (and sub-directories, I believe). With the 'mmm' command, you specify a directory or list of directories, and it builds those.

To install your changes, do 'make snod' from the top of tree. 'make snod' builds a new system image from current binaries.

Setting module-specific build parameters
Some code in Android system can be customized in the way they are built (separate from the build variant and release vs. debug options). You can set variables that control individual build options, either by setting them in the environment or by passing them directly to 'make' (or the 'm...' functions which call 'make'.)

For example, the 'init' program can be built with support for bootchart logging by setting the INIT_BOOTCHART variable. (See Using Bootchart on Android for why you might want to do this.)

You can accomplish either with:

$ touch system/init/init.c
$ export INIT_BOOTCHART=true
$ make
or

$ touch system/init/init.c
$ m INIT_BOOTCHART=true

At last, after makefiles optimize all the processes and build the device specific parts including binaries and libs and apps necessary for it to get booted, the 'system' folder and the 'boot.img' folder are prepared in the out/target/product/device. The META-INF folder is prepared at instance and the system and boot.img are packed into a zip file(whose name is also processed by the makefiles ) and md5 sum prepared. The flashable zip gets prepared only if you run the "brunch" command or "lunch + mka" command.

The Build Tricks aren't *for fun*. This stuff is always gonna help you in the long run!

Credits -
>Firstly, my parents.
>Google, for Android!
>All the people over xda who have written awesome guides and done great works(Because that's what inspires me)
> @galaxyfreak , @speed_bot, @Red Devil, @thewisenerd, @vishal_android freak and all others who helped me every time and answered every(sensible) question of mine.
>The CyanogenMod, AOKP and OmniROM team!
> All those whom I missed.

Have a Happy time learning!
Last edited by v_superuser; 14th May 2014 at 06:34 PM.
The Following 62 Users Say Thank You to v_superuser For This Useful Post: [ View ]
14th May 2014, 06:32 PM   |  #2  
v_superuser's Avatar
OP Senior Member
Flag Jammu
Thanks Meter: 1,346
 
484 posts
Join Date:Joined: Jan 2013
Donate to Me
More
Reserved.
The Following 4 Users Say Thank You to v_superuser For This Useful Post: [ View ]
14th May 2014, 06:44 PM   |  #3  
Abhinav2's Avatar
Recognized Contributor / Themer
Thanks Meter: 4,656
 
2,319 posts
Join Date:Joined: Jul 2012
More
Good work. Another guide to feature in portal (?) Tipped

Cheers,
AJ
The Following User Says Thank You to Abhinav2 For This Useful Post: [ View ]
14th May 2014, 07:11 PM   |  #4  
vishal_android freak's Avatar
Recognized Developer / Recognized Contributor
Flag Mumbai
Thanks Meter: 5,284
 
1,784 posts
Join Date:Joined: Dec 2012
More
Awesome. Tipped second. All the best

Sent from my Nexus 5 using Tapatalk
The Following User Says Thank You to vishal_android freak For This Useful Post: [ View ]
14th May 2014, 07:13 PM   |  #5  
amit.lohar's Avatar
Senior Member
Flag Pune,India
Thanks Meter: 841
 
905 posts
Join Date:Joined: Jun 2013
More
Great work.. Nice guide
Thanks

Sent from xt1033, India
The Following User Says Thank You to amit.lohar For This Useful Post: [ View ]
14th May 2014, 08:32 PM   |  #6  
galaxyfreak's Avatar
Recognized Contributor
Flag Split
Thanks Meter: 5,657
 
1,653 posts
Join Date:Joined: May 2012
More
Another great one man!
The Following User Says Thank You to galaxyfreak For This Useful Post: [ View ]
15th May 2014, 05:52 AM   |  #7  
Dilesh Perera's Avatar
Senior Member
Flag Colombo
Thanks Meter: 806
 
1,302 posts
Join Date:Joined: May 2012
More
thanks very useful
The Following User Says Thank You to Dilesh Perera For This Useful Post: [ View ]
15th May 2014, 06:06 AM   |  #8  
TPD-21's Avatar
Senior Member
Flag Southern Borneo Island
Thanks Meter: 311
 
736 posts
Join Date:Joined: Feb 2014
More
Cool Noobs become a developer, is it possible?!
Could I be a developer by following this tutorial incredible? By the way thank you very much for the guidance.
15th May 2014, 08:16 AM   |  #9  
Red Devil's Avatar
Recognized Contributor
Flag Mumbai
Thanks Meter: 2,428
 
1,404 posts
Join Date:Joined: Oct 2012
More
Very good guide. Well written and easy to understand.
The Following User Says Thank You to Red Devil For This Useful Post: [ View ]
15th May 2014, 09:11 AM   |  #10  
Shivam Kuchhal's Avatar
Recognized Contributor
Flag Bijnor, Uttar Pradesh
Thanks Meter: 1,462
 
828 posts
Join Date:Joined: Dec 2012
More
Really good guide bro.
Easy to understand and very helpful, thanks for writing it.

The Following User Says Thank You to Shivam Kuchhal For This Useful Post: [ View ]
Post Reply Subscribe to Thread

Tags
android, aosp, envsetup, guide, process
Previous Thread Next Thread
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes