5,598,086 Members 42,858 Now Online
XDA Developers Android and Mobile Development Forum

Learn about the repo tool , manifests and local manifests and 5 important tips !

Tip us?
 
Red Devil
Old
(Last edited by Red Devil; 27th June 2013 at 06:55 PM.)
#1  
Red Devil's Avatar
Recognized Contributor - OP
Thanks Meter 2302
Posts: 1,381
Join Date: Oct 2012
Location: Mumbai
Default Learn about the repo tool , manifests and local manifests and 5 important tips !



Why this thread?

Hello xda-university !

As a newbie ( still remain a big newbie ) , I always had a tough time managing my ROMs and kernel projects.
I blindly followed what guides said .. and then realized how much time and energy I was wasting on downloading unnecessary $#!t from the internet as I have a very limited bandwidth and a particularly slow speed. ( around 80 kbps download speed )

While I am not a pro at this stuff, I would like to share a few tips and tricks about ROM building which will save some of your time , energy, space as well as a lot of the money while returning back to our great community out here.

After all efficiency is what matters in the end right ?

Also , understanding how the process works is more important that the process itself .

For what its worth , even a build bot can compile a ROM.. thats no biggie.

Learning how it works matters more !

Tired of waiting for that repo sync to end ?!!!

READ ON !
Prerequisites / Requirements


Some rough knowledge on compiling a ROM or experience of compiling a ROM.

Check out this thread ! : COMPILE From Source For Any Phone On Linux Distros [BUILD ENVIRONMENT TO COMPILE] by @mithun46
That was just to scare you . Though this will be more useful if you do have that knowledge .

Ok now moving on to the guide :


Google's Repo Tool


I ll just start off with a basic introduction of

Repository :
Repository is basically like a store room for code. It can be either selfmade i.e. you have yourself contributed it or claim to have written that code .. ( circled in yellow )

Or it can also be forked from someone else ... that is .. it is a version of someone else's code that you have modified a bit and pushed it to the remote server/ website. ( circled in black )

Look at the icons on the left of the name of the repository to know .


There are a variety of websites that offer such free hosting for code for eg : Github.com(which I will be using in this tutorial) and BitBucket.org among others..


Repo tool :
You maybe very familiar with the words repo init , repo sync , etc while downloading source.

Where did that repo command come from ?

Remember this ? :

Code:
curl https://dl-ssl.google.com/dl/googlesource/git-repo/repo > ~/bin/repo
Yep thats where it came from .

So in order to use the repo tool you need to run this command at least once.

What this does is it download the repo tool from Google's source and makes it executable for your own use.
How does the repo tool work ?

The repo tool is used for downloading a bunch of remote repositories that are on the internet thru one single command :

Code:
repo sync -j4
Details , details and more details
To explain this , I will take an example of a ROM, say AOKP.

For building AOKP you need to download its source code.

AOKP ROM developers have maintained the ROM's source code in a large number of Repositories that are located at www.github.com/AOKP.

What they have done is they have organized their source code into a number of folders for and uploaded it to separate repositories online.

What this does is , it enables all the ROM developers to work on a specific part of the source on their own computers and then push the source online to remote repository online so that he can share the code with all the developers of the same ROM. ( just a basic idea. it involves a lot more including pull requests and code reviews but we will stick with this for now )

Social coding at its best.

So during the command :

Code:
 repo sync -j4
What is done is the repo tool fetches all the remote repositories that the AOKP ROM devs have deemed necessary for building the ROM for a device through 4 threads.

If you have a slow internet connection then it is advised to use -j1 or -j2.
If you have a fast internet connection then you can use a -j16 or a -j32 whatever suits you the best.


Where does the repo tool take all the reference when downloading ??

The repo tool takes all the reference for the code to be downloaded from a Default.xml or a Manifest.xml.

The manifest.xml is a file which is written in xml markup and directs the repo tool to the repositories and their respective branches that need to be downloaded .

This is basically a huge list of all the repositories that need to be downloaded by the repo tool.
Where can this list of yours be found ?

Once you have initialized your repo sync which will be somewhat similar to this :

Code:
akshat@vaio:~$ mkdir -p kang
akshat@vaio:~$ cd kang
akshat@vaio:~/kang$ repo init -u https://github.com/AOKP/platform_manifest.git -b jb-mr1
Get https://gerrit.googlesource.com/git-repo
remote: Counting objects: 101, done
remote: Finding sources: 100% (101/101)
Receiving objects: 100% (2526/2526), 1.69 MiB | 67 KiB/s, done.
remote: Total 2526 (delta 1442), reused 2526 (delta 1442)
Resolving deltas: 100% (1442/1442), done.
From https://gerrit.googlesource.com/git-repo
 * [new branch]      maint      -> origin/maint
 * [new branch]      master     -> origin/master
 * [new branch]      stable     -> origin/stable
 * [new tag]         v1.0       -> v1.0
 * [new tag]         v1.0.1     -> v1.0.1
 * [new tag]         v1.0.2     -> v1.0.2
 * [new tag]         v1.0.3     -> v1.0.3
 * [new tag]         v1.0.4     -> v1.0.4
 * [new tag]         v1.0.5     -> v1.0.5
 * [new tag]         v1.0.6     -> v1.0.6
 * [new tag]         v1.0.7     -> v1.0.7
 * [new tag]         v1.0.8     -> v1.0.8
 * [new tag]         v1.0.9     -> v1.0.9
 * [new tag]         v1.1       -> v1.1
 * [new tag]         v1.10.0    -> v1.10.0
 * [new tag]         v1.10.1    -> v1.10.1
 * [new tag]         v1.10.2    -> v1.10.2
 * [new tag]         v1.10.3    -> v1.10.3
 * [new tag]         v1.10.4    -> v1.10.4
 * [new tag]         v1.11.0    -> v1.11.0
 * [new tag]         v1.11.1    -> v1.11.1
 * [new tag]         v1.12.0    -> v1.12.0
 * [new tag]         v1.12.1    -> v1.12.1
 * [new tag]         v1.12.2    -> v1.12.2
 * [new tag]         v1.2       -> v1.2
 * [new tag]         v1.3       -> v1.3
 * [new tag]         v1.3.1     -> v1.3.1
 * [new tag]         v1.3.2     -> v1.3.2
 * [new tag]         v1.4       -> v1.4
 * [new tag]         v1.4.1     -> v1.4.1
 * [new tag]         v1.4.2     -> v1.4.2
 * [new tag]         v1.4.3     -> v1.4.3
 * [new tag]         v1.4.4     -> v1.4.4
 * [new tag]         v1.5       -> v1.5
 * [new tag]         v1.5.1     -> v1.5.1
 * [new tag]         v1.6       -> v1.6
 * [new tag]         v1.6.1     -> v1.6.1
 * [new tag]         v1.6.10    -> v1.6.10
 * [new tag]         v1.6.10.1  -> v1.6.10.1
 * [new tag]         v1.6.10.2  -> v1.6.10.2
 * [new tag]         v1.6.2     -> v1.6.2
 * [new tag]         v1.6.3     -> v1.6.3
 * [new tag]         v1.6.4     -> v1.6.4
 * [new tag]         v1.6.5     -> v1.6.5
 * [new tag]         v1.6.6     -> v1.6.6
 * [new tag]         v1.6.7     -> v1.6.7
 * [new tag]         v1.6.7.1   -> v1.6.7.1
 * [new tag]         v1.6.7.2   -> v1.6.7.2
 * [new tag]         v1.6.7.3   -> v1.6.7.3
 * [new tag]         v1.6.7.4   -> v1.6.7.4
 * [new tag]         v1.6.7.5   -> v1.6.7.5
 * [new tag]         v1.6.8     -> v1.6.8
 * [new tag]         v1.6.8.1   -> v1.6.8.1
 * [new tag]         v1.6.8.10  -> v1.6.8.10
 * [new tag]         v1.6.8.11  -> v1.6.8.11
 * [new tag]         v1.6.8.2   -> v1.6.8.2
 * [new tag]         v1.6.8.3   -> v1.6.8.3
 * [new tag]         v1.6.8.4   -> v1.6.8.4
 * [new tag]         v1.6.8.5   -> v1.6.8.5
 * [new tag]         v1.6.8.6   -> v1.6.8.6
 * [new tag]         v1.6.8.7   -> v1.6.8.7
 * [new tag]         v1.6.8.8   -> v1.6.8.8
 * [new tag]         v1.6.8.9   -> v1.6.8.9
 * [new tag]         v1.6.9     -> v1.6.9
 * [new tag]         v1.6.9.1   -> v1.6.9.1
 * [new tag]         v1.6.9.2   -> v1.6.9.2
 * [new tag]         v1.6.9.3   -> v1.6.9.3
 * [new tag]         v1.6.9.4   -> v1.6.9.4
 * [new tag]         v1.6.9.5   -> v1.6.9.5
 * [new tag]         v1.6.9.6   -> v1.6.9.6
 * [new tag]         v1.6.9.7   -> v1.6.9.7
 * [new tag]         v1.6.9.8   -> v1.6.9.8
 * [new tag]         v1.7       -> v1.7
 * [new tag]         v1.7.1     -> v1.7.1
 * [new tag]         v1.7.2     -> v1.7.2
 * [new tag]         v1.7.3     -> v1.7.3
 * [new tag]         v1.7.3.1   -> v1.7.3.1
 * [new tag]         v1.7.4     -> v1.7.4
 * [new tag]         v1.7.4.1   -> v1.7.4.1
 * [new tag]         v1.7.4.2   -> v1.7.4.2
 * [new tag]         v1.7.4.3   -> v1.7.4.3
 * [new tag]         v1.7.5     -> v1.7.5
 * [new tag]         v1.7.6     -> v1.7.6
 * [new tag]         v1.7.6.1   -> v1.7.6.1
 * [new tag]         v1.7.7     -> v1.7.7
 * [new tag]         v1.7.7.1   -> v1.7.7.1
 * [new tag]         v1.7.7.2   -> v1.7.7.2
 * [new tag]         v1.7.7.3   -> v1.7.7.3
 * [new tag]         v1.7.7.4   -> v1.7.7.4
 * [new tag]         v1.7.7.5   -> v1.7.7.5
 * [new tag]         v1.7.7.6   -> v1.7.7.6
 * [new tag]         v1.7.8     -> v1.7.8
 * [new tag]         v1.7.8.1   -> v1.7.8.1
 * [new tag]         v1.7.8.2   -> v1.7.8.2
 * [new tag]         v1.8.0     -> v1.8.0
 * [new tag]         v1.8.1     -> v1.8.1
 * [new tag]         v1.8.2     -> v1.8.2
 * [new tag]         v1.9.0     -> v1.9.0
 * [new tag]         v1.9.1     -> v1.9.1
 * [new tag]         v1.9.2     -> v1.9.2
 * [new tag]         v1.9.3     -> v1.9.3
 * [new tag]         v1.9.4     -> v1.9.4
 * [new tag]         v1.9.5     -> v1.9.5
 * [new tag]         v1.9.6     -> v1.9.6
Get https://github.com/AOKP/platform_manifest.git
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0
curl: (22) The requested URL returned error: 404 Not Found
Server does not provide clone.bundle; ignoring.
remote: Counting objects: 1722, done.
remote: Compressing objects: 100% (1012/1012), done.
remote: Total 1722 (delta 839), reused 1552 (delta 709)
Receiving objects: 100% (1722/1722), 249.52 KiB | 50 KiB/s, done.
Resolving deltas: 100% (839/839), done.
From https://github.com/AOKP/platform_manifest
 * [new branch]      ics        -> origin/ics
 * [new branch]      jb         -> origin/jb
 * [new branch]      jb-mr1     -> origin/jb-mr1
 * [new tag]         Build-2    -> Build-2
 * [new tag]         Build-3    -> Build-3
 * [new tag]         Build-4    -> Build-4
 * [new tag]         Build-5    -> Build-5
 * [new tag]         Milestone-1 -> Milestone-1
 * [new tag]         jb-mr1_build-1 -> jb-mr1_build-1
 * [new tag]         jb-mr1_build-4 -> jb-mr1_build-4
 * [new tag]         jb-mr1_build-6 -> jb-mr1_build-6
 * [new tag]         jb-mr1_milestone-1 -> jb-mr1_milestone-1

Your identity is: akshatshenoy <akshat.shenoy@gmail.com>
If you want to change this, please re-run 'repo init' with --config-name

repo has been initialized in /home/akshat/kang
So I have to go to : the /kang folder in my home directory and open the .repo file. Which will look somewhat like the following images :


Important tip ! : While browsing through files in a directory , to view hidden files press control + H and to view the exact directory of the folder which you are browsing through press control + L ..



Opening this file with a text editor will tell you the list of the repos that need to be downloaded by the repo tool :

For AOKP JB-MR1 it will look somewhat like this


Now just a flashback to the original repo init command we had applied earlier
Code:
akshat@vaio:~/kang$ repo init -u https://github.com/AOKP/platform_manifest.git -b jb-mr1
When the repo init command is carried out , the repo tool grabs the default.xml / manifest.xml from AOKP's github which is mentioned in the init command :

So open up a web-browser and navigate to that page and you will realize that it is the same manifest.xml that is present in the .repo of the source directory where you have performed a repo init at.

Now when we repo sync ,
the repo tool downloads each repo as a specific project as shown in the picture : (highlighted)



So, basically, there are 415 projects to be synced in order to build AOKP that has been defined in the manifest.xml

Once those 415 projects have been downloaded , they will be stored in the .repo and a copy of it will be synced as a work-tree .

This work-tree comprises of the files and folders that appear automagically once the repo sync is done with all 415 projects synced !

It is that part of the source code which you actually work on and make your device specific changes to !
Understanding the manifest :

1) The remote tag :
syntax :
Code:
<remote  name=" any name here "
           fetch="url of the remote website/hosting/server " 
            review="url of the code review of the repo :optional: " />
This tag just specifies a " remote " location for the repo tool to get your repositories from . This allows you to then associate a particular repository with that remote.

examples of remotes :

Code:
  

<remote  name="github"
           fetch="https://github.com/" />

<remote name="bitbucket" fetch="https://bitbucket.org"/>


  <remote  name="aokp"
           fetch="https://github.com/"
           review="gerrit.aokp.co" />
This allows you to use multiple code hosting sites , private or public and thereby enabling a trouble-free maintenance of your team's source code.


Now like the remote is specified , you can base you're manifest at a particular remote and define it as your default remote.

The advantages of this will be seen in (2) and (3) during the adding of the project tags to the manifest.xml

The syntax of the default remote is as follows :

Code:
 
<remote  name=" any name here "
           fetch="url of the remote website/hosting/server " 
            review="url of the code review of the repo :optional: " />

<default revision="any/path/here"
           remote="remote hosting name defined earlier"
           sync-j="x" />
where x is the default number of threads to be used when it is not defined in the repo sync command.

best example of a default revision that is widely used in almost every ROM default.xml is

Code:
  <remote  name="aosp"
           fetch="https://android.googlesource.com/"
           review="https://android-review.googlesource.com/" />

   <default revision="refs/tags/android-4.2.2_r1.2"
           remote="aosp"
           sync-j="4" />
2) The base projects
As explained earlier the projects refer to the repositories that have to be fetched by the repo tool.

The 2) tag in the picture above is used to download the repository from your default remote's server.

The basic syntax of it is :

Code:
 <project path=" path/in/work/tree" name="/repo/name/in/your/default" />
Examples :

Code:
   
<project path="abi/cpp" name="platform/abi/cpp" /> 
 <project path="bootable/diskinstaller" name="platform/bootable/diskinstaller" />
This downloads those particular repos from the actual android source .

It is advisable to use this particular tag from the remote you will download the most number of repositories from as it cuts down on the effort of editing the manifest.xml

Many ROMs use the AOSP as the base as it is easier for them to keep their source code upto date without wasting their bandwidth on uploading the same repositories to their own gits .

3) The other projects
This is for the non default remote projects

Their general syntax is :
Code:
 
<project path=" directory in work tree" name="name of the repository in remote" remote="name of remote defined earlier" revision="branch of the repository" />
NOTE : Default revision is master. So if you dont define the revision it is considered as the master branch!

Examples for this are :
Code:
 
<project path="bootable/recovery" name="CyanogenMod/android_bootable_recovery" remote="cm" revision="cm-10.1" />
For this of course the earlier remote "cm" should be mentioned.
Quick tip !
Indirectly , the total url path to the repository should be = the fetched URL of the remote + the name of the repo in the project line.

In this case
Code:
https://github.com/ && CyanogenMod/android_bootable_recovery 

in total : 

https://github.com/CyanogenMod/android_bootable_recovery
For the sole purpose of making the life of a developer easier , CyanogenMod / AOKP and almost all other ROMs named their own repositories as

Code:
 android_path_to_directory_in_work_tree || path_to_directory_in_work_tree
Such as :



If you notice the name of the repository : vendor_aokp corresponds to the aokp folder in the source folder in my source.

Thus the name of the repository gives you the path to the same files in your repository.

So much win !
Maintaining your own projects using manifest.xml
Now that you have understood how the manifest.xml works .. you can even maintain your own projects with the help of a manifest.xml ! it is not necessary it has to be used only for ROMs !

Thats the best bit !

So the next time you have to pull a large number of repositories and maintain them you can create your own default manifest in your own github and add the repos as mentioned above.

Example I use sometimes :

Code:
 <?xml version="1.0" encoding="UTF-8"?>
<manifest>

<remote  name="red-devil"
         fetch="https://github.com/red-devil" /> 

<project path="Cherrypicking/frameworks/base" name="AeonROM/android_frameworks_base" remote="red-devil" revision="master" />
<project path="Cherrypicking/frameworks/native" name="AeonROM/android_frameworks_native" remote="red-devil" revision="master" />
<project path="Cherrypicking/frameworks/av" name="AeonROM/android_frameworks_av" remote="red-devil" revision="master" />

</manifest>

Understanding local manifests :

What is the local_manifest.xml ?
A local manifest is like a specific copy of the manifest of a particular ROM that is maintained by YOU the ROM compiler in your own computer.

This local manifest allows you to override some of the projects mentioned in the manifest and hence allows you to delete / add / modify the source code according to your preferences.

Best part about this is that every time you sync your source , even the onces mentioned in the local manifest get updated by themselves.

Hence, you dont need to keep track of all your device trees and kernel sources manually, let the repo tool do the work for you.

Let us see how this wonderful tool works :

  1. Go to the folder where you have inited the repo and open the .repo ( press ctrl + H if it is NOT visible ).
  2. Create a folder named local_manifests
  3. Download this local_manifest.txt and rename it to local_manifest.xml and copy it in the / .repo / local_manifests / directory

NOTE : In some older ROMs where an older version of the repo tool is used the local manifests has to be placed like




As opposed to a folder named local_manifests as illustrated in the above pictures
Functions of the local manifest :

The local manifest looks somewhat similar to the main manifest.xml ... however in this you can even remove projects or add them

> To remove projects the basic syntax is :

Code:
 
<remove-project name="name of the project" />
What to remove :

Consider my case

I am building AOKP JB-MR1 for the HTC Explorer.

As a result of that I do not need the various device trees that are officially supported by AOKP.

Such all these :




Quote:
So why do I waste my data and time downloading those ?

I think I should modify the default.xml / manifest.xml .

NO !


Beware ! If you try to edit the default.xml / manifest.xml the git will tell you to commit these changes as they do NOT match with the online / and the synced version of the default.xml that you have originally initializeed with.

So this means you will either have to commit those changes , push it to github and update the default.xml
This is not possible unless you are a device maintainer for a ROM and even if you are such drastic changes mean it can affect the other users of the same default.xml . So chances are you can never bring about a change in the default.xml.

So dont even think about editing the default.xml


OR

You will have to revert back to the original default.xml that you was present in your .repo and create a local_manifest.xml and remove projects / add them .


How do I do that ?

Basic syntax for removing them is :
Code:
<remove-project name="name_of_project" />
As shown in the image since I dont need those projects the syntax will be somewhat like :

Code:
 
 <remove-project name="AOKP/device_samsung_d2att" />
  <remove-project name="AOKP/device_samsung_d2tmo" />
For removing the 1st two projects as shown in the picture above

In a similar way , I went on removing projects that were not required... only device trees and saved the local_manifests.xml

Remember when I earlier synced there were 415 projects ?




Look at it now .. slimmed down around 50 projects !

That means a much quicker repo sync for me !
> How to add projects to get it synced :

To add projects just use the code like I explained in the manifests.xml

Basic syntax is :
Code:
 
<remote  name=" any name here "
           fetch="url of the remote website/hosting/server " 
            review="url of the code review of the repo :optional: " />

<project path=" directory in work tree" name="name of the repository in remote" remote="name of remote defined earlier" revision="branch 
of the repository" />
EG :

I wanted to add my own device tree of AOKP and the kernel source for my device in the local manifests cause I am too lazy to manually download them

Code:
 <project path="device/htc/pico" name="android_device_htc_pico_aokp" remote="red-devil" revision="master"/>
  <project path="kernel/htc/pico" name="android_kernel_htc_pico" remote="red-devil" revision="cm-10.1" />



So ultimately this is the sample local_manifest.xml I used :

 
Code:
 
<?xml version="1.0" encoding="UTF-8"?>
<manifest>
  <remote  name="red-devil"
           fetch="https://github.com/red-devil/" />
  <project path="device/htc/pico" name="android_device_htc_pico_aokp" remote="red-devil" revision="master"/>
  <project path="kernel/htc/pico" name="android_kernel_htc_pico" remote="red-devil" revision="cm-10.1" />
  <remove-project name="AOKP/device_acer_a510" />
  <remove-project name="AOKP/device_asus_tilapia" />
  <remove-project name="AOKP/device_htc_evita" />
  <remove-project name="AOKP/device_htc_s4-common" />
  <remove-project name="AOKP/device_lge_iprj-common" />
  <remove-project name="AOKP/device_lge_mako" />
  <remove-project name="AOKP/device_lge_p930" />
  <remove-project name="AOKP/device_lge_su640" />
  <remove-project name="AOKP/device_lge_vs920" />
  <remove-project name="AOKP/device_motorola_maserati" />
  <remove-project name="AOKP/device_motorola_omap4-common" />
  <remove-project name="AOKP/device_motorola_solana" />
  <remove-project name="AOKP/device_motorola_spyder" />
  <remove-project name="AOKP/device_motorola_targa" />
  <remove-project name="AOKP/device_motorola_umts_spyder" />
  <remove-project name="AOKP/device_samsung_aries-common" />
  <remove-project name="AOKP/device_samsung_d2-common" />
  <remove-project name="AOKP/device_samsung_celox-common" />
  <remove-project name="AOKP/device_samsung_d2att" />
  <remove-project name="AOKP/device_samsung_d2tmo" />
  <remove-project name="AOKP/device_samsung_d2usc" />
  <remove-project name="AOKP/device_samsung_d2vzw" />
  <remove-project name="AOKP/device_samsung_galaxys2-common" />
  <remove-project name="AOKP/device_samsung_hercules" />
  <remove-project name="AOKP/device_samsung_i605" />
  <remove-project name="AOKP/device_samsung_i9100" />
  <remove-project name="AOKP/device_samsung_i9100g" />
  <remove-project name="AOKP/device_samsung_i9300" />
  <remove-project name="AOKP/device_samsung_l900" />
  <remove-project name="AOKP/device_samsung_maguro" />
  <remove-project name="AOKP/device_samsung_manta" />
  <remove-project name="AOKP/device_samsung_msm8660-common" />
  <remove-project name="AOKP/device_samsung_msm8960-common" />
  <remove-project name="CyanogenMod/android_device_samsung_omap4-common" />
  <remove-project name="AOKP/device_samsung_smdk4412-common" />
  <remove-project name="CyanogenMod/android_device_samsung_smdk4412-qcom-common" />
  <remove-project name="AOKP/device_samsung_t0lte" />
  <remove-project name="AOKP/device_samsung_t0lteatt" />
  <remove-project name="AOKP/device_samsung_t0ltetmo" />
  <remove-project name="AOKP/device_samsung_toro" />
  <remove-project name="AOKP/device_samsung_toroplus" />
  <remove-project name="AOKP/device_samsung_tuna" />
  <remove-project name="AOKP/device_samsung_vibrantmtd" />
  <remove-project name="AOKP/device_sony_hayabusa" />
  <remove-project name="AOKP/device_sony_mint" />
  <remove-project name="AOKP/device_sony_odin" />
  <remove-project name="AOKP/device_sony_tsubasa" />
  <remove-project name="AOKP/device_sony_yuga" />
  <remove-project name="AOKP/device_sony_blue-common" />
  <remove-project name="AOKP/device_sony_fusion3-common" />
  <remove-project name="CyanogenMod/android_device_sony_qcom-common" />
  <remove-project name="AOKP/device_sony_common" />
  <remove-project name="AOKP/packages_apps_Torch" />
</manifest>


Result of this : the number of projects being downloaded reduced from 415 to 364 !




So , I hope you guys understood at least a bit about manifests and how the repo sync process actually works..


Read on for some tips !

Remember the drill
→ Follow the Forum Rules → Use Search before posting → Post in the right sections → Do not spam the board

The Following 97 Users Say Thank You to Red Devil For This Useful Post: [ Click to Expand ]
 
Red Devil
Old
(Last edited by Red Devil; 31st March 2014 at 06:39 AM.)
#2  
Red Devil's Avatar
Recognized Contributor - OP
Thanks Meter 2302
Posts: 1,381
Join Date: Oct 2012
Location: Mumbai
Some miscellaneous tips :



Tip # 1 : What all can I remove using local manifests ?


Since we just learnt about the local_manifests, I thought I'd just explain this first !

Note : What I refer to as remove from manifest is actually removing using local manifest.

If you don't know what that is yet then read the the 1st post !


The following repos are what you can remove :
  1. Remove official device trees if you are not planning to build for them :
     


    What this includes :

    Code:
    android_device_htc_codename 
    android_device_sony_codename 
    android_device_samsung_codename
    If you don't own an HTC device you don't need any of their repos ! Same goes with Sony , Samsung and all other manufacturers.

    Note: Do NOT delete generic repos like android_device_common etc.

    If you own a samsung/sony/htc/blah device DO NOT Remove their common/ generic repos using the local manifest !

    Also download a few sample device trees which are similar to your device so that you can reference the way the device trees are while compiling your unofficial ROM.

    This process is called " KangBang"-ing.

  2. Remove all kernel source repos that you do not need !

     

    Remove all projects named :
    Code:
     
    android_kernel_oem_codename 
    kernel_oem_codename
    or similar !

    If you don't plan on building for that device , Why do you need its kernel source code !

    And most of these sources are HUGE !


  3. Remove all the unnecessary packages that are present in the manifest; for example if your phone does not have flash, there is no point downloading the torch.apk.. so might as well remove that from manifest
     


    This includes many packages like CDMA specific packages for a GSM phone , torch , flashlight packages for a phone that doesnt have flash , unnecessary live wallpapers , and many more stuff.

    If your phone's ROM is less you can also remove many packages from syncing itself instead of removing them from the build !


    Saves a lot of bandwidth as well as time !

    Mostly all these packages have project names similar to :
    Code:
     
    android_packages_apps_appname
    packages_apps_appname
    android_packages_wallpapers_wallpapername
    and so on.


  4. Remove all unnecessary vendor trees from the manifest.. Why do you need the blobs for a device you are not building for !!

     


    Remove all device specific vendor packages that include the blobs of the device.

    Eg:

    Code:
     
    android_vendor_oem_codename_proprietary
    Use your common sense to remove these repos.

    Nexus devices use the blobs from TheMuppets.. so if you are not building for a Nexus then remove those for example !


  5. Remove all the darwin tool chains, and other darwin packages from the manifest if you are not building from a mac.
     


    Remove all of these toolchains if you are NOT building on a macintosh :

    Code:
     
    <project path="prebuilts/gcc/darwin-x86/arm/arm-eabi-4.6" name="platform/prebuilts/gcc/darwin-x86/arm/arm-eabi-4.6" groups="pdk,darwin,arm" />
    <project path="prebuilts/gcc/darwin-x86/arm/arm-linux-androideabi-4.6" name="platform/prebuilts/gcc/darwin-x86/arm/arm-linux-androideabi-4.6" groups="pdk,darwin,arm" />
    <project path="prebuilts/gcc/darwin-x86/mips/mipsel-linux-android-4.6" name="platform/prebuilts/gcc/darwin-x86/mips/mipsel-linux-android-4.6" groups="pdk,darwin,mips" />
    <project path="prebuilts/gcc/darwin-x86/x86/i686-linux-android-4.6" name="platform/prebuilts/gcc/darwin-x86/x86/i686-linux-android-4.6" groups="pdk,darwin,x86" />
    Other packages may vary according to the ROM source you are syncing.
  6. Remove all unnecessary repos that may not be needed.. for eg exynos specific hardware packages if your building for a qcom device!
     


    What I said .. or in some cases like my phone the HTC Explorer, uses display-legacy from Qcom instead of the default one..

    So I just remove all the display specific packages that are not needed from manifest.

    Caution ! I am not mentioning examples as this may even break your build if you DON'T know what you're doing! So be careful and use your common sense !

Tip # 2 : Use the reference flag before you initialize a repo !



The repo tool is somewhat of a blessing for android ROM developers.

There is a certain flag called as a Reference flag that can be used before you repo sync while initializing a repo.

This flag has a basic syntax as follows :
Code:
repo init --reference=~/root/of/already/downloaded/source -u git://url.com/ROM.git -b branch
Consider the following scenario:

I have already downloaded the CyanogenMod 10.1 source mechanically as many guides state :

Code:
repo init -u git//www.github.com/CyanogenMod/android.git -b cm-10.1
repo sync -j4
and this is what I ended up with


Quote:
So , now I have compiled CyanogenMod and I am bored of it. Want to get hands on a new ROM, say AOKP.

So now another 15 GB of bandwidth for this ?!

NO !



I will use the reference flag!
as follows :



How does this work ?

What the repo tool does it checks the locally downloaded source and skips downloading that it Symlinks the code from the existing source to the new source.

Cool tool isn't it ?

This and local_manifests cut down a majority of the download time during repo sync !

Some more help with the Reference flag errors !

http://forum.xda-developers.com/show...&postcount=112 - By thewisenerd
Tip # 3 : Use CCACHE while compiling


CCACHE : that is also known as compiler cache is another trusted friend of the ROM / Kernel developers.

It cuts down a lot of the time required for compilation.

How to use ccache :

1) Open a terminal ( cntrl + alt + t )
2) Enter gedit .bashrc
3) Add the following code :
For Jellybean :
Code:
 
export USE_CCACHE=1
export CCACHE_DIR=/source-directory/prebuilts/misc/linux-x86/ccache
and for ICS and older
Code:
export USE_CCACHE=1
export CCACHE_DIR=/source-directory/prebuilt/linux-x86/ccache/ccache
Once that is done just type in this command while you are in youre source directory
Code:
 

prebuilts/misc/linux-x86/ccache/ccache -M 50G
ICS and lower change it accordingly.

50 G ---> 50 GB of ccache..

Ideal values range for 50-100 GB.

Though if you want you can set it to around 40 GB

How does ccache work ?

Well , in short , what it does is that for C and C++ programs it caches the output. One it detects that the program is getting repeated it directly sends out the output and thus reduces the time of compilation.

After using ccache you will really feel the difference in the compilation speeds of your ROMs and kernels .



Tip # 4 : Use a custom toolchain

Try to use toolchains like linaro toolchains for compiling your ROMs and kernels.

For me personally it made a lot of difference in the performance of the ROM compiled. Nearly a 20-30 % increase in the performance.

Where can I get these toolchains ?

This is the link to download linaro's toolchains. HERE.

Get the latest one from the android section.

How to setup the toolchain?

Just extract the toolchain in a directory similar to the one I have mentioned in the picture




and edit these 2 lines in the envsetup.sh




To compile the inline kernel using a custom toolchain :

Place your toolchain in the ~source~/prebuilt/linux-x86/toolchains/



and in your device tree edit the BoardConfig.mk for eg :



In addition to this, you may need some device specific patches too. That maybe you will be able to find out in your device specific forums.


Tip # 5 Cherry-picking



So here's the scenario.

I have built a ROM and simeltaneously CyanogenMod development is taking place and their developer is fixing some bugs that are present in the device tree of the ROM.

He has implemented some changes that might help me fix stuff in my device tree thereby improving my ROM .

So what do I do ?

Manually apply those commits ?

There is a better way than that. Manually applying commits may result in a few human errors in your source code and thereby may cause some unwanted errors. So why not apply the CyanogenMod dev's commits itself!

To do this, we use a git feature called cherry-picking.

How to cherry pick commits :



First change your directory to the directory in which you want to apply the commits in.

I will take an example of my phone's device tree.

Code:
 
cd device/htc/pico
The best thing is to initialize an empty git and pull your own repo from here.

This allows you to push your changes to your github so that you can track your changes effectively.

If you dont have a repo on your github then push one right away or fork someone else and do :


So I do this :
Code:
git init
git pull git@github.com:red-devil/android_device_htc_pico_aokp.git
Once that is done , I just make sure everything is committed and local device tree matches the device tree online at my github.

then I pull all the commits from the CyanogenMod maintainer's github.

Basic syntax for this is :
Code:
 
git remote add somename git://github.com/user/repo.git

git fetch somename
So what I do is :

Code:
git remote add cyanogenpico git@github.com:TeamPico/android_device_htc_pico.git
git fetch cyanogenpico
Now just check using the git log function wheter the last commit shown is the last commit on YOUR repository on github.

In this case the last commit at git@github.com:red-devil/android_dev..._pico_aokp.git should match the last commit on the git log of my local source.

This just ensures that you are going right at least up till now.

Now start the cherry picking.

The syntax is

Code:
 
git cherry-pick <sha-of-commit>
So what I do is :

Code:
 
git cherry-pick aa39fd899e321395f3c08c9a7f606ac458bbbd1b
and grab that commit.

If that is done smoothly

The terminal will start a new line automatically

and you can do this :

Code:
git push git-ssh
in my case

Code:
git push git@github.com:red-devil/android_device_htc_pico_aokp.git

If something goes wrong.

it may look something like this

Code:
akshat@vaio:/media/akshat/HD/android/kang/device/htc/pico$ git cherry-pick aa39fd899e321395f3c08c9a7f606ac458bbbd1b
error: could not apply aa39fd8... cut out debugging
hint: after resolving the conflicts, mark the corrected paths
hint: with 'git add <paths>' or 'git rm <paths>'
hint: and commit the result with 'git commit'
This means that there are some lines in a file that this particular commit is looking for which are not present, thus it shows that the git has added / modified a few lines to apply that commit.

It marks those changes as uncommitted and thus it wont allow you to push those changes.

To make it push-able.. you can do a

Code:
 
git status
Check which files have changed.

Then

manually edit and logically correct those changes .

That for you is hacking my friends.

See this image for an example of how a bad head looks like




This is how I corrected the changes..



after this is done.

Code:
git add .
git commit -m " name of commit "
git push url.git



Important Note :

For this case I just had to remove the head and the settings worked after that..

This may NOT be the case with your repo.

Overlapping commits sometimes may break the build .

So you need to logically remove the bad heads and also ensure the package/code isn't damaged during the process !

Check out Commits from ROMs like PAC , RootBox , ReVolt etc to know how exactly to cherry-pick stuff and take reference from them to make your code work.


Still got some doubts ?

Ask on the thread, not via pm. This ensures everyone is cleared of the doubt.
Got a compilation error ?


Head over here for some more help :
[HELP/Q&A][SourceBuilding,AllDevices] The Source Building Q&A Help Thread by @speed_bot
Finally the last bit

This guide had featured on the portal : Article : http://www.xda-developers.com/androi...the-repo-tool/
This guide has also been mentioned in the official Omni Wiki : http://docs.omnirom.org/Setting_Up_A...le_Environment
The Credits

I would like to thank @speed_bot and @legen_dary , @Nihar.G among others who solved cleared my doubts whenever I asked them anything silly.

You guys for reading this whole piece and xda-developers for the platform.

Always remain a newbie and NOT a noob.
Know the difference.

Remember the drill
→ Follow the Forum Rules → Use Search before posting → Post in the right sections → Do not spam the board

The Following 94 Users Say Thank You to Red Devil For This Useful Post: [ Click to Expand ]
 
Red Devil
Old
(Last edited by Red Devil; 11th December 2013 at 06:09 AM.)
#3  
Red Devil's Avatar
Recognized Contributor - OP
Thanks Meter 2302
Posts: 1,381
Join Date: Oct 2012
Location: Mumbai
<reserved for the future>

Remember the drill
→ Follow the Forum Rules → Use Search before posting → Post in the right sections → Do not spam the board

The Following 6 Users Say Thank You to Red Devil For This Useful Post: [ Click to Expand ]
 
Nihar.G
Old
#4  
Nihar.G's Avatar
Recognized Contributor
Thanks Meter 1185
Posts: 642
Join Date: May 2012
Location: Pune
Very very well written!

Good work. Keep it up!

Current Device: Google Nexus 5


The Following User Says Thank You to Nihar.G For This Useful Post: [ Click to Expand ]
 
Rishik999
Old
#5  
Rishik999's Avatar
Senior Member
Thanks Meter 604
Posts: 360
Join Date: Aug 2012
Great work sir!
Bookmarked it,soon gonna need it
Thanks
 

My Work : http://pastebin.com/aehdzuA8

Active on Facebook?
Here you are with Gaming Geeks aka G-Droids!
Pico lover?Here is the right place

Github :https://github.com/Rishik999
The Following User Says Thank You to Rishik999 For This Useful Post: [ Click to Expand ]
 
-NickHalden-
Old
#6  
-NickHalden-'s Avatar
Recognized Contributor
Thanks Meter 1804
Posts: 1,035
Join Date: Apr 2012
Location: Gotham

 
DONATE TO ME
Very well done. Thanks for sharing
Guide Ride-Everything you need, From a Noob to a Dev. 80+ awesome guides to help you in all your endeavors

Device hierarchy. Galaxy MINI>Xperia Tipo Dual>HTC Pico>Moto G

Agree or disagree, idgaf. Adios ! Savvy, aye?
The Following User Says Thank You to -NickHalden- For This Useful Post: [ Click to Expand ]
 
usmslm102
Old
#7  
usmslm102's Avatar
Senior Member
Thanks Meter 152
Posts: 192
Join Date: Mar 2012
Location: Mumbai
Great guide ...
Really need this.

Sent from my GT-S7562 using Tapatalk 4 Beta
The Following User Says Thank You to usmslm102 For This Useful Post: [ Click to Expand ]
 
matt95
Old
#8  
matt95's Avatar
Recognized Contributor
Thanks Meter 2292
Posts: 5,765
Join Date: Aug 2010
Location: Verona

 
DONATE TO ME
yeah good guide now my mind is much more clearer than before


Donate and support my studies at the MIT University! Unruu HTC RUUs | Setup Android Development Environment on Ubuntu | Decrypt HTC RUUs
The Following User Says Thank You to matt95 For This Useful Post: [ Click to Expand ]
 
karanrajpal14
Old
#9  
Senior Member
Thanks Meter 305
Posts: 936
Join Date: Feb 2011
Location: Bangalore
Thanks you so much!
The Following 2 Users Say Thank You to karanrajpal14 For This Useful Post: [ Click to Expand ]
 
# root
Old
#10  
# root's Avatar
Recognized Contributor
Thanks Meter 1423
Posts: 458
Join Date: Feb 2012
Location: Mumbai

 
DONATE TO ME
Good job!!!
"It Only Exists, If You Believe In It"

Current Phone: Nexus 4 (16GB)
ROM: Random!
Kernel version: 3.4.0

Sold Devices: Samsung Galaxy Fit (beni), HTC Explorer (pico)

My Work:
CyanMobileX, CM7, AOSParanoidAndroid 2+ & 3+, MiniCM10, MiniCM7, GingerDX (Device Maintainer), AOSP ICS, #TeamFun, OmniRom, CM10.1, CM11 And Much More!
GingerDX for Galaxy Mini and HTC Wildfire S

The Following User Says Thank You to # root For This Useful Post: [ Click to Expand ]
Tags
compiling, guide, repo tool, rom, sync
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes