FORUMS
Remove All Ads from XDA

[Guide] How-to Automate Your ROM Build Process Using Jenkins - Setup Nightlies

n/a posts
Thanks Meter: 0
 
By AndroidSlave, Guest on 2nd October 2013, 05:40 PM
Thread Closed Email Thread
HOW TO AUTOMATE YOUR ROM BUILDING PROCESS

... using


Special thanks to @chasmodo for all his support, as well as his wonderful How-To build CM thread located HERE.

I. Donations and Support...

Feel free to post any question here, as well as provide your own scripts. As far as I'm concerned, there are no stupid questions; you are, however, required to search before posting.

Please do not donate to me, donate to the Testicular Cancer Society (Click Here).

II. Introduction...

This guide will direct you on how to setup your android build environment to automate your builds using Jenkins and various shell/bash scripts. As this was initially drafted, it was targeted towards CyanogenMod 10.2, but I am quite confident it can easily be easily modified to work with anything you choose, including LP builds.

This can be a useful process if you want to setup nightly/weekly builds, or if you just want to have a easy-to-use web interface to run your builds without entering the shell.

This guide assumes you have an active build environment (preferably Ubuntu-based), and general linux knowledge.

III. Initializing your build environment...

You will first have to have your build environment setup, and you should already be able to successfully build CM, AOSP, AOKP, or other variants, before moving forward. Furthermore, you are expected to have simple scripting knowledge, or you will not be able to do this. If you need assistance with setting up an android build environment (which is not the focus of this thread), you can get started by Clicking Here, or by looking at @chasmodo's thread by Clicking Here.

It is important to note that your Jenkins home directory is at "/var/lib/jenkins" (not /home/jenkins). As such, to get your build directory in the proper location, you will login normally to your shell, and "su jenkins" (you may have to use su policy, if you don't know how to do this, you may just want to login as root). Do Not login to the system-jenkins-user shell by changing the jenkins user password (I am referring to the System User named jenkins, not your jenkins web-interface). Once logged in as jenkins, do a "cd ~", create your directory (e.g., "cm10.2"), and repo init there (e.g., in /var/lib/jenkins/cm10.2 directory).

IV. Installing Jenkins and Apache Daemons...

Once your build environment has been initialized, you will have to get Jenkins/Hudson setup and running. You can read more about Jenkins software by Clicking Here.

Further, you will need to install and setup apache web server in order to access the Jenkins web interface. Click Here for more information on apache.

The jenkins install process can easily be setup by following the guide Located Here.

After you have successfully installed Jenkins and apache, you may proceed to setting up your automated builds.

V. Setting Up Jenkins Security...

Be sure to setup a Jenkins user account, or a remote user can get your setup and screw up your machine.
Follow the article here:
https://wiki.jenkins-ci.org/display/...Security+Setup

Continued in Post #2
The Following 32 Users Say Thank You to AndroidSlave For This Useful Post: [ View ] Gift AndroidSlave Ad-Free
 
 
AndroidSlave
2nd October 2013, 05:44 PM |#2  
Guest
Thanks Meter: 0
 
More
V. Configuring your first build...

.... A. Creating the new build...

Once you have setup your jenkins server (by going to http://localhost:8080, or port 80 if you set it that way), log in with the username/password you configured.

After you have logged in, click "New Job" in the left toolbar.

Screenshot:

Then, enter the "Job name". Please Note to NOT use spaces or slashes in the name, as it causes build issues. If you need a space, use an underscore ("_"). Check the radio-box "Build a free-style software project", and hit "OK".

Screenshot:

Now, continue to the procedure listed directly below (i.e., in section "B", "How to have Jenkins call your local scripts...")

.... B. How to have Jenkins call your local scripts...

As opposed to attempting to have Jenkins' propriety interface try to run your build process directly, it is best to have it call out for your individualized shell/bash scripts.

To do so, under the "Execute Shell" Field in the "Build" section, you will call out your script by entering the following code:

Code:
#!/bin/bash
cd ~/FOLDER/
source SCRIPT_NAME.sh
Screenshot:




Once your code has been input, the next step is to create your script in your build directory. In this instance, the script is called "buildscriptclobber.sh" and the directory is "~/cm10.2" (~ is home). It is important to remember that your home directory is the Jenkins user home directory, which gets installed at /var/lib/jenkins (NOT /home/jenkins).

As above mentioned, you will login normally to your shell, and "su jenkins". Once logged in as jenkins, do a "cd ~/FOLDER" (e.g., "~/cm10.2"). Then, use your favorite text editor (e.g., pico, installed by sudo apt-get install pico), and enter your script.

A good script I wrote will make clobber and repo sync every time. It is as follows:
Code:
#!/bin/bash
cd ~/FOLDER/
make clobber
cd ~/FOLDER/
export PATH=~/bin:$PATH
export USE_CCACHE=1
repo sync
vendor/cm/get-prebuilts
source build/envsetup.sh && time brunch DEVICE
You may notice above that I call out the folder ("cd"), and set the "PATH" multiple times, especially when its already set in your ".bashrc". This is because the script will give errors if you do not do so, because Jenkins won't consistently read your ".bashrc" or other external configurations. It is important to always do this type of thing when shell scripting with Jenkins. The "time" command may be omitted, because Jenkins log will show the total build time throughout, so its partially redundant, but I like it in there.

Once the script has been setup, do not forget to "chmod a+x" or it will not run.

VI. Automating your first build...

WIP SETTING UP H H H TRIGGER

VII. Configuring and automating additional builds...

Follow the same steps as above, just be sure the Cron Triggers are set to not overlap. For Example, if you have setup H H H(0-5) for one, and you know your builds may take up to two hours, then setup H H H(2-7) for the next.

Continued in Post #3
The Following 13 Users Say Thank You to For This Useful Post: [ View ] Gift Ad-Free
AndroidSlave
2nd October 2013, 05:45 PM |#3  
Guest
Thanks Meter: 0
 
More
VIII. Scripting your builds in general...

.... A. Use individual shell scripts for each process...

The beauty of having a Jenkins environment is your scripting capabilities. There are many possibilities, and many plugins available within the Jenkins interface. I, however, prefer just executing Shell Scripts, since you will have more control over what you are doing via bash scripting. If you are a bash scripting novice, you can Click Here to view a nice guide.

When scripting, realize that every change you make must be changed in your other scripts. For example, if I use the change filename script below, obviously I will need to modify my upload script to reflect the changed filename. I think of it in a way that "one thing breaks another". This is why its great to have an independent script for every process you do, so that only one file requires modification if changes are made.

.... B. How to set your scripts to run after the initial compile process has completed...

In order to run ANY OTHER PROCESS after the initial build script is complete, you must run another script after build. To do this, you will use the "Add build step" feature to "Execute Shell", located in the "Build" section. As such, for ANY new script you want to run after the build script, you must add another build step. Consequently, every single script below requires a new build script.

Screenshot:


Like above when you setup your initial build script, you will call a script out. This can be done the same way, using:
Code:
#!/bin/bash
cd ~/FOLDER/
source SCRIPT_NAME.sh
Screenshot:



Continued in Post #4
The Following 10 Users Say Thank You to For This Useful Post: [ View ] Gift Ad-Free
AndroidSlave
2nd October 2013, 05:45 PM |#4  
Guest
Thanks Meter: 0
 
More
IX. Specific post-build scripts...

.... A. Automating Filename Changes, and/or Moving The Build to a Different Directory Automatically...

This essentially moves the build .zip to the same folder, but renames it to the name you choose. Thus, when it uploads, you don't have to manually rename the ROM Firmware file.

Code:
#!/bin/bash
cd /var/lib/jenkins/FOLDER/out/target/product/DEVICE/
mv cm-10*.zip BUILD-NAME.zip
If you want to add the date to the filename, you can add the following string within the filename:

Code:
$(date +%Y%m%d)
You are free to modify the ordering of %Y%m%d as you see fit.

Therefore, if I wanted my build to say "SGT7-DATE-n7000-NIGHTLY.zip", I would use the following as the filename:

Code:
SGT7-$(date +%Y%m%d)-n7000-NIGHTLY.zip
As such, you're final code would be:

Code:
#!/bin/bash
cd /var/lib/jenkins/FOLDER/out/target/product/DEVICE/
mv cm-10*.zip BUILD-NAME-$(date +%Y%m%d).zip
Alternatively, if you want to move to a local directory, such as your public ftp folder, you can use the following code instead, adding the date string above if you want:

Code:
#!/bin/bash
cd /var/lib/jenkins/FOLDER/out/target/product/DEVICE/
mv cm-10*.zip /PATH_TO_PUB_FTP_FOLDER/BUILD-NAME.zip

.... B. Automating uploads to "goo.im" or any other FTP/SFTP server...

First, you will have to install yafc in ubuntu. This is done via "sudo apt-get install yafc".

Then, create a script using the following code:
Code:
#!/bin/bash
cd /var/lib/jenkins/FOLDER/out/target/product/DEVICE/
yafc  <<**
open ftp://USER:PASS@upload.goo.im:21/ 
cd public_html/FOLDER
put -f -r /var/lib/jenkins/FOLDER/out/target/product/DEVICE/cm-10*.zip
close
.... C. Automating build.prop modifications...

MARKER

.... D. Automating changelogs...

MARKER

Continued in Post #5
The Following 12 Users Say Thank You to For This Useful Post: [ View ] Gift Ad-Free
AndroidSlave
2nd October 2013, 05:45 PM |#5  
Guest
Thanks Meter: 0
 
More
X. Troubleshooting...

Here are some common solutions to your issues.
  • Reboot your system ("reboot -n") and try again. You may need a daily, or even weekly reboot depending on your RAM, software causing memory leaks, etc. You can setup a CRONJOB (Click Here for more info) to do a reboot daily. If you choose to setup a job in the crontab, be ensure it doesn't conflict with the timing of your Jenkins cron builds.
  • If you keep getting java out of memory errors, you probably need more RAM.
  • Other errors may be caused by your CCACHE being set to low. If you continue having issues caused by ccache, even when its high, you may want to install a global ccache in /usr/bin or the like.
  • Did you remember to set permissions on your scripts ("chmod a+x")
  • need to reset your username? See this post: http://forum.xda-developers.com/show...php?p=51248853
  • having ota jenkins.zip NOT FOUND ERROR (or no such file or directory)? See this post: http://forum.xda-developers.com/show...php?p=48454597; see also this post: http://forum.xda-developers.com/show...php?p=55954963
The Following 9 Users Say Thank You to For This Useful Post: [ View ] Gift Ad-Free
2nd October 2013, 07:48 PM |#6  
MikeCriggs's Avatar
Recognized Developer / Contributor
Flag Catskills, NY
Thanks Meter: 5,999
 
Donate to Me
More
Subscribed!

Sent from my Nexus 4 using Tapatalk 4
The Following User Says Thank You to MikeCriggs For This Useful Post: [ View ]
AndroidSlave
2nd October 2013, 09:34 PM |#7  
Guest
Thanks Meter: 0
 
More
Quote:
Originally Posted by MikeyCriggz

Subscribed!

Sent from my Nexus 4 using Tapatalk 4

I have lots of great scripts and stuff I will be posting! just have to juggle my real job with throwing in some time every day to build this up...
The Following 3 Users Say Thank You to For This Useful Post: [ View ] Gift Ad-Free
2nd October 2013, 10:34 PM |#8  
chasmodo's Avatar
Senior Member
Flag Novi Sad
Thanks Meter: 41,311
 
More
Well done, mate. This is something that hasn't been covered so far methinks.
The Following 3 Users Say Thank You to chasmodo For This Useful Post: [ View ] Gift chasmodo Ad-Free
AndroidSlave
3rd October 2013, 12:37 AM |#9  
Guest
Thanks Meter: 0
 
More
Quote:
Originally Posted by chasmodo

Well done, mate. This is something that hasn't been covered so far methinks.

Thanks buddy. I wanted to PM you a question, but you don't accept PMs...
I'll G+ you later on.
3rd October 2013, 09:51 AM |#10  
Galaxtus's Avatar
Senior Member
-82.540604,29.472649
Thanks Meter: 2,324
 
More
I have become your slave
Very good

#KitKat - Having one right now.
3rd October 2013, 09:57 AM |#11  
sunny2303's Avatar
Senior Member
Flag Mumbai
Thanks Meter: 2,274
 
Donate to Me
More
This is what you call, "Taking it to next level"

:thumbup:

Sent from Asylum HQ
The Following User Says Thank You to sunny2303 For This Useful Post: [ View ] Gift sunny2303 Ad-Free
Thread Closed Subscribe to Thread
Previous Thread Next Thread
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes