[DEV] Kernel development HOWTO and Interactive menu

I havent yet found a simple guide for compiling kernels. Some of them assume too much, and some are just outdated. So I thought I'd write my own for devs/budding devs. Here you go!

Note:
This is not a guide for newbies. It's a dev guide for devs.
Research before asking questions, please

For The Menu driven interactive kernel build script, see Post #31


I will be developing this guide as I go, so it will be incomplete initially, or lacking in detailed explanations.

Essentials:

1. Ubuntu Box (By this I mean a PC with a Ubuntu installation, not a live CD)
2. A toolchain-Either the Android NDK, or your own toolchain
3. HTC Desire GB/Froyo source from htcdevs.com, or sources from github
4. Familiarity with the linux shell and basic linux commands.
5. The will to learn

First things first,

1. Getting the sources


The HTC Desire source is available from two kinds of resources-you can either get it from htcdevs.com (official HTC Dev site), or from source code uploaded from someone else. For the purpose of this tutorial, I'll assume we're working on the official HTC GB source code. So download bravo_2.6.35_gb-mr.tar.gz from htcdevs.com.


2. Setting up the compilation box and preparing source code

2.1 Install some essential linux packages from the Linux terminal:



2.2 Extract the source code

The file you downloaded is a tar archive (like a zip file), so you need to extract it to a convenient location. Let's hit the linux shell-open a terminal window in linux (Accessories->Terminal)

Type:
Let's go to our home directory:
Now, create the directories for our kernel compilation box.

Now you need to copy the tar.gz file from wherever you downloaded it to, to this dir.

Extract the archive:
Now we can view the extracted files within the directory ~/android/kernel/bravo_2.6.35_gb-mr/

2.3 Set up the toolchain

A toolchain is a set of programs which allow you to compile source code (any source code, not just kernels). The toolchain is specific for the processor and hardware, so we need a toolchain specific for Android and especially the Desire. If you're a semiadvanced-pro user, you may consider compiling your own toolchain (See theGanymedes' guide for doing so). If compilation of kernels is all that you require, fortunately for you, there is an easy way-the Android NDK - v7 (latest as of now) is available here

Get the NDK for Linux - android-ndk-r7-linux-x86.tar.bz2


Now copy the NDK file to ~/android/ndk

Whenever I say copy, you have to manually copy the file with any file manager. Nautilus comes with Ubuntu, and Dolphin with Kubuntu. You may also use the shell of course with

Extract it:

Now add the path for your toolchain to the env variable:

At the end of the file, add this line:


3. Setting up kernel parameters

Kernels are compiled with a program called gnu make, and use a set of configuration options specified within a file called Makefile.

A vital point to note is that kernels are compiled with a program called gcc (basically the gnu C compiler), and our NDK itself has its own optimized version of gcc. While compiling, we're actually cross compiling it (meaning compiling a binary package on a system which is different from the actual system which is meant to run it- you're compiling it on your PC while it's actually meant to run on your Desire)

This means that when you compile it, you have to make sure that you compile it with the NDK's version of gcc instead of the system version. Otherwise you end up with a kernel meant to run on your pc, duh! Specifying which gcc to use is by the CROSS_COMPILE variable. You can set it up with this command:
Note the hyphen (-) at the end, and do not forget to include it! At compilation time, system will actually use this variable to find all the programs it needs. Eg: The path for gcc will become arm-linux-androideabi-gcc

We can compile kernels with many different options, like with ext4 support, or without; ext4 support as part of the kernel zImage (in which case it makes the kernel larger), or as a loadable module (of the form somename.ko, which is loaded at init.d/init.rc with the command insmod modulename.ko)

We specify the exact options we require with the help of a useful configuration program called menuconfig (which as the name suggests, is a menu for configuration of make options).

An important thing to note is that as far as kernel compilation is concerned, there are a vast amount of options to setup, and unless you're thorough with kernel compilation, you wont be able to set up the options correctly and get your kernel to boot. Fortunately for us, the kernel source already comes with a default set of parameters which can be easily set up.

Note that all make commands must be executed within the directory bravo_2.6.35_gb-mr. Let's go there now:

This produces a .config file (used by the menuconfig) containing essential parameters to produce a booting kernel for the Desire.

Note: There is a simpler way to get the basic .config file, and this is to get it from a running kernel built by someone else. You can extract the .config from a running kernel with these commands:

Now we can open menuconfig and add anything we need in addition.

You can view the huge amount of options available in menuconfig.

You can add ext4 support for example (See image above)
Once you're done choosing options, you can exit menuconfig.

4. Compiling it

This is simple. The basic command is:

make ARCH=arm CROSS_COMPILE=arm-linux-androideabi- -j10

The -j10 specifies the number of jobs to execute per operation. I can usually go upto 50 on my Quad core CPU. Beware, this can bring a slow CPU to a crawl and freeze up linux itself.

During compilation, you will see all sorts of messages, which may include warnings too. In most cases, its safe to ignore warnings. If there are errors, the compilation will stop, and you will have to fix the issues.

5. Distributing your kernel to users
At the end of compilation, it generates files named zImage, and various .ko files.

You have to copy them from their default location to a zip file. The best way is to use my variant of koush's Anykernel, and copy the files to it. Then, you can zip the whole folder and lo and behold-you have your flashable kernel zip which you can distribute to others.

You can also remove the zImage and the modules from /system/lib/modules of any kernel zip available with you, and copy over your files to it, at the correct location.

So, let's say that you have extracted an existing kernel zip to the location ~/flashable

The file structure should be like this:

I've included my flashable zip directory along with this post. Download file kernel_flashable.tar.bz2.zip to ~/

This will create the directory structure outlined above.

Now after every compilation of the kernel, execute these commands from where you executed make:

This will create mykernel.zip at ~/kernel_flashable. You can distribute this to your users to flash. Make sure you edit updater-script before though

Common errors and other stuff

Ok, post #1 was simple stuff. Now, supposing you get errors while compiling. Post #2 is about that, and ups the level of knowledge a bit..

Some kernel compilation errors:

Treat warnings as errors-Solved by removing the string "-Werror" from all Makefiles of the file which failed to compile. Some people had said that the real error (Array out of bounds warning) was because of gcc optimizations. But putting -O2 to -O0 didnt do a thing.
No of jobs - ought not to exceed 50.
"warning: variable set but not used [-Wunused-but-set-variable]"-Look at KBUILD_CFLAGS in the main Makefile. Add -Wno-error=unused-but-set-variable to the existing set of flags.


Note the following from gcc manual:


-WerrorMake all warnings into hard errors. Source code which triggers warnings will be rejected.
-w Inhibit all warning messages. If you're familiar with C code and like to fix stuff, rather than ignoring potential bugs, use this only as a last resort- A 'brahmastram' (most powerful weapon in your time of gravest need) as the epics would say
-WerrorMake all warnings into errors.
-Werror=Make the specified warning into an error. The specifier for a warning is appended, for example -Werror=switch turns the warnings controlled by -Wswitch into errors. This switch takes a negative form, to be used to negate -Werror for specific warnings, for example -Wno-error=switch makes -Wswitch warnings not be errors, even when -Werror is in effect. You can use the -fdiagnostics-show-option option to have each controllable warning amended with the option which controls it, to determine what to use with this option.

So what I did to suppress errors was to add:


Though the -Wunused-but-set-variable is not a real issue in itself, it generates so much "noise" that you may miss actual make errors.

This is the error what I was talking about..


Solution:
Edit drivers/net/wireless/bcm4329_204/Makefile

Locate -Werror within DHDCFLAGS, and delete it.



This will prevent gcc from treating mere warnings as errors.

How to modify kernels by applying mods - Applying Kernel Patches

Ok, you have compiled a simple stock kernel. Now what? Would you like to add fixes/mods developed by other kernel devs? This post explains patches and how exactly to do this.

Patches to the kernel are applied via patch files. Patch files are simple text files generated by the linux diff program which takes two text files, compares them and writes the differences (hence called diff) to another text file which by convention has the extension .patch

Attached to this post is a patch containing my "Extended battery" fix with Sibere's battfix. I'll explain patching with this. Let's understand the patch file. Open it up in any text editor.

Note the first line:
diff -rupN basically describes the command that was used to generate this patch. The -u means that the patch file is something called a universal patch

bravo_2.6.35_gb-mr/drivers/power/ds2784_battery.c was the original file, and bravo_2.6.35_gb-mr.main//drivers/power/ds2784_battery.c was the target file or file which contains the mod..

How to apply patch files?

The command depends on where your current directory is. If you're in ~/android/kernel/bravo_2.6.35_gb-mr/ and your current directory contains the directory 'drivers', you can apply this patch with this command:


If you're within drivers, then you have to modify the command like this:


Hope you get the gist. Basically, as you move into the source tree, you have to increment the patch level by the number of directories you've moved down into. Very simple, isnt it?

Sharing and Collaborating - Using Github and Commits

Kernel compilation is a group effort (at least it ought to be). When different devs work on different parts of the code and create their own mods, development progresses. For this purpose, it is important that you share your code with other devs. The best way to do this to upload your sources to github.

First, create a github account.

Next you can view other devs' github sources and examine their commits. Commits are basically patches applies to the previous source uploaded. Github commits use the universal patch format and can be viewed directly, downloaded as patch files, and applied to your code. You can also choose to download the whole source tree uploaded by another dev and examine it.

Ok, the basic guide is done, guys... If you have doubts, I'll try to clear them