[DEV] Building a custom kernel and kernel modules for stock kernel

Search This thread

madushan1000

Senior Member
Feb 25, 2013
84
62
Ratnapura
Since fire phone doesn't have a bootloader unlock at the moment. There is no point in building a custom kernel. But By building a kernel we can build kernel modules which work on the stock kernel. And yes you can load unsigned kernel modules without a problem since fire phone doesn't use tz apps to verify kernel modules like Samsung does.

Setup

Source
Download the fire phone sources for firmware 4.6.1 from here. And extract the platfrom.tar inside the archive to somewhere(KERNEL_DIR).

toolchain
You can use the android ndk from google, But it requires some setup. I'm using linaro toolchain from here. You can use compiler version 4.7, 4.8 or 4.9. Kernel I'm using (Firmware 4.6.3 - Linux 3.4-perf-g280c96c) is built with gcc-4.7. But I'm using this gcc-4.9. Download it, extract is somewhere(TOOLCHAIN_DIR) and add the $TOOLCHAIN_DIR/bin to your PATH. Theoretically you would be able to build the kernel on windows using Cygwin or MSYS tools but using Linux is better.

config
Connect your phone trough adb and run
Code:
adb pull /proc/config.gz
zcat config.gz > $KERNEL_DIR/kernel/qcom/3.4/.config
With this config you will run into some problems because of a missing "trapz_generated_kernel.h". I don't know if this is an auto generated file when they build android as a whole or amazon removed this explicitly(can they do that without violating GPL?). Anyway It looks trapz is some low level kernel debugging function(comment here if you know more about it). We can safely disable it. Open $KERNEL_DIR/kernel/qcom/3.4/.config in a text editor and change the lines
Code:
CONFIG_TRAPZ=y
CONFIG_TRAPZ_TP=y
CONFIG_TRAPZ_TRIGGER=y
CONFIG_HAVOK=y
to
Code:
#CONFIG_TRAPZ=y
#CONFIG_TRAPZ_TP=y
#CONFIG_TRAPZ_TRIGGER=y
#CONFIG_HAVOK=y

building
Now edit the $KERNEL_DIR/kernel/qcom/3.4/Makefile and add this changes

Code:
EXTRAVERSION = -perf-g280c96c
This is at the top of the makefile. If we don't add this, vermagic for the modules will differ from stock kernel and they won't load.

ARCH=arm
CROSS_COMPILE=arm-linux-gnueabihf-
Here arm-linux-gnueabihf- is my cross compiler frefix. Look in $TOOLCHAIN_DIR/bin/ to find it.

Now cd into $KERNEL_DIR/kernel/qcom/3.4/ and do
Code:
make

The build will fail a few times complaining about missing headers. Most of the time it's just
Code:
#include <myheader.h>
instead of
Code:
#include "myheader.h"
Edit the source file where the build fails and change <>s to ""s. (maybe android ndk ignores the difference and include the headers anyway)

After kernel compiles, we are good to go. We can use this kernel sources to build kernel modules for stock kernel.

Kernel modules
To build the kernel modules, we basically need two things. An approximate kernel source and the Module.symvers file from the original kernel. We can get the Module.symvers file by building the complete kernel as explained above or Just extract it from our stock kernel.
To extract the Module.symvers from the stock kernel, extract the boot.img file from firmware update image. Get mkbootimg tools from here compile it and run
Code:
unmkbootimg --kernel zImage ---ramdisk ramdisk.cpio.gz -i boot.img

After you get the zImage. Download extract-symvers script from here and run
Code:
python2 extract-symvers.py -B 0xc0008000 zImage > Module.symvers
place this file in $KERNEL_DIR/kernel/qcom/3.4/ (You still have to do the changes mentioned above in kernel config and building section run make in the $KERNEL_DIR/kernel/qcom/3.4 and intrupt it after few seconds)

Now you can build loadable modules against this source. Here is a hello world kernel module.
Code:
//hello.c
#include<linux/module.h>
#include<linux/kernel.h>
#include<linux/init.h>

static int __init hello_start(void)
{
  printk("hello to the world from module");
  return 0;
}

static void __exit hello_end(void)
{
  printk("heloo exit");
}

module_init(hello_start);
module_exit(hello_end);
Code:
#Makefile
KERNEL_DIR=<your kernel dir>/kernel/qcom/3.4

obj-m := hello.o
PWD := $(shell pwd)

default:
	$(MAKE) ARCH=arm CROSS_COMPILE=armeb-linux-gnueabi- -C $(KERNEL_DIR) SUBDIRS=$(PWD) modules

Put this files in a folder and run make in it. Change the paths and cross compiler prefix according to your setup. and run make.
After the build push the hello.ko to the phone.
Code:
adb push hello.ko /sdcard/
adb shell
su
cd sdcard
insmod hello.ko
run dmesg and you'll see the message.

I'm currently trying to build kexec module from hashcode's sources and USB OTG modules.

I'm attaching a few thing helped me do this.
 

Attachments

  • config_module_symvers_makefile.zip
    137.4 KB · Views: 401
  • hello.zip
    10.6 KB · Views: 198
  • config.gz
    22.5 KB · Views: 155
Last edited:

veti

Senior Member
May 12, 2015
105
47
since they have released this version of the fire os they have to provide the source code
see
http://www.gnu.org/licenses/gpl-faq.html#GPLRequireSourcePostedPublic

you have just shown that the source code they releases for the kernel does not match the one used to build the kernel. This means it is a clear violation of the gpl and amazon is in breach and can be sued.

on another note.
are the drivers for the nfc and camera compiled as a module or into the kernel?
 

madushan1000

Senior Member
Feb 25, 2013
84
62
Ratnapura
They have yet to provide 4.6.3 and 4.6.4 kernel sources too.

I don't know exactly but in order for NFC and camera to work drivers are required and they are in fact compiled into the kernel.
The problem we currently have with NFC and camera is proprietary hal (hardware abstraction libraries) They are a part of Android and does not subject to GPL. Amazon changed the original android way how hal works and didn't release the sources!

by looking at the kernel drivers maybe we would be able to implement hal from scratch. But I don't see that intense dev support for fire phone. If you are up for it camera sources are at $KERNEL_DIR/kernel/qcom/3.4/drivers/media/platform/msm/camera_v2/
 
Last edited:
  • Like
Reactions: Moogagot
O

Okarin

Guest
Could someone please provide the config extracted from /proc/config.gz?
I can't find this on CM11 rom for some reason.
 
O

Okarin

Guest
Building the kernel now.

Some bugs are in the code and -Wall and gcc-wrapper.py escalate the warnings.
I wonder if those errors are there on purpose XD

helloworld.ko loaded successfully :)

I was able to execute kexec without anything. Just the binary.

Will keep you posted - this hacking might take a while to figure it all out.

I already have 3.4 kernel from the amazon sources.
I have the kexec userland program.
What is left is a loadable kexec kernel module (if that is possible at all).
 
Last edited:

madushan1000

Senior Member
Feb 25, 2013
84
62
Ratnapura
Are we even sure those Amazon Kernel Sources are correct?
Those errors caught by the wrapper scripts are giving me the creeps.

Git the kexec_load.ko build.

Currently hands on insmod.

Phone doesn't do a reboot any longer:
insmod kexec_load.ko

init_module(0xb6e6c008, 408241, "") = -1 ENOENT (No such file or directory)
write(2, "insmod: init_module '/sdcard/kex"..., 79insmod: init_module '/sdcard/kexec_load.ko' failed (No such file or directory)
) = 79
munmap(0xb6e6c000, 409600) = 0
mprotect(0xb6f8c000, 4096, PROT_READ|PROT_WRITE) = 0
mprotect(0xb6f8c000, 4096, PROT_READ) = 0
close(0) = 0
close(1) = 0
close(2) = 0
futex(0xb6f6cd74, FUTEX_WAKE_PRIVATE, 2147483647) = 0
munmap(0xb6f8c000, 4096) = 0
exit_group(-1) = ?

First goal is to get module loaded.

Goal reached:
kexec_load 27813 0 - Live 0x00000000 (O)
procfs_rw 12770 0 - Live 0x00000000 (O)
wlan 3793980 0 - Live 0x00000000 (O)

Shouldn't be functional at all..

I disabled some function calls just to get the module loaded.

The missing symbols are:
soft_restart
arch_kexec
machine_shutdown
And the version I use does some insane function hooking ..
More rework is needed.

root@kodiak:/data/local # ./kexec /sdcard/vmlinux
kernel: 0xaf12d008 kernel_size: 7e1354c
unrecoverable error: could not scan "/proc/device-tree/": No such file or directory

<6>[ 97.681256] Kexec_load: Replacement... :
<6>[ 97.681344] kexec_load : my_syscall_table : c0106244
<6>[ 97.681405] kexec_load : kexec_load before replacement : c01b346c
<6>[ 97.681480] kexec_load : kexec_load after replacement : bf3a5650
<6>[ 97.681546] kexec_load : reboot before replacement : c01a83f0
<6>[ 97.681616] kexec_load : reboot after replacement : bf3a6348
<6>[ 97.681675] Kexec_load: End replacement... :
<6>[ 202.694691] Kexec: - Starting kexec_load...
<6>[ 202.694849] Kexec: - ---- kexec_load - result : '0'

It gets better:
255|root@kodiak:/data/local # ./kexec --dtb=/sdcard/zImage-dtb /sdcard/vmlinux
kernel: 0xaf1b1008 kernel_size: 7e1354c
kexec-zImage-arm : dtb.img BEFORE CUT : Start : '0xae66f008' - Length : '0xb411e9' - End : '0xaf1b01f1'
Segmentation fault

More tomorrow.
Where are you getting your kexec module sources from? BTW try using the original amazon kernal binary the phone is shipped with (we are sure it works). Don't use the custom kernel for the kexec tests (We don't know the custom kernel actually works)
 
O

Okarin

Guest
Where are you getting your kexec module sources from? BTW try using the original amazon kernal binary the phone is shipped with (we are sure it works). Don't use the custom kernel for the kexec tests (We don't know the custom kernel actually works)

Here is the thread I used as a starting point.
I will put up my "fork" on github after I get permission to do that :)

The userland part build like a charm once I took the compiler you recommended.
The kernel-module was tricky because the whole thing is modded like hell.
To be able to load I had to comment out some hard coded addresses and the calls to unresolvable symbols.
 
Last edited:
O

Okarin

Guest
Okay the kernel gets loaded.
But
kexec -e shuts off the device.
strace doesn't help.

On the plus side:
Devices are shutdown
Look promising

I need a way to tail dmesg ...

Okay a lot of digging around and I found out that the reboot syscall doesn't work properly..

It doesn't look like it hits the kexec_module it looks more like it hits the actualy sys_reboot

Okay reboot syscall hits my reboot-hook.
But the softreboot doesn't work now.


Okay there is some kind of watchdog runnig which doesn't like my kexec.
I need to kill it - that should happen tomorrow.
 
Last edited:

madushan1000

Senior Member
Feb 25, 2013
84
62
Ratnapura
I hit the same wall when I tried to isolate the kexec code from the kernel itself to a module. I stopped working on it because I lacked the time. BTW the error you are facing now
<3>[ 80.580644] BUG: scheduling while atomic: kexec/4067/0x00000002
is because memory allocator is trying to switch threads while you are in a syscall. it's because of lines like this
image = kzalloc(sizeof(*image), GFP_KERNEL);
Try changing GFP_KERNEL to GFP_ATOMIC. Other than that, I have another suggestion. Try to get the kernel to run in a single core mode before running kexec code. This might simplify things. I don't know how to do this though.
 
O

Okarin

Guest
I hit the same wall when I tried to isolate the kexec code from the kernel itself to a module. I stopped working on it because I lacked the time. BTW the error you are facing now

is because memory allocator is trying to switch threads while you are in a syscall. it's because of lines like this

Try changing GFP_KERNEL to GFP_ATOMIC. Other than that, I have another suggestion. Try to get the kernel to run in a single core mode before running kexec code. This might simplify things. I don't know how to do this though.

The atmic error is gone now. It went away after I disabled the watchtog.

smp_disable() is what you are looking for - but this causes the system to hard_reboot ATM XD
 
O

Okarin

Guest
#define tomorrow

Okay .. I worked out the preemption thing.
At least it does something.

Still a black screen and the MSM_WATCHDOG is a ***** again.
It needs to be suspended .. at least that what I get from the code I read here.
If I remove the driver too early the output in /proc/kmsg stops ..
If I try to remove it too late ... well it causes a resched while atomic.
 

Top Liked Posts

  • There are no posts matching your filters.
  • 16
    Since fire phone doesn't have a bootloader unlock at the moment. There is no point in building a custom kernel. But By building a kernel we can build kernel modules which work on the stock kernel. And yes you can load unsigned kernel modules without a problem since fire phone doesn't use tz apps to verify kernel modules like Samsung does.

    Setup

    Source
    Download the fire phone sources for firmware 4.6.1 from here. And extract the platfrom.tar inside the archive to somewhere(KERNEL_DIR).

    toolchain
    You can use the android ndk from google, But it requires some setup. I'm using linaro toolchain from here. You can use compiler version 4.7, 4.8 or 4.9. Kernel I'm using (Firmware 4.6.3 - Linux 3.4-perf-g280c96c) is built with gcc-4.7. But I'm using this gcc-4.9. Download it, extract is somewhere(TOOLCHAIN_DIR) and add the $TOOLCHAIN_DIR/bin to your PATH. Theoretically you would be able to build the kernel on windows using Cygwin or MSYS tools but using Linux is better.

    config
    Connect your phone trough adb and run
    Code:
    adb pull /proc/config.gz
    zcat config.gz > $KERNEL_DIR/kernel/qcom/3.4/.config
    With this config you will run into some problems because of a missing "trapz_generated_kernel.h". I don't know if this is an auto generated file when they build android as a whole or amazon removed this explicitly(can they do that without violating GPL?). Anyway It looks trapz is some low level kernel debugging function(comment here if you know more about it). We can safely disable it. Open $KERNEL_DIR/kernel/qcom/3.4/.config in a text editor and change the lines
    Code:
    CONFIG_TRAPZ=y
    CONFIG_TRAPZ_TP=y
    CONFIG_TRAPZ_TRIGGER=y
    CONFIG_HAVOK=y
    to
    Code:
    #CONFIG_TRAPZ=y
    #CONFIG_TRAPZ_TP=y
    #CONFIG_TRAPZ_TRIGGER=y
    #CONFIG_HAVOK=y

    building
    Now edit the $KERNEL_DIR/kernel/qcom/3.4/Makefile and add this changes

    Code:
    EXTRAVERSION = -perf-g280c96c
    This is at the top of the makefile. If we don't add this, vermagic for the modules will differ from stock kernel and they won't load.

    ARCH=arm
    CROSS_COMPILE=arm-linux-gnueabihf-
    Here arm-linux-gnueabihf- is my cross compiler frefix. Look in $TOOLCHAIN_DIR/bin/ to find it.

    Now cd into $KERNEL_DIR/kernel/qcom/3.4/ and do
    Code:
    make

    The build will fail a few times complaining about missing headers. Most of the time it's just
    Code:
    #include <myheader.h>
    instead of
    Code:
    #include "myheader.h"
    Edit the source file where the build fails and change <>s to ""s. (maybe android ndk ignores the difference and include the headers anyway)

    After kernel compiles, we are good to go. We can use this kernel sources to build kernel modules for stock kernel.

    Kernel modules
    To build the kernel modules, we basically need two things. An approximate kernel source and the Module.symvers file from the original kernel. We can get the Module.symvers file by building the complete kernel as explained above or Just extract it from our stock kernel.
    To extract the Module.symvers from the stock kernel, extract the boot.img file from firmware update image. Get mkbootimg tools from here compile it and run
    Code:
    unmkbootimg --kernel zImage ---ramdisk ramdisk.cpio.gz -i boot.img

    After you get the zImage. Download extract-symvers script from here and run
    Code:
    python2 extract-symvers.py -B 0xc0008000 zImage > Module.symvers
    place this file in $KERNEL_DIR/kernel/qcom/3.4/ (You still have to do the changes mentioned above in kernel config and building section run make in the $KERNEL_DIR/kernel/qcom/3.4 and intrupt it after few seconds)

    Now you can build loadable modules against this source. Here is a hello world kernel module.
    Code:
    //hello.c
    #include<linux/module.h>
    #include<linux/kernel.h>
    #include<linux/init.h>
    
    static int __init hello_start(void)
    {
      printk("hello to the world from module");
      return 0;
    }
    
    static void __exit hello_end(void)
    {
      printk("heloo exit");
    }
    
    module_init(hello_start);
    module_exit(hello_end);
    Code:
    #Makefile
    KERNEL_DIR=<your kernel dir>/kernel/qcom/3.4
    
    obj-m := hello.o
    PWD := $(shell pwd)
    
    default:
    	$(MAKE) ARCH=arm CROSS_COMPILE=armeb-linux-gnueabi- -C $(KERNEL_DIR) SUBDIRS=$(PWD) modules

    Put this files in a folder and run make in it. Change the paths and cross compiler prefix according to your setup. and run make.
    After the build push the hello.ko to the phone.
    Code:
    adb push hello.ko /sdcard/
    adb shell
    su
    cd sdcard
    insmod hello.ko
    run dmesg and you'll see the message.

    I'm currently trying to build kexec module from hashcode's sources and USB OTG modules.

    I'm attaching a few thing helped me do this.
    3
    So u guys are giving up on kexec? If so, I don't expect you to answer. @Okarin What is with this code in kexec.c line 1457 and 1458-- "kernel_restart_prepare(NULL); machine_shutdown();" ? A stub to make it compile (line numbers might be wrong but should be close)? We aren't using kernel_restart_prepare anymore? Or machine_shutdown? They sound important....When I uncomment reboot.h in kexec.c #includes in @madushan1000 build I get unknown symbol errors. I know how to work around this but it would be great to unify the code base for (now obsolete) kexec_load.ko. @delewer @Crabapple @Surge1223 @calcprogrammer and that ryan guy who talks to surge (and others who I'd like to mention--can you post additional code). @Surge1223 you only built in kernel? No external module? I have modified Madushan1000's kexec.c so that the commented out includes buld (as in yours Okarin) on my gcc 4.4.3 (crusty, I know, don't ask). More code to look at would be great. For the exercise more than eventual success, I have to believe.... c_p

    Nah I prefer to build out of tree, makes it easier to modify and add code/files and see the results. I just use local copies of some includes, like kexec.h, etc. When I was working on this, I used a makefile similar to this

    Code:
    EXTRA_CFLAGS=-mno-unaligned-access
    EXTRA_CFLAGS=-fno-pic
    KDIR := /Users/surge/Downloads/kernel_samsung-jfltevzw-VRUGOC1-master
    
    KVER := $(shell uname -r)
    
    PWD := $(shell pwd)
    
    CCPATH := /Users/surge/arm-eabi-4.8/bin
    ARCH		= arm
    CPPFLAGS	=  -I$kernel/
    EXTRA_CFLAGS += -I${KDIR}/include
    
    obj-m += lkm_kexec.o
    lkm_kexec-objs := kexec.o mmu.o sys.o core.o  sleep.o \
    	            proc-v7.o tlb-v7.o cache-v7.o pabort-v7.o abort-ev7.o copypage-v6.o \
    	            mm_init.o entry-common.o driver_sys.o suspend.o idmap.o 
    	    
    obj-m += msm_kexec.o
    
    obj-m += proc_atags.o
    
    obj-m += atag.o
    atag-objs := atags_compat.o atags.o
    
    obj-m += arm_kexec.o
    arm_kexec-objs := machine_kexec.o relocate_kernel.o
    
    default:
    	$(MAKE) ARCH=arm CROSS_COMPILE=$(CCPATH)/arm-eabi- EXTRA_CFLAGS="-I${KDIR}/include/linux -fno-pic -mno-unaligned-access" -C $(KDIR) M=$(PWD) modules
    	cp -f `find $(PWD) -name "*.ko"` $(KDIR)/build/
    
    clean:
    	rm -f *.o.d *.o *.ko *.mod.c Module.symvers Module.markers modules.order \.*.o.cmd 	\.*.ko.cmd \.*.o.drm -rf \.tmp_versions

    lkm_kexec.ko ends up being pretty large (~800kb)

    in order for me to test kexec using these modules, I have to load them in the following order:

    insmod msm_kexec.ko
    insmod lkm_kexec.ko
    insmod proc_atags.ko
    insmod arm_kexec.ko

    usually ends up getting to soft_restart and freezes when switching to the static identity mapping to continue with reset.

    EDIT: if you're wondering why I dont just combine msm_kexec.ko, lkm_kexec.ko and arm_kexec.ko, its because I intentionally want kexec.c and machine_kexec.c code to be housed in different modules, also for some testing, I want the driver for char device registration to occur without initializing some of the machine_kexec and relocation info. A side benefit of this is that it also becomes very easy to see if the failure is kexec.c (and by extension, kexec-tools) related, or arch specific kexec implementation errors (machine_kexec, relocate_kernel, atags, etc)
    1
    They have yet to provide 4.6.3 and 4.6.4 kernel sources too.

    I don't know exactly but in order for NFC and camera to work drivers are required and they are in fact compiled into the kernel.
    The problem we currently have with NFC and camera is proprietary hal (hardware abstraction libraries) They are a part of Android and does not subject to GPL. Amazon changed the original android way how hal works and didn't release the sources!

    by looking at the kernel drivers maybe we would be able to implement hal from scratch. But I don't see that intense dev support for fire phone. If you are up for it camera sources are at $KERNEL_DIR/kernel/qcom/3.4/drivers/media/platform/msm/camera_v2/
    1
    Major MAJOR respect for all of you making the Fire Phone even better!
    1
    Yeah that's what I want to try.
    After fooling around with kexec I need to find the correct arguments for the kernel registers.
    A machine_number is required and currently this is 0xFFFFFFFF for some reason :p

    Will see what I can do over the weekend.

    I need to update the kexec version I am using .. this will be fun :D
    I should have bookmarked that thread about the trouble with older versions and wrong kernel support ...

    And of course .. the stuff I started from hacked around multiple things too...
    No I read the dtb file and what do I find ... it's not valid ... oh yeah

    On further inspection there is a little header before the realy device tree .. damn it XD need to strip that off
    Got rid of that tiny header -- whatever it was ...
    Now we have a kexec-2.0.11 running in userland and a more or less hacky kexec-module
    This is my kexec mod sources before I stopped working on it
    https://github.com/madushan1000/kexec-mad
    I have no idea which kexec-tools version I used because I have about 10 repos cluttering on my laptop :p (It's probably the one that uses a /dev/ driver to communicate instead of the syscall)