[DEV] Building a custom kernel and kernel modules for stock kernel

Search This thread

Surge1223

Recognized Contributor
Nov 6, 2012
2,622
7,466
Florida
Google Pixel 6 Pro
@Surge1223 Thanks but I think I'm missing one or more of your files (that is why I asked about out-of-tree builds). I've got all the stuff you included with this kernel from github: kernel_samsung-jfltevzw-VRUFNC5-master. I believe I would need (at a minumum) atags_compat.c. I guess I could just grab one from some (roughly) compatible source tree but I'd like to replicate your work as much as possible. After all, who wouldn't want 4 modules instead of just 2 :)
Oh I haven't updated those sources in forever, I'm not even sure I have uploaded any of the out of tree source anywhere yet, and they are significantly different from what's on that git. I'll upload them to git in a bit.

Also I don't usually load the atag.o module, because I coded it in an attempt to make it dump atags on the next reboot, but currently it panics so I never attempted to fix it and I instead opted to have someone on an unlocked bootloader flash a kernel i made that dumps atags out and had them send it to me.

The proc_atags.ko just creates /proc/atags entry for convenience as that's how kexec-tools checks for atags

Sent from my Nexus 6P using XDA-Developers mobile app
 
Last edited:
  • Like
Reactions: capt_planit

capt_planit

Senior Member
Apr 6, 2016
63
17
Oh I haven't updated those sources in forever, I'm not even sure I have uploaded any of the out of tree source anywhere yet, and they are significantly different from what's on that git. I'll upload them to git in a bit.

Also I don't usually load the atag.o module, because I coded it in an attempt to make it dump atags on the next reboot, but currently it panics so I never attempted to fix it and I instead opted to have someone on an unlocked bootloader flash a kernel i made that dumps atags out and had them send it to me.

The proc_atags.ko just creates /proc/atags entry for convenience as that's how kexec-tools checks for atags

Sent from my Nexus 6P using XDA-Developers mobile app
@Surge1223 Oh, thanks for the reply. I just grabbed the atags_compat.c from Torvald's git. I also had to get file.h because the one in my kernel tree doesn't have fdget. Unfortunately kexec.c has references to purgatory which I haven't messed with since I built my kexec tools (both from delewer's thread and now here from okarin). Here is the first of those errors: "s4zoom/kexec-surge/kexec.c:643: error: 'struct kimage' has no member named 'purgatory_info". I was wondering about that /proc/atags entry created when we load the procfs_rw module. I thought maybe we were supposed to write something to it, but it has pretty strict permissions. If you get a chance to upload a snapshot of your out-of-tree module build that would be great.
 

Surge1223

Recognized Contributor
Nov 6, 2012
2,622
7,466
Florida
Google Pixel 6 Pro
@Surge1223 Oh, thanks for the reply. I just grabbed the atags_compat.c from Torvald's git. I also had to get file.h because the one in my kernel tree doesn't have fdget. Unfortunately kexec.c has references to purgatory which I haven't messed with since I built my kexec tools (both from delewer's thread and now here from okarin). Here is the first of those errors: "s4zoom/kexec-surge/kexec.c:643: error: 'struct kimage' has no member named 'purgatory_info". I was wondering about that /proc/atags entry created when we load the procfs_rw module. I thought maybe we were supposed to write something to it, but it has pretty strict permissions. If you get a chance to upload a snapshot of your out-of-tree module build that would be great.
You have to add this to the struct kimage in kexec.h

Code:
struct purgatory_info purgatory_info;

The /proc/atags doesn't have to be read if you provide a atags file pointer instead, but unfortunately some early code in kexec-tools has the ability to be able to use atags (regardless of whether or not you provide atags as a file) dependent on if it detects /proc/atags existing. The good news is that the atags segment it actually will use is whatever the file pointer you provide is (i.e., --atags-FILE=./atags).

For a lot of this to compile out of tree a few includes need to be changed and a few things commented out. You'll see what I mean when I get a chance to upload.





Sent from my Nexus 6P using XDA-Developers mobile app
 
  • Like
Reactions: capt_planit

madushan1000

Senior Member
Feb 25, 2013
84
62
Ratnapura
So u guys are giving up on kexec? If so, I don't expect you to answer. @Okarin What is with this code in kexec.c line 1457 and 1458-- "kernel_restart_prepare(NULL); machine_shutdown();" ? A stub to make it compile (line numbers might be wrong but should be close)? We aren't using kernel_restart_prepare anymore? Or machine_shutdown? They sound important....When I uncomment reboot.h in kexec.c #includes in @madushan1000 build I get unknown symbol errors. I know how to work around this but it would be great to unify the code base for (now obsolete) kexec_load.ko. @delewer @Crabapple @Surge1223 @calcprogrammer and that ryan guy who talks to surge (and others who I'd like to mention--can you post additional code). @Surge1223 you only built in kernel? No external module? I have modified Madushan1000's kexec.c so that the commented out includes buld (as in yours Okarin) on my gcc 4.4.3 (crusty, I know, don't ask). More code to look at would be great. For the exercise more than eventual success, I have to believe.... c_p
I'm not really working on any of these right now. I keep hoping to take a look at qsee stuff when I get a free time.
 
  • Like
Reactions: capt_planit

madushan1000

Senior Member
Feb 25, 2013
84
62
Ratnapura
I'm bored so I gonna try around a bit.

qsee functions in kernel confirmed.
echo 1 > /proc/sys/kernel/kptr_restrict
grep -i qsee /proc/kallsyms

I build the shared library now .. just seeing how to use it.
254|root@kodiak:/ # /data/local/wotw
[-] Failed to load Widevine: No such file or directory

strace was my friend :)
It wanted to open the widevine files in the wrong place - I just straced for the open call and put them where they should be - no need to to anything to the code that can be worked around otherwise.

254|root@kodiak:/ # /data/local/wotw
[+] Widevine load res: 0
[+] Trying to probe 0x0D62D000
[+] Crashed, jumping ahead
[+] Trying to probe 0x0D65A000
[+] Crashed, jumping ahead
[+] Trying to probe 0x0D687000
[+] Crashed, jumping ahead
[+] Trying to probe 0x0D6B4000
[+] Crashed, jumping ahead
[+] Trying to probe 0x0D6E1000
[+] Crashed, jumping ahead
[+] Trying to probe 0x0D70E000
[+] Crashed, jumping ahead
[+] Trying to probe 0x0D73B000
[+] Crashed, jumping ahead
[+] Trying to probe 0x0D768000
[+] Crashed, jumping ahead
[+] Trying to probe 0x0D795000
[+] Crashed, jumping ahead
[+] Trying to probe 0x0D7C2000
[+] Crashed, jumping ahead
[+] Trying to probe 0x0D7EF000
[+] Crashed, jumping ahead
[+] Trying to probe 0x0D81C000
[+] Crashed, jumping ahead
[+] Trying to probe 0x0D849000
[+] Crashed, jumping ahead
[+] Trying to probe 0x0D876000
[+] Crashed, jumping ahead
[+] Trying to probe 0x0D8A3000
[+] Crashed, jumping ahead
[+] Trying to probe 0x0D8D0000
[+] Crashed, jumping ahead
[+] Trying to probe 0x0D8FD000
[+] Crashed, jumping ahead
[+] Trying to probe 0x0D92A000
[+] Crashed, jumping ahead
[+] Trying to probe 0x0D957000
[+] Crashed, jumping ahead
[+] Trying to probe 0x0D984000
[+] Crashed, jumping ahead
[+] Trying to probe 0x0D9B1000
[+] Crashed, jumping ahead
[+] Trying to probe 0x0D9DE000
[+] Crashed, jumping ahead
[+] Trying to probe 0x0DA0B000
[+] Crashed, jumping ahead
[+] Trying to probe 0x0DA38000
[+] Crashed, jumping ahead
[+] Trying to probe 0x0DA65000
[+] Crashed, jumping ahead
[+] Trying to probe 0x0DA92000
[+] Crashed, jumping ahead
[+] Trying to probe 0x0DABF000
[+] Crashed, jumping ahead
[+] Trying to probe 0x0DAEC000
[+] Crashed, jumping ahead
[-] Failed to find application
: No such file or directory
Now I need to get the memory region right ..

Right now I am wondering how he found the memory regions ... I am reading back his blog but it doesn't explain that.

Enough for today.
What I gathered is that we somehow need to find that memory region.
Maybe I missed something but I can't find a hint on how to do that.

Back again ...
Now I need to somehow extract the correct values for the Safe-Memory-Area.
I try to understand how he came to that specific value.
I guess we have to load the library in the mediaserver process. I'm not sure how to do that though.
 
O

Okarin

Guest
So u guys are giving up on kexec? If so, I don't expect you to answer. @Okarin What is with this code in kexec.c line 1457 and 1458-- "kernel_restart_prepare(NULL); machine_shutdown();" ? A stub to make it compile (line numbers might be wrong but should be close)? We aren't using kernel_restart_prepare anymore? Or machine_shutdown? They sound important....When I uncomment reboot.h in kexec.c #includes in @madushan1000 build I get unknown symbol errors. I know how to work around this but it would be great to unify the code base for (now obsolete) kexec_load.ko. @delewer @Crabapple @Surge1223 @calcprogrammer and that ryan guy who talks to surge (and others who I'd like to mention--can you post additional code). @Surge1223 you only built in kernel? No external module? I have modified Madushan1000's kexec.c so that the commented out includes buld (as in yours Okarin) on my gcc 4.4.3 (crusty, I know, don't ask). More code to look at would be great. For the exercise more than eventual success, I have to believe.... c_p

machine_shutdown - is in machine_kexec.c
kernel_restart_prepare - is in sys.c

Both functions are delivered with the module and do some basic setup.
Be aware of the freaking watchDog - one wrong step and it barks and the whole SoC resets itself.
 

capt_planit

Senior Member
Apr 6, 2016
63
17
Thanks for the answer @Okarin. I saw the commented sea of code around machine_shutdown and kernel_restart_prepare and thought you had done it. I went back to Delewer's V11 and realize it has been like that since at least then. I'm interested because I get symbol errors from these 2 in two build trees but not the other three (with the code near identical). Just grasping at straws. Have you considered running a chron job to kill the watchdog? Maybe through init.d script or something? I'm still stuck at "function not found" with one module, no memory hole with another and machine_kexec_prepare---- kexec_load - result : '0' with Delewer's V10 so it will be some time until I encounter the dog, if at all.
 

madushan1000

Senior Member
Feb 25, 2013
84
62
Ratnapura
so I asked gal how to find secapp region start address, It's printed into dmesg at boot
0.451751] QSEECOM: qseecom_probe: secure app region addr=0x7b00000 size=0x500000
I tried running the exploit with these values but no luck yet. We might have to follow gals complete procedure and find the correct symbols for the fire phone ourselves.
And apparently we can run the exploit as root, without loading it into the mediaserver.
 
Apr 2, 2016
32
2
This looks SERIOUSLY promising.

I'm dare not be that hopeful yet, but please forgive me for asking (I hope I'm not violating any rules with this):

Can someone explain to me how much closer are we with this? The KeyMaster keys must mean that effectively we are able to completely blitz through the bootloader lock now and sign our own roms and boot them, right? Thus moving on to latest kernels and android versions, 6.0.1 or even Android N, yes?

This means we can also decrypt the HAL and enable full fledged hardware interaction, perhaps?
 

capt_planit

Senior Member
Apr 6, 2016
63
17
This is my kexec mod sources before I stopped working on it
https://github.com/madushan1000/kexec-mad
I have no idea which kexec-tools version I used because I have about 10 repos cluttering on my laptop :p (It's probably the one that uses a /dev/ driver to communicate instead of the syscall)

I believe Mad's module is from hashcode's implementation. I base this on the fact that it creates a /dev/kexec entry and some of hash's commits implement this design. It is also why I've had trouble debugging it from delewer's kexec tools, which uses /dev/kexec_driver. Additional. If I load Okarin's module, run kexec (errors out) unload module and load Mad's module I can trigger a spontaneous reboot by evoking kexec again...woot!
 

madushan1000

Senior Member
Feb 25, 2013
84
62
Ratnapura
This looks SERIOUSLY promising.

I'm dare not be that hopeful yet, but please forgive me for asking (I hope I'm not violating any rules with this):

Can someone explain to me how much closer are we with this? The KeyMaster keys must mean that effectively we are able to completely blitz through the bootloader lock now and sign our own roms and boot them, right? Thus moving on to latest kernels and android versions, 6.0.1 or even Android N, yes?

This means we can also decrypt the HAL and enable full fledged hardware interaction, perhaps?
not really, keymaster keys are only valid for the secure storage android uses to store secret keys like vpn passwords. By having the keymaster keys you can decrypt them but not the bootloader stuff AFAIK. But we can certainly try to do that. If Amazon was dumb enough to use the same key for signing kernels we will be in luck.

But the other ways is much more promising, hot swapping the kernel trough tz exploit. Or maybe unlocking the bootloder by overwriting the key verification segments in the SBL or something.
 
O

Okarin

Guest
I tried around and couldn't get it to work.
None of the exploits. Sorry folks I am dropping out.
Maybe I'll try the kexec-approach again. That one was the most promising I could find.
 

capt_planit

Senior Member
Apr 6, 2016
63
17
@Okarin why not try hashcode's kexec-tools coupled with your take on madushan's module? I'm currently lost in debugging hell with this implementation "c0 BUG: sleeping function called from invalid context at kernel/rwsem.c:21". Also, surge1223 posted his sources and you could try those as well. All are designed to use /dev/kexec instead of delewar's /dev/kexec_driver. I never got the /dev/kexec_driver to communicate properly. I will get back to it later. Good luck. c_p
 
  • Like
Reactions: azirgi

Top Liked Posts

  • There are no posts matching your filters.
  • 16
    Since fire phone doesn't have a bootloader unlock at the moment. There is no point in building a custom kernel. But By building a kernel we can build kernel modules which work on the stock kernel. And yes you can load unsigned kernel modules without a problem since fire phone doesn't use tz apps to verify kernel modules like Samsung does.

    Setup

    Source
    Download the fire phone sources for firmware 4.6.1 from here. And extract the platfrom.tar inside the archive to somewhere(KERNEL_DIR).

    toolchain
    You can use the android ndk from google, But it requires some setup. I'm using linaro toolchain from here. You can use compiler version 4.7, 4.8 or 4.9. Kernel I'm using (Firmware 4.6.3 - Linux 3.4-perf-g280c96c) is built with gcc-4.7. But I'm using this gcc-4.9. Download it, extract is somewhere(TOOLCHAIN_DIR) and add the $TOOLCHAIN_DIR/bin to your PATH. Theoretically you would be able to build the kernel on windows using Cygwin or MSYS tools but using Linux is better.

    config
    Connect your phone trough adb and run
    Code:
    adb pull /proc/config.gz
    zcat config.gz > $KERNEL_DIR/kernel/qcom/3.4/.config
    With this config you will run into some problems because of a missing "trapz_generated_kernel.h". I don't know if this is an auto generated file when they build android as a whole or amazon removed this explicitly(can they do that without violating GPL?). Anyway It looks trapz is some low level kernel debugging function(comment here if you know more about it). We can safely disable it. Open $KERNEL_DIR/kernel/qcom/3.4/.config in a text editor and change the lines
    Code:
    CONFIG_TRAPZ=y
    CONFIG_TRAPZ_TP=y
    CONFIG_TRAPZ_TRIGGER=y
    CONFIG_HAVOK=y
    to
    Code:
    #CONFIG_TRAPZ=y
    #CONFIG_TRAPZ_TP=y
    #CONFIG_TRAPZ_TRIGGER=y
    #CONFIG_HAVOK=y

    building
    Now edit the $KERNEL_DIR/kernel/qcom/3.4/Makefile and add this changes

    Code:
    EXTRAVERSION = -perf-g280c96c
    This is at the top of the makefile. If we don't add this, vermagic for the modules will differ from stock kernel and they won't load.

    ARCH=arm
    CROSS_COMPILE=arm-linux-gnueabihf-
    Here arm-linux-gnueabihf- is my cross compiler frefix. Look in $TOOLCHAIN_DIR/bin/ to find it.

    Now cd into $KERNEL_DIR/kernel/qcom/3.4/ and do
    Code:
    make

    The build will fail a few times complaining about missing headers. Most of the time it's just
    Code:
    #include <myheader.h>
    instead of
    Code:
    #include "myheader.h"
    Edit the source file where the build fails and change <>s to ""s. (maybe android ndk ignores the difference and include the headers anyway)

    After kernel compiles, we are good to go. We can use this kernel sources to build kernel modules for stock kernel.

    Kernel modules
    To build the kernel modules, we basically need two things. An approximate kernel source and the Module.symvers file from the original kernel. We can get the Module.symvers file by building the complete kernel as explained above or Just extract it from our stock kernel.
    To extract the Module.symvers from the stock kernel, extract the boot.img file from firmware update image. Get mkbootimg tools from here compile it and run
    Code:
    unmkbootimg --kernel zImage ---ramdisk ramdisk.cpio.gz -i boot.img

    After you get the zImage. Download extract-symvers script from here and run
    Code:
    python2 extract-symvers.py -B 0xc0008000 zImage > Module.symvers
    place this file in $KERNEL_DIR/kernel/qcom/3.4/ (You still have to do the changes mentioned above in kernel config and building section run make in the $KERNEL_DIR/kernel/qcom/3.4 and intrupt it after few seconds)

    Now you can build loadable modules against this source. Here is a hello world kernel module.
    Code:
    //hello.c
    #include<linux/module.h>
    #include<linux/kernel.h>
    #include<linux/init.h>
    
    static int __init hello_start(void)
    {
      printk("hello to the world from module");
      return 0;
    }
    
    static void __exit hello_end(void)
    {
      printk("heloo exit");
    }
    
    module_init(hello_start);
    module_exit(hello_end);
    Code:
    #Makefile
    KERNEL_DIR=<your kernel dir>/kernel/qcom/3.4
    
    obj-m := hello.o
    PWD := $(shell pwd)
    
    default:
    	$(MAKE) ARCH=arm CROSS_COMPILE=armeb-linux-gnueabi- -C $(KERNEL_DIR) SUBDIRS=$(PWD) modules

    Put this files in a folder and run make in it. Change the paths and cross compiler prefix according to your setup. and run make.
    After the build push the hello.ko to the phone.
    Code:
    adb push hello.ko /sdcard/
    adb shell
    su
    cd sdcard
    insmod hello.ko
    run dmesg and you'll see the message.

    I'm currently trying to build kexec module from hashcode's sources and USB OTG modules.

    I'm attaching a few thing helped me do this.
    3
    So u guys are giving up on kexec? If so, I don't expect you to answer. @Okarin What is with this code in kexec.c line 1457 and 1458-- "kernel_restart_prepare(NULL); machine_shutdown();" ? A stub to make it compile (line numbers might be wrong but should be close)? We aren't using kernel_restart_prepare anymore? Or machine_shutdown? They sound important....When I uncomment reboot.h in kexec.c #includes in @madushan1000 build I get unknown symbol errors. I know how to work around this but it would be great to unify the code base for (now obsolete) kexec_load.ko. @delewer @Crabapple @Surge1223 @calcprogrammer and that ryan guy who talks to surge (and others who I'd like to mention--can you post additional code). @Surge1223 you only built in kernel? No external module? I have modified Madushan1000's kexec.c so that the commented out includes buld (as in yours Okarin) on my gcc 4.4.3 (crusty, I know, don't ask). More code to look at would be great. For the exercise more than eventual success, I have to believe.... c_p

    Nah I prefer to build out of tree, makes it easier to modify and add code/files and see the results. I just use local copies of some includes, like kexec.h, etc. When I was working on this, I used a makefile similar to this

    Code:
    EXTRA_CFLAGS=-mno-unaligned-access
    EXTRA_CFLAGS=-fno-pic
    KDIR := /Users/surge/Downloads/kernel_samsung-jfltevzw-VRUGOC1-master
    
    KVER := $(shell uname -r)
    
    PWD := $(shell pwd)
    
    CCPATH := /Users/surge/arm-eabi-4.8/bin
    ARCH		= arm
    CPPFLAGS	=  -I$kernel/
    EXTRA_CFLAGS += -I${KDIR}/include
    
    obj-m += lkm_kexec.o
    lkm_kexec-objs := kexec.o mmu.o sys.o core.o  sleep.o \
    	            proc-v7.o tlb-v7.o cache-v7.o pabort-v7.o abort-ev7.o copypage-v6.o \
    	            mm_init.o entry-common.o driver_sys.o suspend.o idmap.o 
    	    
    obj-m += msm_kexec.o
    
    obj-m += proc_atags.o
    
    obj-m += atag.o
    atag-objs := atags_compat.o atags.o
    
    obj-m += arm_kexec.o
    arm_kexec-objs := machine_kexec.o relocate_kernel.o
    
    default:
    	$(MAKE) ARCH=arm CROSS_COMPILE=$(CCPATH)/arm-eabi- EXTRA_CFLAGS="-I${KDIR}/include/linux -fno-pic -mno-unaligned-access" -C $(KDIR) M=$(PWD) modules
    	cp -f `find $(PWD) -name "*.ko"` $(KDIR)/build/
    
    clean:
    	rm -f *.o.d *.o *.ko *.mod.c Module.symvers Module.markers modules.order \.*.o.cmd 	\.*.ko.cmd \.*.o.drm -rf \.tmp_versions

    lkm_kexec.ko ends up being pretty large (~800kb)

    in order for me to test kexec using these modules, I have to load them in the following order:

    insmod msm_kexec.ko
    insmod lkm_kexec.ko
    insmod proc_atags.ko
    insmod arm_kexec.ko

    usually ends up getting to soft_restart and freezes when switching to the static identity mapping to continue with reset.

    EDIT: if you're wondering why I dont just combine msm_kexec.ko, lkm_kexec.ko and arm_kexec.ko, its because I intentionally want kexec.c and machine_kexec.c code to be housed in different modules, also for some testing, I want the driver for char device registration to occur without initializing some of the machine_kexec and relocation info. A side benefit of this is that it also becomes very easy to see if the failure is kexec.c (and by extension, kexec-tools) related, or arch specific kexec implementation errors (machine_kexec, relocate_kernel, atags, etc)
    1
    They have yet to provide 4.6.3 and 4.6.4 kernel sources too.

    I don't know exactly but in order for NFC and camera to work drivers are required and they are in fact compiled into the kernel.
    The problem we currently have with NFC and camera is proprietary hal (hardware abstraction libraries) They are a part of Android and does not subject to GPL. Amazon changed the original android way how hal works and didn't release the sources!

    by looking at the kernel drivers maybe we would be able to implement hal from scratch. But I don't see that intense dev support for fire phone. If you are up for it camera sources are at $KERNEL_DIR/kernel/qcom/3.4/drivers/media/platform/msm/camera_v2/
    1
    Major MAJOR respect for all of you making the Fire Phone even better!
    1
    Yeah that's what I want to try.
    After fooling around with kexec I need to find the correct arguments for the kernel registers.
    A machine_number is required and currently this is 0xFFFFFFFF for some reason :p

    Will see what I can do over the weekend.

    I need to update the kexec version I am using .. this will be fun :D
    I should have bookmarked that thread about the trouble with older versions and wrong kernel support ...

    And of course .. the stuff I started from hacked around multiple things too...
    No I read the dtb file and what do I find ... it's not valid ... oh yeah

    On further inspection there is a little header before the realy device tree .. damn it XD need to strip that off
    Got rid of that tiny header -- whatever it was ...
    Now we have a kexec-2.0.11 running in userland and a more or less hacky kexec-module
    This is my kexec mod sources before I stopped working on it
    https://github.com/madushan1000/kexec-mad
    I have no idea which kexec-tools version I used because I have about 10 repos cluttering on my laptop :p (It's probably the one that uses a /dev/ driver to communicate instead of the syscall)