FORUMS
Remove All Ads from XDA

[REFERENCE] How to get an Android kernel up to date with Linux upstream

11,507 posts
Thanks Meter: 35,030
 
Post Reply Email Thread
Introduction

Hello everyone! This will be a thread to assist people with getting their device's Android kernel up to date with upstream Linux from kernel.org. This process will henceforth be referred to as "upstreaming". This thread will assume that you understand how to compile a kernel as well as working with git. If you need assistance with that, please refer to one of the following threads (or search for others):

The case for upstreaming

TL;DR: Adding upstream is important because these patches are added with stability and bug fixes in mind.

This section is necessary as many kernel developers have built up this false narrative that upstream is unstable and breaks drivers and OEM code. That's not universally true, at all, and defies the very definition of a stable kernel. Please read this section if nothing else, it is the most important!

Every Android device runs a version of the Linux kernel. However, that version may not be the latest. For example, my primary device at the time of writing this, the OnePlus 5, is running a 4.421 on stock whereas the latest upstream offers is 4.4.89, which is something on that magnitude of 1500-2000 patches different. Largely, the reason for this is CAF (Code Aurora Forum, an open source collaborative project largely sponsors by Qualcomm) does not merge upstream updates into their kernels, thus OEMs do not do it either.

It's important to stay up to date with the latest from kernel.org as there are many security and bug fixes coming down from the mainline kernel that fix bugs that have been present since 2007 in some cases. As stated by Greg Kroah-Hartman, the second most influential kernel developer in the entire world, "If you are not using a[n up to date] stable/longterm kernel, your machine is insecure" (source, at 17:10 but I recommend watching the whole thing).

A lot of naysayers will say a lot of the patches from upstream are irrelevant to the architecture of most phones (arm/arm64) or are for drivers we don't have; while that may be true, there are several patches that are relevant and as Greg said in the above video, if you think you can sort through what is relevant, good luck. He later says not to try that in a talk he gave at Kernel Recipes 2017, stating "Changes to parts of the kernel that you do not build/run are fine and can cause no problems to your system. To try and filter out only the changes you run will cause a kernel tree that will be impossible to merge correctly with future upstream...". I personally don't think I am better than the professionals that push this stuff for free, thus I add it all. Greg's been sponsored by Google to do this for Google's kernel/common (3.18, 4.4, and 4.9). He's helping the Google Pixel kernel team upstream the kernel for Android O, as evident by the android-msm-marlin-3.18-o-preview-3 branch. Lastly, Greg committed to extending long term support (LTS) to SIX years from his normal two years for certain kernels that Android devices ship with. If this were a pointless process, why would Google and Greg even be investing time and energy into promoting adding upstream and changing their architecture and routine to support this?

Others will say that upstream can cause instability. While I will concede there may be a few problem patches here and there (listed below), these are NOT the norm, at all. I will argue that if something breaks while adding upstream, you most likely did it wrong (didn't pay attention to conflicts, read the context of every patch, etc) and it's SUPER easy to come back from (see the section below regarding bisecting). Furthermore, read the stable kernel documentation, they are small fixes to real world problems. Follow the below process while paying attention and I can GUARANTEE you will have very few issues, if any at all.

As another example/point, merging upstream allows you to get ahead of Google's security upstream. The infamous Dirty COW vulnerability, the fix for which released in 3.10.104 (committed on October 20th), was added by Google in the December security update. Sure, you could just add that one patch but you still have to be paying attention to upstream and filtering. It takes less effort to merge it all than it does to pick and choose. Even looking at the December bulletin, there is another critical security bug that they say affects the Pixel devices. It was fixed in 3.18.37, almost six months before it was added by Google. The stable kernels are used by MANY OEMs in real world critical situations and have no issues. Facebook has stated that they upgrade their internal kernels many times with zero issues. These patches are also paid for by a lot of OEMs (check the signoffs since 2015) pushed to the public for free; I personally think it is extremely stupid to pass up on these fixes.


How to

The process is fairly simple but requires finesse when running into conflicts, which I will go over in the tips and tricks section.

1. Figure out which kernel version you are on

This can be done one of two ways: going into Settings > About Phone or opening up the Makefile within your kernel source and looking at the top.

There are four major kernel versions that are used to this day, along with the common chipsets that use them:
3.4 - MSM8974
3.10 - APQ8084, MSM8992, MSM8994
3.18 - MSM8996, MSM8996 Pro
4.4 - MSM8998

The last number in the version (or EXTRAVERSION in the Makefile) will tell you where you need to start. For example, if you are on 3.18.31, you'd be starting at 3.18.32. It's important to get the starting point right so you don't get a ton of conflicts right off the bat.


2. Fetch the latest kernel source from kernel.org
kernel.org houses the latest kernel source in its git repository. At the bottom of that page, there will be three fetch links. In my experience, Google's mirror tends to be the fastest but YMMV. Run the following command:
Code:
git fetch --tags https://kernel.googlesource.com/pub/scm/linux/kernel/git/stable/linux-stable.git <branch>
Where <branch> is the branch with your kernel version (linux-3.4.y, linux-3.10.y, etc).

The --tags flag allows you to easily cherry-pick/merge based on the tagged release commit, rather than digging for SHA-1 hashes.


3. Decide if you want to merge or cherry-pick the commits in

Next, you will need to choose if you want to merge the commits or cherry-pick. Here's the pros and cons of each and when you may want to do them.

NOTE: If your kernel source is in the form of a tarball, you will most likely need to cherry-pick, otherwise you will get thousands of file conflicts because git is populating the history based purely on upstream, not what the OEM or CAF has changed. Just skip to step 4.

Pros of cherry-picking:
  • Easier to resolve conflicts as you know exactly what conflict is causing an issue.
  • Easier to rebase as each commit is on its own.

Cons of cherry-picking:
  • It takes longer as each commit has to be individually picked.
  • Harder to see on first glance

This may be better for people who like to rebase or bisect a lot as each commit is applied to your own branch, rather than merged together.

Pros of merging:
  • It's faster as you do not have to wait for all of the clean patches to merge.
  • It's easier to see when a commit is from upstream as you will not be the committer, the upstream maintainer will be.

Cons of merging:
  • Resolving conflicts can be a bit more difficult as you will need to look up which commit is causing the conflict, it will not directly tell you.
  • Rebasing is difficult as you cannot rebase a merge, it will offer to cherry-pick all of the commit individually.

This is good for people who don't rebase or redo history often (like those who use Gerrit).


4. Add the commits into your source, one version at a time
The most important part of this process is the one version at a time part. There MAY be a problem patch in your upstream series, which could cause a problem with booting or break something like sound or charging (explained in the tips and tricks section). Doing incremental version changes is important for this reason, it's easier to find an issue in 50 commits than upwards of 2000 commits for some versions.

With cherry-picking:
Code:
git cherry-pick <previous_tag>..<current_tag>
Example:
Code:
git cherry-pick v3.10.73..v3.10.74

With merging:
Code:
git merge <current_tag>
Example:
Code:
git merge v3.10.74
WIth merge commits, I recommend adding the git log --oneline output like this, so it is easier to see what each merge contained and where the conflicts were. This snippet will help you with that (obviously replace my signoff with your own or remove it). If you run into conflicts after running that git merge command, you just need to fix them then run
Code:
git add . && git merge --continue
to get the editor to pull up with the log and conflict results. You may need to uncomment the conflict results for them to be visible in the commit message (just removing the # symbols).



Tips and tricks

1. How to resolve conflicts

While I cannot possibly teach you how to resolve every conflict given that a lot of that involves understanding C, I can give you some helpful hints about how I resolve conflicts.
  • Figure out if you already have the commit: Some vendors like Google or CAF will attempt to look upstream for critical bugs, like the Dirty COW fix, and their backports could conflict with upstreams. You can run
    Code:
    git log --grep="<part_of_commit_message>"
    and see if it returns anything. If it does, you can skip the commit (if cherry-picking) or ignore the conflicts (remove the <<<<< and everything between the ====== and >>>>>>).
  • Read what the commit is trying to do and see if the problem is already fixed: Sometimes CAF may fix a bug independent of upstream, meaning you can either overwrite their fix for upstream's or discard it, like above.

Otherwise, it may just be a result of a CAF addition, in which case you just need to shuffle some things around.

I have a mirror of the kernel.org repository on GitHub, which can be easier for looking up commit lists and diffs for conflict resolution. I recommend going to the commit list view first and locating the problem commit to see the original diff to compare it to yours:
https://github.com/nathanchance/linux-stable/commits/<branch>/<path>

Example:
https://github.com/nathanchance/linu...arm64/mm/mmu.c


2. Enable rerere

Git has a feature called rerere (standing for reuse recorded resolution), meaning that when it detects a conflict, it will record how you resolved it so you can reuse it later. This is especially helpful for both chronic rebasers with both merging and cherry-picking as you will just need to run
Code:
git add . && git <merge|cherry-pick> --continue
when redoing the upstream bringup as the conflict will be resolved how you previously resolved it.

It can be enabled by running the following command in your kernel repo:
Code:
git config rerere.enabled true


3. How to git bisect when running into a compiler or runtime error

Given that you will be adding a sizable number of commits, it's very possible for you to introduce a compiler or runtime error. Instead of just giving up, you can use git's built-in bisect tool to figure out the root cause of the issue! Ideally, you will be building and flashing every single kernel version as you add it so bisecting will take less time if needed but you can bisect 5000 commits without any issues.

What git bisect will do is take a range of commits, from where the issue is present to where it wasn't present, and then start halving the commit range, allowing you to build and test and let it know if it is good or not. It will continue this until it spits out the commit causing your issue. At that point, you can either fix it or revert it.

Process:
  1. Start bisecting:
    Code:
    git bisect start
  2. Label the current revision as bad:
    Code:
    git bisect bad
  3. Label a revision as good
    Code:
    git bisect good <sha1>
  4. Build with the new revision
  5. Based on the result (if the issue is present or not), tell git:
    Code:
    git bisect good OR git bisect bad
  6. Rinse and repeat steps 4-5 until the problem commit is found!
  7. Revert or fix the problem commit.

NOTE: Mergers will need to temporarily run git rebase -i <good_sha> to apply all the patches to your branch for proper bisecting, as bisecting with the merges in place will often times checkout onto the upstream commits, meaning you have none of the Android specific commits. I can go into more depth on this upon request but trust me, it is needed. Once you have identified the problem commit, you can revert or rebase it into the merge.


4. Do NOT squash upstream updates

A lot of new developers are tempted to do this as it is "cleaner" and "easier" to manage. This is terrible for a few reasons:
  • Authorship is lost: It's unfair to other developers to have their credit stipped for their work.
  • Bisecting is impossible: If you squash a series of commits and something is an issue in that series, it's impossible to tell what commit caused an issue in a squash.
  • Future cherry-picks are harder: If you need to rebase with a squashed series, it is difficult/impossible to tell where an conflict results from.

Leave the updates unsquashed, you'll thank me later.


5. Subscribe to the Linux Kernel mailing list for timely updates

In order to get notified whenever there is an upstream update, subscribe to the linux-kernel-announce list. This will allow you to get an email every time a new kernel is released so you can update and push as quick as possible.


6. Commits known to cause runtime or build errors

3.10:
3.18:



Custom kernels that are upstreamed



ROMs with upstreamed kernels built in



Receiving help

I am more than happy to answer any questions regarding this process in this thread. You can use my 7.1.2-base branch as a reference for solving conflicts and seeing how the merge commit style looks. Additionally, if you feel any information is missing or can be improved upon, please let me know! I want this to be a collaborative effort.

Good luck!
The Following 118 Users Say Thank You to nathanchance For This Useful Post: [ View ]
 
 
25th June 2017, 03:04 PM |#3  
DodoGTA's Avatar
Inactive Recognized Developer
Flag Šilutė
Thanks Meter: 1,006
 
Donate to Me
More
Quote:
Originally Posted by The Flash

/dev/null

Here's a note you should add: Don't "git cherry-pick", "git merge" or "git bisect" during a thunderstorm
The Following User Says Thank You to DodoGTA For This Useful Post: [ View ] Gift DodoGTA Ad-Free
25th June 2017, 06:13 PM |#5  
nathanchance's Avatar
OP Recognized Developer / Recognized Contributor
Flag Mesa, AZ
Thanks Meter: 35,030
 
More
I have clarified on how to know where to start (although it is fairly obvious) in section 1 of the how to.

Sent from my Nexus 6P using XDA Labs
The Following 2 Users Say Thank You to nathanchance For This Useful Post: [ View ]
25th June 2017, 07:28 PM |#6  
mosimchah's Avatar
Senior Member
Thanks Meter: 3,255
 
More
Thanks for the guide, it's very helpful, also I personally use this command to merge the tags, it's just one command which is nice
git pull https://kernel.googlesource.com/pub/...nux-stable.git -t (tag)


Sent from my LEX727 using XDA Labs
The Following 3 Users Say Thank You to mosimchah For This Useful Post: [ View ] Gift mosimchah Ad-Free
25th June 2017, 07:39 PM |#7  
nathanchance's Avatar
OP Recognized Developer / Recognized Contributor
Flag Mesa, AZ
Thanks Meter: 35,030
 
More
Quote:
Originally Posted by mosimchah

Thanks for the guide, it's very helpful, also I personally use this command to merge the tags, it's just one command which is nice
git pull https://kernel.googlesource.com/pub/...nux-stable.git -t (tag)

Sent from my LEX727 using XDA Labs

Yup, git pull is shorthand for git fetch && git merge, so it's fetching the tag then merging it. Definitely quicker if you are going fast.

To everyone, I have added section 5 and 6 under tips and tricks, going over how to easily get notified of upstream updates and known problem commits to help you in the journey upwards!
The Following 13 Users Say Thank You to nathanchance For This Useful Post: [ View ]
26th June 2017, 10:34 AM |#8  
ZawZaw's Avatar
Senior Member
Flag Pyay, Myanmar
Thanks Meter: 2,721
 
Donate to Me
More
Thanks Sir @The Flash for your new Guide.
I'm now learning kenel development.
Your guide is very useful.

Thank You very Much.

BTW,
I want to request to use your GitHub's GCC Prebuilt Toolchain Repo for my Nexus5X kernel building.
I want to use this ToolChain Compiler for my Kernel.

Have a permission?
OR
How to request to you?

•••

Sent from my Google Nexus 5X using XDA Labs
26th June 2017, 01:55 PM |#9  
Senior Member
Flag Montreal
Thanks Meter: 33
 
More
Thank you very much, well done.
26th June 2017, 04:19 PM |#10  
nathanchance's Avatar
OP Recognized Developer / Recognized Contributor
Flag Mesa, AZ
Thanks Meter: 35,030
 
More
Quote:
Originally Posted by ZawZaw

Thanks Sir @The Flash for your new Guide.
I'm now learning kenel development.
Your guide is very useful.

Thank You very Much.

BTW,
I want to request to use your GitHub's GCC Prebuilt Toolchain Repo for my Nexus5X kernel building.
I want to use this ToolChain Compiler for my Kernel.

Have a permission?
OR
How to request to you?

•••

Sent from my Google Nexus 5X using XDA Labs

Go for it.
The Following User Says Thank You to nathanchance For This Useful Post: [ View ]
26th June 2017, 05:16 PM |#11  
ZawZaw's Avatar
Senior Member
Flag Pyay, Myanmar
Thanks Meter: 2,721
 
Donate to Me
More
Quote:
Originally Posted by The Flash

Go for it.

Thank You Sir.

•••

Sent from my Google Nexus 5X using XDA Labs
Post Reply Subscribe to Thread
Previous Thread Next Thread
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes