• XDA Forums have been migrated to XenForo. We are aware of several issues including missing threads, logins not working, and more. To discuss, use this thread.

[XZ1] rooted kernel hiding bootloader unlock with working fota

should my unlock hiding kernel be pre-rooted?

  • yes, a kernel with magisk ready for FOTA upgrade

    Votes: 21 56.8%
  • no, will patch myself using current magisk version

    Votes: 12 32.4%
  • no, alt use flashing it and current magisk over it

    Votes: 4 10.8%

  • Total voters
    37

j4nn

Recognized Developer
Jan 4, 2012
1,204
2,318
0
rooted kernel hiding bootloader unlock
with working sony stock fw fota updates
for Sony Xperia XZ1

Firmware Over the Air system updates have been disabled/not working with sony xperia phones with unlocked bootloader.
Also many sony drm functions are disabled if fw detects unlocked bootloader even if device master key was recovered.

I've implemented a kernel patch for xperia XZ1 Compact / XZ1 / XZ Premium phones that properly masks bootloader unlock status so it appears as still locked for sony stock firmwares.
This allows FOTA updates to be installed if running completely unmodified stock firmware. This is possible if this kernel is just booted from usb via fastboot instead of flashing it.
The kernel is pre-rooted, so you can have root as usual with magisk when running this kernel (you can use magisk system less patching to make changes to system/vendor partitions without actually modifying them).
For oreo fw the boot process is patched to hide magisk from sony ric daemon that stops the boot in case it thinks the bootloader is still locked. This special patch allows to pass safetynet including cts while having properly working magisk.

This kernel may be used (flashed) just to properly enable sony drm features, like video image enhancements, if device master key was recovered via locked state TA restore.
The bellow described way to install FOTA system update works with both - phone with TA restored and phone with drm keys lost. Both variants have been tested with xz1c.


How to use this kernel while planning to do FOTA system update eventually

Update: please see here for the latest usage instructions for kernels in flashable zip archive.

Please see screenshots bellow for this kernel in action (with xz1c) doing fota system update from oreo to pie and from pie to next pie version. There is also a video documenting this here. Few longer waiting parts have been cut out to fit the video under 15 minutes of youtube limit for not verified accounts.
  1. if your bootloader is still locked
    Use renoroot exploit to backup your TA, unlock your bootloader and restore TA-locked to recover device master key as described in
    [XZ1c/XZ1/XZp] temp root exploit to backup drm keys implemented thread.
  2. select one of the prepared kernels and download it
  3. make sure you are running unmodified stock firmware
    You need the version corresponding to the selected kernel - reflash the firmware to make sure it is unmodified.
    Please note: any mount of /system or /vendor partitions in write mode would result with modifications even if nothing is copied there.
    Be aware that some zip packages flashed from twrp may mount the partitions for write access even when that is not needed.
  4. reboot the phone to fastboot mode
    Use either "adb reboot bootloader" or
    enter fastboot by holding powered off phone's volume up key while connecting it to PC via usb cable and use 'fastboot reboot bootloader' command.
  5. boot the downloaded kernel via fastboot
    For example (xz1c):
    Code:
    fastboot boot boot-G8441-47.1.A.16.20-hideunlock-rooted.img
  6. enjoy your rooted phone which thinks it is still locked
    Sony apps will be offered to install/update. System FOTA update may come.
    Magisk will provide your root when magisk manager app is installed (offered on the first boot).
  7. if you need to use a custom recovery, like TWRP
    Do not flash it. If you do, FOTA update verification will fail.
    Instead use 'fastboot boot' the same way as with the kernel above, but instead of the kernel, boot the twrp image without flashing it.
  8. to install a FOTA system update
    • just start the update as usual
    • let it run until it finishes the installation
    • try to catch the restart then and hold volume up that time to enter fastboot
      you need to use following command to make next boot working
      Code:
      fastboot reboot bootloader
    • use 'fastboot boot' to boot kernel for fw to which fota updating to,
      for example (xz1c):
      Code:
      fastboot boot boot-G8441-47.2.A.4.45-hideunlock-rooted.img
    • if you miss the restart (or do not have the right kernel version),
      it does not matter, the installation will finish even when bootloader unlock is detected with the last reboot to updated system,
      so just 'fastboot boot' the corresponding 'hideunlock-rooted' kernel then

Alternative use of this kernel
  • If you do not like booting from usb via fastboot to startup your phone, you can flash the kernel and boot normally.
    But if you like to install FOTA system update then, you would need to flash the stock kernel first in order to make the fw untouched again (assuming no other changes to the fw, like system or vendor partitions, have been done) and boot the patched kernel via 'fastboot boot' as described above.
    You can backup stock kernel (and recovery) to avoid need to download full stock fw when you need to restore stock kernel & recovery when you decide to install fota system update - see here and following post for more details please.
  • If you do not care about FOTA, just do not install it.
    And use this kernel just to enable all sony drm features that are available on a locked phone (assuming locked state TA has been restored).
    In case you like to make some modifications to system or vendor partitions (as you do not care about fota), you would need to disable verity in the kernel - please see post#3 for noverity variants of oreo kernels and linked post describing howto switch verity off via magisk in all pie kernels.

Downloads
See the post#2 please.

Source code
The patches are provided under GPL (that means you may include them in your builds, but you need to provide buildable source of released binaries /true for any kernel change btw/).

Credits
  • Thanks to @tonsofquestions for lot of initial testing of this concept when I did not have a phone with unlocked bootloader and for discovering the need to reboot to fastboot by a command to make the 'fastboot boot' command properly boot the supplied kernel image.
  • Thanks to @topjohnwu for his excelent magisk tool.

If you find my work useful, consider donating here please:
https://j4nn.github.io/donate/
Thank you.


XDA:DevDB Information
kernel_bluhide_poplar, Kernel for the Sony Xperia XZ1

Contributors
j4nn
Source Code: https://github.com/j4nn/sonyxperiadev-kernel-copyleft

Kernel Special Features: proper hiding of bootloader unlock, sony ric with magisk hack

Version Information
Status: Stable
Stable Release Date: 2019-02-10

Created 2019-02-10
Last Updated 2019-08-07
 

j4nn

Recognized Developer
Jan 4, 2012
1,204
2,318
0

SGH-i200

New member
Aug 31, 2009
1,040
241
0
This should be the first paragraph! ;)
[...] use this kernel just to enable all sony drm features that are available on a locked phone (assuming locked state TA has been restored).
Since I do not care about OTA updates, because we can download every firmware via XperiFirm and flash it via newflasher, I will use your modded kernel in the 'alternative' way! :good:
Alternative use of this kernel
If you do not like booting from usb via fastboot to startup your phone, you can flash the kernel and boot normally.
 
Last edited:
  • Like
Reactions: DHGE and tombbb

SGH-i200

New member
Aug 31, 2009
1,040
241
0
@j4nn - I flashed "boot-G8341-47.2.A.6.30-hideunlock-rooted.img" on my G8341 which has the latest Pie firmware (G8341_47.2.A.6.30_Customized DE_1310-4290_R6C) installed, but the device did not boot into Android after that! :crying: I flashed the latest official Magisk before your kernel. Do I need to use your patched Magisk instead?

Did you forgot to add the patched magisk flashable img file? "hide magisk from sony ric daemon on early boot phase (v18.1-manager-v7.0.0-ric branch)"
Downloads
Maybe the firmware version is not compatible with the one you took the kernel from? The firmware folder includes:
kernel_X-FLASH-ALL-C93B.sin
boot/bootloader_X_BOOT_MSM8998_LA2_0_P_107_X-FLASH-ALL-C93B.sin
...
 
Last edited:
  • Like
Reactions: tombbb

j4nn

Recognized Developer
Jan 4, 2012
1,204
2,318
0
@SGH-i200, I just compared the stock kernel boot image extracted from G8341_Customized DE_1310-4290_47.2.A.6.30_R6C (which I have used to create it) and it seems all good to me:
- os version and patch level match
- device tree blobs are the same
- kernel command line is the same
- extracted kernel config is (except few comments as it is usual) the same, with poplar target (i.e. single sim XZ1)
- ramdisk is the same except changes that are expected (integrated magisk in order to provide root when booted from usb having unmodified stock fw)

Did it result with an infinite hang during boot (which phase of the boot splash animation)?
Or did it reboot during boot (a boot-loop) - which phase?

I am sorry I forgot to mention the kernels for other targets than xz1c have not been tested (I have only xz1c for testing).
But it should be safe to test the above howto as described. Even if the kernel was flashed (instead of just fastboot booted), recovering from this should be as simple flashing stock kernel back.
Anybody else tested this with XZ1?
 
Last edited:

SGH-i200

New member
Aug 31, 2009
1,040
241
0
Did it result with an infinite hang during boot (which phase of the boot splash animation)?
Or did it reboot during boot (a boot-loop) - which phase?
The SONY white splash screen appeared (with only black SONY text) and then it hang forever.

Even if the kernel was flashed (instead of just fastboot booted), recovering from this should be as simple flashing stock kernel back.
I flashed the kernel_X-FLASH-ALL-C93B.sin (and everything else as I did to upgrade to latest Pie) from the stock fw to recover.

After the XZ1 was booting again, I executed the fastboot booting, without problems! :)
adb reboot bootloader
fastboot boot boot-G8341-47.2.A.6.30-hideunlock-rooted.img
 
Last edited:
  • Like
Reactions: tombbb

j4nn

Recognized Developer
Jan 4, 2012
1,204
2,318
0
@SGH-i200, that sounds good. Now just flash it:
Code:
adb reboot bootloader
fastboot flash boot boot-G8341-47.2.A.6.30-hideunlock-rooted.img
disconnect from usb and power on.
If it booted from usb, I believe it should boot just fine if flashed as above too.
 
  • Like
Reactions: tombbb

SGH-i200

New member
Aug 31, 2009
1,040
241
0
If it booted from usb, I believe it should boot just fine if flashed as above too.
I flashed your patched kernel again and got into a bootloop: Bootloader unlocked warning and white SONY splash screen in an endless loop.

Since my XZ1 was connected to my Windows workstation already, I simply long pressed the volume up button till the notification light went blue, and booted into twrp to restore the stock kernel (rooted by Magisk to remove the dmverity stuff).
 
  • Like
Reactions: j4nn

j4nn

Recognized Developer
Jan 4, 2012
1,204
2,318
0
@SGH-i200, you are right, it is the verity thing.
I've assumed that verity is ignored when kernel detects unlocked bootloader, because the dm-android-verity kernel component used that is_unlocked() call to:
/* Allow invalid metadata when the device is unlocked */
- that comment is in the dm-android-verity.c source code.

But it looks like that is only a corner case and if verity metadata is valid, verity is active (if not disabled in device tree) even with unlocked bootloader.

Now I am wondering: for OTA updates we need untouched system, so having verity enabled in the kernel is good for this purpose - when you flash something that writes to system or vendor, you may detect it soon enough to be able to tell what it was. The main feature of magisk is that you should be able to "modify" system "systemlessly", i.e. without really writing to system or vendor partitions.
From this point of view it should be all good then.

But I understand that the "alternative" use of this kernel, i.e. not planning to do FOTA at all, just using only the unlock hide feature restoring drm functionality and using root the old way (with real writes to system) may be liked.
For this, just use magisk manager main screen, in "Advanced Settings" unselect the "Preserve AVB 2.0/dm-verity" option and then use the "Magisk INSTALL" button, confirm install of Magisk-v*.zip, select "Patch Boot Image File", browse to sdcard where you put my kernel image and select it.
Magisk will repack the boot image disabling dm-verity in device tree blobs, noting where you can find patched_boot.img.
Flash that and you should be good to go.
This procedure is valid for all my pie kernel builds.
I will repack the oreo 16.20 build as using magisk manager gui would not keep the magisk patch for sony ric daemon.
-- edit --
Updated post#3 with downloads of oreo kernels noverity repacks.
 
Last edited:

SGH-i200

New member
Aug 31, 2009
1,040
241
0
Now I am wondering: for OTA updates we need untouched system, so having verity enabled in the kernel is good for this purpose - when you flash something that writes to system or vendor, you may detect it soon enough to be able to tell what it was. The main feature of magisk is that you should be able to "modify" system "systemlessly", i.e. without really writing to system or vendor partitions. From this point of view it should be all good then.
I use AdAway and activated NightLight by copying a apk to /vendor/overlay .

Thanks for creating the noverity Oreo kernel versions! :good:

---------- Post added at 09:04 AM ---------- Previous post was at 08:57 AM ----------

The main feature of magisk is that you should be able to "modify" system "systemlessly", i.e. without really writing to system or vendor partitions.

But I understand that the "alternative" use of this kernel, i.e. not planning to do FOTA at all, just using only the unlock hide feature restoring drm functionality and using root the old way (with real writes to system) may be liked.

For this, just use magisk manager main screen, in "Advanced Settings" unselect the "Preserve AVB 2.0/dm-verity" option and then use the "Magisk INSTALL" button, confirm install of Magisk-v*.zip, select "Patch Boot Image File", browse to sdcard where you put my kernel image and select it.
Magisk will repack the boot image disabling dm-verity in device tree blobs, noting where you can find patched_boot.img. Flash that and you should be good to go. This procedure is valid for all my pie kernel builds.
If I flash your patched Pie kernel via TWRP and flash Magisk right after, this should have the same effect as patching your kernel via the Magisk app, right?

@j4nn - "using root the old way (with real writes to system)" - will flashing Magisk via TWRP change this to system-less root?
 
Last edited:

j4nn

Recognized Developer
Jan 4, 2012
1,204
2,318
0
Thanks for the kernel but after flashing the kernel the phone don't go into doze (deep sleep). Any suggestions on this?
I have no idea. In my opinion, if you get such behaviour with my kernel, you will get exactly the same behaviour with stock kernel of the same version.
My kernel is compiled from sony's open source kernel package, with kernel config that's identical (except few comment lines) to the one compiled in stock kernel.
My kernel patch only fixes kernel command line that comes from bootloader and hijacks trust zone api to mask the bootloader status to appear as still locked to firmware's userspace.
These changes hardly could have any influence on power saving behaviour.
Please flash the same version stock kernel and test again for deep sleep.
Could not be that caused by some app you've installed?

If I flash your patched Pie kernel via TWRP and flash Magisk right after, this should have the same effect as patching your kernel via the Magisk app, right?
@j4nn - "using root the old way (with real writes to system)" - will flashing Magisk via TWRP change this to system-less root?
Most likely flashing magisk via twrp after flashing the patched pie kernel would disable verity too.
Using the image file - you have it under your control.
Flashing from twrp - some magisk scripting tries to detect if verity should be disabled or not.
Also a kernel image already containing magisk contains the setting of verity and encryption - not sure how that is used when flashing magisk again over it.
I tried it with my xz1c and verity was flipped from enabled to disabled - so the same effect (this time).
But as described, some detection is used, so the results might not always be the same.
 
  • Like
Reactions: SGH-i200

sinkoo1979

Active member
Oct 29, 2016
548
79
28
Yigo
I have no idea. In my opinion, if you get such behaviour with my kernel, you will get exactly the same behaviour with stock kernel of the same version.
My kernel is compiled from sony's open source kernel package, with kernel config that's identical (except few comment lines) to the one compiled in stock kernel.
My kernel patch only fixes kernel command line that comes from bootloader and hijacks trust zone api to mask the bootloader status to appear as still locked to firmware's userspace.
These changes hardly could have any influence on power saving behaviour.
Please flash the same version stock kernel and test again for deep sleep.
Could not be that caused by some app you've installed?

reflashed stock firmware and everything is fine. Thanks for the kernel.
 
  • Like
Reactions: j4nn

leonaheidern

New member
Dec 3, 2010
41
10
0
Downloads
This is for alternative use only - please see post#10 for more details.
boot-G8341-47.1.A.16.20-hideunlock-rooted-noverity.img
boot-G8342-47.1.A.16.20-hideunlock-rooted-noverity.img

Screenshots of XZ1c FOTA system update from pie 47.2.A.4.45 to pie 47.2.A.6.30 version
(video available here since 08:10 time)
Hi j4nn

I unlocked my bootloader on 47.1.A.16.20 and lost my drm keys. I upgraded via newflasher_v13 to 47.2.A.4.45 Customised SG as that is where i am based and where my phone is from. Unfortunately when I flash boot-G8342-47.2.A.4.45-hideunlock-rooted.img my phone still boots up showing the device has been unlocked and the Backup and Restore feature of Xperia devices does not work. Am I correct to say if I lost my drm keys that I am unable to use this kernel to hide my bootloader unlock?

Thank you for answering me.
 

j4nn

Recognized Developer
Jan 4, 2012
1,204
2,318
0
@leonaheidern, the kernel would hide the unlocked bootloader regardless you lost drm keys or restored locked TA.
Please check you kernel build date when you boot your phone in about menu.
 

lilbrat

Well-known member
Jun 13, 2011
991
364
63
Amman
@j4nn, I have a quick question and a bit off topic, I'm still tinkering a bit but I did manage to backup my ta. and I flashed janjan's so things may seem a bit off, (see pic).. but I did restore my ta. before flashing Jan's kernel, any idea what may happen if I flashed yours on top?

oh and everything is working..
 

Attachments

Last edited:

j4nn

Recognized Developer
Jan 4, 2012
1,204
2,318
0
@lilbrat, I guess that the result would be the same if you flashed my kernel without flashing janjan's before it.
Just note please, my kernels as they are have verity enabled, so you need to have unmodified stock fw.
Or disable verity as hinted for alternative use.
With my kernel and restored TA your security screen would look perfect as with a still locked phone.
 

lilbrat

Well-known member
Jun 13, 2011
991
364
63
Amman
@lilbrat, I guess that the result would be the same if you flashed my kernel without flashing janjan's before it.
Just note please, my kernels as they are have verity enabled, so you need to have unmodified stock fw.
Or disable verity as hinted for alternative use.
With my kernel and restored TA your security screen would look perfect as with a still locked phone.
OK.. thanks for the info... now I have another question, now that xz1 has a new update out and granted it will take you a bit to catch up if you need to redo all your kernels you have going... (great work by the way) Will I need to reflash my ta. backup for the new firmware and or just your kernel?