FORUMS
Remove All Ads from XDA
OnePlus 6 On Sale

[2018.4.29] Magisk v16.0 - Root & Universal Systemless Interface [Android 5.0+]

1,790 posts
Thanks Meter: 44,453
 
By topjohnwu, Recognized Developer / Recognized Contributor on 3rd October 2016, 06:00 PM
Thread Closed Email Thread
14th October 2017, 08:43 PM |#31  
OP Recognized Developer / Recognized Contributor
Flag Taipei
Thanks Meter: 44,453
 
Donate to Me
More
2017.10.15 Magisk v14.3
Here is another release, which comes with quite a few exciting updates!

Introduce Invincible Mode
Let's get straight into the interesting part!
Due to the nature of Magisk, the daemon is responsible for everything: initial boot scripts, magic mount, root, MagiskHide, logging etc., so it is important to make sure it never fails. Even though I have spent A LOT of effort in making the daemon as stable as possible, there are still tons of probability that might cause the Magisk daemon to crash: modules that messes with the internal workings of Magisk; root apps that have unexpected behaviors etc.. With the daemon crashed, root will lost, and SafetyNet will no longer pass, which would be very annoying and frustrating.
Even though I personally never experienced any root loss issues, the number of complaints urged me to implement a "self recoverable" daemon, which I would like to call it "invincible mode". Basically, the system will make sure the Magisk daemon is always running, and will restart if crashes (or even manually force killing the daemon!). Most of the effort is to make the restart seamlessly: that is logging will continue, and MagiskHide will still start up if enabled.

Rewrite Magisk Logcat Handling
The daemon uses logcat in many different situations: the general logging (/cache/magisk.log), verbose boot+debug logging (/data/magisk_debug.log)(only in beta builds), and MagiskHide (monitor process startups). In the worst case scenario, the daemon will come along with 3 logcat child processes; in addition, each process will require its own implementation to handle logcat errors and restarting. In this new update, I created a single logcat subprocess for the daemon, and each worker threads that need logs can simply plug in its own listener on-the-go, and leave anytime if not needed. By doing so, each process can access the "real-time" logs (if starting a new logcat process, it will first dump logs in the buffer before showing the real-time logs), and don't need to worry about restarting logcat if anything happens.

Magisk Manager Now 100% FOSS
The Magisk Manager also got a pretty sweet update! Tons of improvements are added, check the changelog for the detailed new features. (The new manager will be pushed to stable channel)
One thing I would like to highlight is that - Magisk Manager is now officially 100% FOSS! You might be wondering: isn't Magisk already open source? The truth is that Magisk Manager contains part of Google's proprietary "Google Play Service SDK" for SafetyNet checking, which make Magisk Manager technically not "FOSS". It is quite stupid to lose the "FOSS" tag for such minor thing as SN checking, but I found it extremely convenient and didn't really want to remove this feature. So what I did is separate the proprietary part from the application, and let users decide whether to download an external code extension (contains GoogleApi) for SafetyNet checking, which actually is pretty challenging since it involves dynamic dex loading and uses quite a few reflection techniques, but it was a lot of Java fun lol.
This makes Magisk Manager feasible to be placed on F-Droid, I will find some time to publish to their repository.

Online Repo
Magisk Manager is now updated to reject all repos with the template version lower than 4. I already filed issues to the outdated repos, if no action is done, your repo will be removed very soon.
For the next stable release, I will push even further to require all repos to be at least version 1400. For developers not using my template, make sure to test your module against Magisk v14+, and please also remember to bump the template entry in module.prop to 1400 to prevent from blacklisting.
These measures are done to make sure developers keep up with the latest changes, and don't let online repos be filled with outdated, boot-loopable modules. Thank you for your cooperation!

If nothing catastrophic happens on this beta build, the new stable release is imminent
The Following 791 Users Say Thank You to topjohnwu For This Useful Post: [ View ]
 
 
22nd November 2017, 05:52 PM |#32  
OP Recognized Developer / Recognized Contributor
Flag Taipei
Thanks Meter: 44,453
 
Donate to Me
More
2017.11.23 Magisk v14.5
Many people dislike the direction Google is leading Android judging from the Pixel lineup. Inevitably we will see Pixel-2-like configurations on all newly released device at some point in the future, since any device shipping with Oreo will need to pass Google's new VTS. And for sure we start seeing cheap shots like lazy OEMs releasing new devices with outdated software, which I'm VERY disappointed to see the "highly praised" OnePlus 5T on the list, but I digress.
The introduction of Project Treble brings a huge change to how we mod our devices, and Magisk is exactly created to accommodate this nature: minimum intrusion with maximum versatility. It is sad to see people in the forums happily rooted their shiny new Pixel 2 XL, but immediately modify system files just like the good old days, completely ignoring what Magisk is created for in the first place. Not blaming anyone here, but I guess the systemless concept still requires some time to be widely accepted.

Forgive my little rant, let's get into the nitty-gritty , here's another release with exciting new features!

Dynamic Runtime Initramfs Setup
As mentioned, installing Magisk on newer devices starting from the original Pixel series requires special treatments, and this has led to two separate logic for installation. I have spend some time rewriting the startup process of Magisk, and unified the installation across all devices. The magisk binary and init scripts are embedded into magiskinit, and magiskinit will dynamically inject these files and patch (or construct in the case of skip_initramfs devices) initramfs and sepolicy on-the-go. The actual modifications done on the device is replacing init with magiskinit in ramdisk of boot.img , and additional dtb/kernel/dtbo patches if required.

MagiskHide - Even More Badass!
More and more apps (those that doesn't use Google's SafetyNet API to check device integrity) has been updated to detect Magisk, such as the popular game Fate Grand Order and a handful of banking apps. I got bored and had my hands on to fix this issue. The fix can be roughly split to two parts, both of them are future proof to a certain extent. There are still some stuffs that can be detected, but even if I can hide them easily I haven't bothered yet. Unless they come up with complete new innovative ways to detect Magisk, the new hiding method should work in all cases (at least for now).

The first part is that Magisk image's mount point is now moved under /sbin/.core (along with several internal paths) , from /magisk to /sbin/.core/img; a symlink (/magisk -> /sbin/.core/img) is still added for legacy support. However, the symlink could be temporarily removed when MagiskHide is working, so scripts that had hardcoded path "/magisk" should really start migrating to "/sbin/.core/img" since "/magisk" doesn't always exists. The result is that the existence of the folders will also be hidden with MagiskHide. Changing the mount point means some modules using bad techniques such as creating symlinks pointing into magisk internal paths might break! You shouldn't assume the Magisk mount point to always be /magisk. One possible way to overcome this is to NOT use symlinks but use hardlinks (command ln without the -s flag), but I still wouldn't recommended it since modules should not depend on Magisk's internal stuffs.
Some modules NOT using the util_functions.sh from Magisk will have high chances to break, please update accordingly, or migrate to use the global util_functions.sh so Magisk internal changes will reflect to your module

The second part of the puzzle is that Magisk Manager can now be properly hidden. Hiding Magisk Manager will now repackage itself with a randomly generated package name to replace the original one. The real magic and challenge here is to transfer root management from the old package name to the new one seamlessly. Even though I have put a ton of effort and this feature has been tested for quite a while, due to the complexity I would still only recommend enabling this if you ACTUALLY need it. If you face any issues, simply just reinstall the proper Magisk Manager back and re-hide again.

Welcome On Board, Pixel 2!
Lastly, let's welcome Pixel 2 (XL) on the official support list! One additional change to the Pixel 2 installation compared with v14.4 is that I added dtbo.img patching support to disable avb-verity of the vendor partition. You can now modify (though not recommended) the partition without issues. The DTBO is a separate partition on the device, and Magisk will backup the original one to /data/stock_dtbo.img.gz for restoration and uninstallation.

P.S. This release was meant to be a stable v15.0, but my testers suggested to remain as beta due to the couple big changes mentioned above. It is now a good time for Magisk Manager translators to submit your patches to Github and prepare for the upcoming stable release!
The Following 658 Users Say Thank You to topjohnwu For This Useful Post: [ View ]
21st December 2017, 11:17 PM |#33  
OP Recognized Developer / Recognized Contributor
Flag Taipei
Thanks Meter: 44,453
 
Donate to Me
More
2017.12.22 Magisk v14.6
Previous implementation of the hiding Magisk Manager mechanism has flaws, the Magisk Manager lack the capability to recover from the repackaged state.
If you have hidden (repackaged) Magisk Manager on v14.5, you HAVE TO upgrade both Magisk Manager and Magisk VIA THE APP to not mess up!!
Please DO check the updated Notes & Troubleshoots section in the OP if you plan to use the Hide Magisk Manager feature!!!


This should be the most thoroughly tested release ever LOL.
Not that much new exciting features, but tons of improvements along the way!!

Project Restructure
For those following the Github source would realize that the project has gone through a major restructure - the whole Magisk source code is now a single Android Studio project. Since Magisk's code is starting to become overwhelmingly huge, and also Magisk Manager (Java) is so deeply integrated with the core C/C++ part, the burden to develop Magisk is becoming more and more annoying. I decided to utilize Android Studio's CLion (a C++ IDE) NDK integration to assist the development, and another advantage is that I can develop Magisk Manager (Java) side-by-side with the native C/C++ part. Those who are willing to dig through Magisk's source can simply select the whole folder in Android Studio, and you can start developing immediately!
Just keep in mind: don't build Magisk with the built-in gradle scripts, the building process is quite complicated and all handled through the Python script "build.py"

Stability Fixes
Invincible mode has been rewritten. A new daemon "magiskinit_daemon" (a fork of magiskinit) will maintain a socket connection with the main magisk_daemon, and once disconnected (which means the daemon unexpectedly terminated), it will respawn the magisk daemon right away. This is much more robust than the previous implementation which relies on a init rc script, and the additional daemon will consume no power and very little memory since it literally does nothing unless the socket connection is broken.
A bug in the logcat monitor (a very essential part of MagiskHide and logging) which causes spawning infinite logcat processes and exhaust all resources until system reboot (not very common but fatal) is resolved. It has also be rewritten to be more efficient.

Huge MagiskSU Improvements
Since I picked up phh's Superuser source code, I have massively rewritten all the daemon part (communication, database handling, mount namespace etc.), but really never touched the actual "su" implementation. All CM based root (phh, Lineage SU, and previous MagiskSU) inherently include the same incomplete daemon-based su implementation (required since Android 4.3+) across the board. For this update I referenced the official Linux su manpage and made the behavior of MagiskSU as close to the reference as possible. The most noticeable change is that environment variables are now preserved from the original shell to the root shell. This was originally a default behavior of all Linux process forks, but since we are calling a new shell via socket by the daemon, additional work has to be done. More detailed info are in the changelogs.

MagiskHide Improvements - Continuation
In the last release, I added the feature to repackage Magisk Manager, and also move the mount point to a volatile location so it can be hidden by MagiskHide. As a continuation in this release, all Magisk related files are moved to /data/adb, a folder that already existed which has access permission (700). By doing so, any non-root process can never know the existence of any possible Magisk file on the device. In theory this should eliminate ANY naive way to detect Magisk (at least not involving hardware based cryptography) from a target process, but I know several apps that still can detect Magisk. Those might have used some other approaches to detect root (e.g. spawn another process like SafetyNet), I'll try to crack them if I find the time to do so. For those worrying the changes might break stuffs, I had added symlinks for backwards compatibility (just like the previous /magisk symlink), so it should work "most of the time" (as long as MagiskHide isn't actively running a hiding session). A complete redesigned Module template will come out along with the stable release (very soon).

Installation Flags
A new install mode: ramdisk high compression mode is introduced. In real world tests, a Magisk installation will add at most 4XX KB to the compressed ramdisk. If a device has a very small and restricted boot partition size, this size change might actually make the patched boot image unable to fit in. Starting from this release, if the free space in the boot partition is lower than 500KB, high compression mode will be triggered, Magisk will recompress your whole ramdisk with xz (lzma2 alg.). In real world tests, this will typically create near 2MB free space, so this means that Magisk should fit any possible device. This is possible thanks to updates to magiskinit, which can dynamically decompress your ramdisk on-the-fly on boot. The KEEPVERITY and KEEPFORCEENCRYPT flags now have new default settings.
If dtbo.img is detected on your device, KEEPVERITY is enabled by default. If your device is already encrypted, KEEPFORCEENCRYPT is enabled by default.
These default values will show up in Magisk Manager, and will also affect the installation via recovery.

Magisk Manager
Probably the biggest improvement for this release should be Magisk Manager. Although nothing massive changed visually, TONS of bugs and improvements are done under-the-hood. Repackaging Magisk Manager now works much better and more reliably. Settings will be migrated from the original manager to the repackaged one, and it can properly handle duplicate Magisk Managers. Tons of compatibility code has been added to support all possible su database setups. Please check the changelog for the full list of changes.
The Following 550 Users Say Thank You to topjohnwu For This Useful Post: [ View ]
25th December 2017, 10:21 PM |#34  
OP Recognized Developer / Recognized Contributor
Flag Taipei
Thanks Meter: 44,453
 
Donate to Me
More
2017.12.26 Magisk v15.0
Merry Christmas! And wow it has been quite some while since the last stable release. Well it's Christmas, let's drag Magisk out of the eternal beta!

To wrap up the changes from v14 to v15, there are two major parts: update Magisk to be modular and adaptive to follow Google's Pixel concept, and completely remove all possible traces of the existence of Magisk.
Ever since I added Pixel support to Magisk, I have been amazed by how Google has designed this whole new system: modding a device has changed completely. I am a strong advocate of these new changes, praising the effort Google has put into designing such weird but fascinating setup despite many view this as "moves against developers". With the announcement of Project Treble, all devices will go through the same "Pixel-ize" treatment, so this future is inevitable. Project Treble is all about "modular", and Magisk has been rewritten to embrace this concept. I have spent extra effort to backport the changes made requires to run on these new modern devices to downward support all devices, so we do not need 2 drastically different installation methods: every single device running Magisk shares the same setup. Most things are now all dynamic, a single binary can handle all sorts of configurations, constructing the environment adaptively, and for Treble enabled devices even works when modular parts are swapped out. Fun fact: for Pixel devices, we only need a single init binary to boot up and properly run Magisk!
The second major part is to theoretically hide Magisk from any possible detection methods. Tons of hints/signal/traces of the existence of Magisk are now removed, and Magisk Manager supports repackaging to completely hide from naive package name detections. These changes requires some major redesign in Magisk structure and previous paths.
The above is a quick recap of what has happened in the past 3 months. The following are some important announcements.

New Template 1500
Magisk v14.5+ has gone through some major redesigns, but thankfully backward compatibility is done to fully support down to vanilla 1400 template. Most existing modules using 1400 template as a base should still install and work fine, however module developers might need to do some minor updates to properly support v15.0, depending on how the developer has implemented things. The new 1500 template strips out even more logic within the module, making it nearly a stub to just call functions from the external util_functions.sh, a script which will be installed along with Magisk. This gives me the ability to fix issues and do adjustments in Magisk's implementation without modules needing to do any adjustments. The template is very unlikely to change in near future, so module developers feel safe to migrate all modules to the new template.

Some notes: in 1500 template, the entry "template" in module.prop is now changed to "minMagisk". Since some talented developers do not use the template, calling it "template" doesn't seem to make much sense to them, that's why I renamed them to "minMagisk", to what the value actually means: the minimum required Magisk version. But don't worry, the two keywords "template" and "minMagisk" is 100% interoperable, you can use either in your module.prop, choose the one that makes sense to you.

Magisk Repo Submission Server
Previously, adding a new module requires me to manually clone developers repo to the online repo, and manually add them to collaborators to allow upgrading. It is quite a pain, and if I was busy I don't even bother to do so, so this causes many submission requests to piled up. I written a server to fix this issue (the code of the server is also open source). The new server will not only monitor the requests and process them within seconds, but also acts as a moderator to make sure all submitted and existing repos are setup properly and not outdated. 10+ incorrectly configured/extremely outdated modules are already removed from the online repo.
New submissions should create issues here: Magisk_Repo_Submissions, the instructions are updated to assist developers to submit and properly maintain their modules. Please note that once you created a new issue for submission, it shall be addressed within seconds if my server is up. The server is currently running on my laptop, so it will not always be online, but overall your requests should be addressed in no more than a day (I don't think I will leave my laptop without Internet for more than a day lol)

This release mostly fixes some small but critical bugs present in the last beta, users running into issues should try out this build. Those coming from v14.6, check the changelogs for all the changes in detail.
The Following 864 Users Say Thank You to topjohnwu For This Useful Post: [ View ]
28th December 2017, 08:49 PM |#35  
OP Recognized Developer / Recognized Contributor
Flag Taipei
Thanks Meter: 44,453
 
Donate to Me
More
2017.12.29 Magisk v15.1
The previous stable release has been put into extensive testing from my testers, and also very thoroughly tested on my side (4 different device setups, 4 Android versions), but still some issues are kept under the radar and went through all the testing......
Please accept my sincere apology! There were some faulty code in magiskboot, which in some cases will cause fstab to be corrupted, leading to bootloops. The issue is now identified, reproduced, fixed, and tested. In addition, with reports and feedback from users, Magisk Manager is also updated with bug fixes and more robust sudb management.

For a more complete release note, please check v15.0 Release Notes

I added a highly requested feature in Magisk Manager: sort repos with last update, hope you will enjoy, forgive me, and please don't hunt me down
The Following 766 Users Say Thank You to topjohnwu For This Useful Post: [ View ]
1st January 2018, 06:35 PM |#36  
OP Recognized Developer / Recognized Contributor
Flag Taipei
Thanks Meter: 44,453
 
Donate to Me
More
2018.1.1 Magisk v15.2
What is a better way to celebrate 2018 than a Magisk update fixing bugs. This update mainly fixes these main issues:

I fixed fstab ramdisk patches in the last release, but forgot to update the dtb part (both uses the same patching logic), my bad. Recently devices are starting to move fstabs from ramdisks into dtbs like OP5/OP5T Oreo beta. Since Magisk's dtb verity removal code is broken in v15.1, this causes devices that triggers dm-verity using the new setup to stuck in a bootloop.
Another major update is that new sepolicy rules are added to support stock Samsung ROMs. Samsung users has been complaining MagiskHide broken since the introduction of the refined MagiskHide system, and finally I spent an excessive amount of time and found the root to the issue: SELinux.
Lastly, thanks to @shakalaca joining the forces, there are some devices in the wild using Treble-like split sepolicy setup on a traditional non skip_initramfs devices, this release is updated to support these Frankenstein configurations.

Sorry for the frequent updates recently, but I really want things to work out on all devices. Hopefully this will be the last update in this series of v15.X release, finger crossed!
For a more complete release note, please check v15.0 Release Notes
The Following 658 Users Say Thank You to topjohnwu For This Useful Post: [ View ]
11th January 2018, 08:28 PM |#37  
OP Recognized Developer / Recognized Contributor
Flag Taipei
Thanks Meter: 44,453
 
Donate to Me
More
2018.1.12 Magisk v15.3
Another bug fix update! This should fix a rather small but pretty serious bug that causes only one script would be executed in bootscripts.

To make things not that boring though, I've bring a small suprise that I think no root management apps has done before (at least I'm not aware of) - Fingerprint Authentication for Superuser requests! Devices using Android 6.0+ with a fingerprint scanner, you can enable this in Magisk Manager's settings.

These bug fix releases on stable channel should come to an end, I'll start adding more experimental features/device support and those will be pushed through the beta channel before it hits the next major v16 release.
  • Magisk v15.3
    - [Daemon] Fix the bug that only one script would be executed in post-fs-data.d/service.d
    - [Daemon] Add MS_SILENT flag when mounting, should fix some devices that cannot mount magisk.img
    - [MagiskBoot] Fix potential segmentation fault when patching ramdisk, should fix some installation failures
  • Magisk Manager v5.4.4
    - Fix dtbo on-boot detection, should follow configured dtbo patching behavior on Pixel 2 devices
    - Add fingerprint authentication for Superuser requests

Lastly for interested developers out there, I've updated the documentations to fully match the latest Magisk v15, please spend some time read through the docs and also the elaborate procedure graphs I've made.

2018.1.13 Hotfix (1531 and v5.5.5):
There is a very minor bug in MagiskBoot, fixed it and updated the zip in the OP.
No notification will be pushed to the stable channel, and users that already have Magisk installed don't need to upgrade.
For Magisk Manager, an update is pushed to fix crashes on Android Lollipop and some devices not following AOSP standards.
Attached Thumbnails
Click image for larger version

Name:	magisk_diagram.jpg
Views:	86362
Size:	229.5 KB
ID:	4386461   Click image for larger version

Name:	fingerprint.png
Views:	86212
Size:	146.8 KB
ID:	4386462  
The Following 619 Users Say Thank You to topjohnwu For This Useful Post: [ View ]
12th February 2018, 11:40 PM |#38  
OP Recognized Developer / Recognized Contributor
Flag Taipei
Thanks Meter: 44,453
 
Donate to Me
More
2018.2.13 Magisk v15.4
Massive MagiskBoot Improvements
A lot more boot images are supported with magiskboot - I would love to say magiskboot could be the most powerful single binary to handle Android boot images! Huge props to @osm0sis with his assistance and his AIK project for making this possible - we have some plans coming

Massive Magisk Manager Optimizations
I've rewritten Chainfire's zipadjust (originally from OpenDelta project) into Java (also fixed a typo in his code :P). This result in the removal of JNI in Magisk Manager, and great simplification in Magisk's building system for signing zips. Also, the portion of dealing with root is migrated to libsu, an Android Library I designed for root app developers, and for an exciting WIP project . I'll officially introduce libsu in a future post very soon, the documentations are not done yet, but developers interested can chime in the code and check out the cool features and designs.
There are also massive improvements in the way Magisk Manager handles the superuser database. It should be much more robust, fixes the bugs people has experienced, and make the repackaging of Magisk Manager (a.k.a hiding Magisk Manager) much more stable and error prone.

Socket Obfuscation
In previous Magisk releases (well, all CM based if not all root solutions), the daemon listens to a specific Unix socket, and requests can be sent via the socket to remotely start a root shell from the daemon and connect STDIN/OUT/ERR to the current terminal/process. So technically speaking, a process can directly talk to the socket instead of calling "su", since what "su" does can be oversimplified as basically establishing a socket connection to the daemon. This means that even if we have MagiskHide enabled, any app can discover the socket entry Magisk uses and detect, or even request root access. Starting from this release, the socket name will be randomly assigned each time the device boots up, so processes with MagiskHide enforced can effectively be blocked from root access.

Magisk Manager v5.6.0 is available through stable channel; Magisk v15.4 is currently in beta, and will soon be pushed to the stable channel once publicly tested.
The Following 564 Users Say Thank You to topjohnwu For This Useful Post: [ View ]