It's been almost three months that I started the development of K3-Note, challenging and funny. The last few minor issues are also fixed now. Since all the major issues are fixed, it's time to move on to other devices. I might take another look when I find something or when new Android arrive.
As a promise at the beginning, I built another custom ROM -- SlimRom. With the Lineage's setup, it only took me a hour to prepare and patch the platform. After the ROM was out, I polished a few things. I built the CarbonROM too. I also have DU in my hard drive but DU-team had forbidden releasing in xda earlier. My 1T hard drive is almost full and there is no room for another ROM. If anyone interest in taking over any of these, please let me know and I might guide you through. The requirement is simple -- you need to share your work too.
The objective of this project is not releasing a particular ROM. With the solutions to the issues, I hope it would enlighten more people to join our development.
After releasing the kernel, I'm glad there are new kernels come. I was hoping many custom ROM would follow. Sadly, seems Lenovo and Mediatek are not friendly to custom ROM developers. I don't think it's intentional. The changes in the AOSP source by MTK and each manufacturer would likely be the reason. Even if we have the full source, it would probably for LP or MM. Making the proprietary blobs to work for Nougat is not a simple task. I think that's the reason why devices with MTK chips haven't received Nougat updates yet. Most of the manufacturer might be waiting for the update from MTK too.
When I finished the job in hand earlier last month, I decided to try it myself. I already had the Carbon CR-5.1 (Android 7.1.1) source ready so I started with it first. Building it out and made it boot was not too difficult. With a few attempts, my first custom ROM for k3-note was out. Some of the features were not working as expected. RILD not started, no audio and GPS, camera crash, etc... RIL logs are disabled by default so I started fixing the camera which I thought would be easier. That was proved over optimistic soon. My experience told me CM always one step ahead so I downloaded the source when I stated. It's time switching to CM. In a week or two, I finally built a barely presentable ROM. There were questions from a few guys from time to time. All I heard were camera not a problem, audio not a problem, etc... I'm new to K3-note and I just can't imagine how "easy". I don't think I'm that good. I also don't have the patience and experience to try all the blobs. Therefore I upload the test builds for those guys to try. Sadly, up to this moment, no respond yet.
Although there is no complete source from Lenovo, other devices might provide something here and there. Fortunately, only a few pieces are required. The source we can found are for LP and for other devices. Make if work for us would be quite a challenge. I finally managed to fix a few things. I seldom release anything that early but there are just so many requests. I don't think I have enough time to make everything work soon. Seeking for other opinions is never a bad move.
I'm now sharing my experiences again. You are welcome to use and share my works. All I asked is "don't keep it to yourself, share yours too!"
If possible, I'll build a few different ROMs later. However, there won't be any weekly or monthly in schedule. This thread is more or less a stepping stone.
At this stage, report is not essential. You don't have to repeat what already mentioned. Thanks!
I. Missing symbols
To deal with missing smybols (usually name space changed in platform upgrade), we may use a shim library and assigned it to LD_PRELOAD at init. GCC doesn't care who provide it. As long as it was loaded, others can access it. The up side is, this way would not touch the original source. The down side is it would be loaded/checked whenever a dlopen() is called. Clearly, it would increase overheads and affect the performance. For our K3-Note, there are also some missing symbols in blobs.
I personally don't like LD_PRELOAD libraries. I would usually add a build flag to Android.mk and add a new file if possible.
**In CM14.1, it already added the build flag BOARD_USES_MTK_HARDWARE in vendor_cm.
Set it to true would define a global flag MTK_HARDWARE.**
The patches are grouped in the device tree. At this moment, there are 4 group of patches:-
1.1 liblog (add the xlog function __xlog_buf_printf() and override the RLOG, see following section)
1.2 libnetutils (required by thermal and RIL). Can use the blob from stock ROM.
2.1 a file mtkaudio_stubs.cpp was added
2.2 missing CameraParameteres
2.3 add extra profiles to AudioPolicy (2016/12/17)
2.4 fix audio recording in AudioFlinger (2016/12/17)
3.1 Add a few symbols for backward compatible with MM calls.
4. sepolicy. - not compulsory
Mediatek never takes sepolicy seriously. They use permissive all the time. I planed to use enforcing eventually so I dealt with all the sepolicy from day one.
II. The RIL
Tool: To find out what's wrong, the logs are essential. I had made a patch to liblog. A new property "persist.ril.log" is added. Set it to 1 will push all RLOGs to main.
1. A missing symbol in mtkrild
With this help, I found a missing symbol in mtkrild. This symbol is in libnetutils.so. You may either use the file in stock ROM as a blob or rebuilt it.
2. The ccc_util
During debugging, I noticed there were garbage in the ccc_util logs. A source can be found for mt6595. After disassembled the binary blob, I found a few differences. I rewrote some part and added support for the two new properties. The final sources are grouped in the new project android_hardware_lenovo_aio_otfp
3. Default modem selection
There are two modem binaries in stock ROMs -- 5 (LWG) and 6 (LTG). They are in /system/etc/firware. In stock ROMs, the default modem is read from nvram. For some unknown reason (seems protection issue), ccci_mdinit can't get this value from nvram in custom ROMs. It would use the default value in Kernel (was 6). I noticed LWG was shown in About of stock ROM so I changed the kernel default to 5. Other areas might required this to be 6. Check the Modem in About for your area. If it is LTG, you may either swap the names of the two modems or change the kernel default. I wanted to use a build.prop but turn out that ccci_dminit is a blob from Mediatek. There are only binaries available. No way to add anything.
4. Custom RIL Class
This class is to handle MTK specific modem messages. Usually its for emergency dialling and modem info. It is not handled in AOSP. CM14.1 already taken care of it and that's one of the reason I switched to CM for testing.
A Custom class can be found in CM's device tree for mt6735. I think its too much to handle the emergency number list but don't want to waste time rewriting it. With a few modifications, it can work for us in Nougat. Seems there are unused/unhandled messages. **Need to verify it later.**
III. The Camera
After my first ROM was out, the most eye catching in the log would be the stack corruption in libcam.halsensor.so.
11-24 21:53:55.967 490 490 F DEBUG : #08 pc 00017459 /system/lib/libcam.halsensor.so (_ZN12ImgSensorDrv13getResolutionEPP34ACDK_SENSOR_RESOLUTION_INFO_STRUCT+60)
After some digging, I was pretty sure it was caused by the alignment (address sanitize) of struct addresses. The latest blob from Lenovo dev.1631 (also the latest for mt6752 devices) were built with gcc 4.9 Clang 3.6. Nougat now use Clang 3.8 by default. Those blob already have alignment support in Clang 3.6 might have issues with the new Clang 3.8.
What is address alignment? In short, a 32 bit processor can handle 32 bit at a time. It would be a easier for the register to locate memories in 32 bits (4 bytes) cells (aligned). If the head or tail for a block of memory is not aligned, extra code would required to separately handle those loose ends. The new clang compiler would fill leading bytes with its own algorithm to make the pointer or variable aligned. Why is ImgSensorDrv::getResolution() being "affected"? The argument of it is a pointer to ACDK_SENSOR_RESOLUTION_INFO_STRUCT. Its definition can be found in the kernel. It is a struct of an array of pointers to another struct. The source from Mediatek use direct reference to the address of struct variables all the time. This is inappropriate in sense of address alignment. Early compilers didn't handle alignments and won't have problem with that. Newer compiler would use its own algorithm to align the structs and pointers. I guess Clang 3.6 didn't handle the above complex struct as argument well. Nougat linker and loader are all built with Clang 3.8. Local variables are allocated at load time which might be handled differently. Thus a call to ImgSensorDrv::getResolution() would cause the stack corruption.
** Those interested may read this article for detail. **
1. Work around
Using older blobs built by gcc 4.8 or earlier might work. Jiayu S3's AOSP were using those old blobs. I'm new to K3-note. I don't think I have the patience and experience to test all the existing blobs. That's the main reason I uploaded the last two test ROM. Trying blobs and hoping some combination would work is passive. Stlport is also obsolete for some time now. I rather deal with the problem directly.
2. The solution
Without the source, building the whole blob is impossible. Even if we have the source from other devices, those custom parameters are unique to each image sensor. It's impossible to guess or back engineer...
If we can change the way ImgSensorDrv::getResolution() is being called or didn't touch the argument within the function. It would avoid the crash. Unfortunately, the function need to fill a local struct within the ImgSensorDrv class. That means it is not seen outside ImgSensorDrv. The only way is to replace the whole ImgSensorDrv class. Fortunately, it is self contained. We can write a wrapper to replace the ImgSensorDrv class only. The source from mt6735 or mt6595 are a bit different and obviously can't be used directly. Most of the struct can be cross referenced to the kernel source. There are two local sturct left unknown. One is for the commands sent to the kernel. The other is some info. collected from the kernel.
With the wrapper, the crash was gone but still no preview. I wrote a test HAL to dump the values in stock ROM. Guessing the values against the names was not a decent job. Fortunately, there were only a few differences. After nights of trial and error, I finally pulled everything together. Camera is working now! Video recording is still not working. Seems codec related. Some say it's not a problem "again". Hoping there will be some real solutions and helps soon...
TODO: HDR crashed
While comparing my log to the stock's, I found all daemons are working properly. It must be the problem of the interface. The java interface of GPS was changed in Nougat. Seems there were transitions from GPS struct to GNSS struct. I accidentally noticed a line in the thousands lines of log.
gps_mtk : copy_GpsCallbacks_mtk: line = 4124Bad callback, size: 96, expected: 88 or 80
The best way is naturally to rebuild the GPS HAL. The sources I had were outdated. After another intensive search, I found the source for mt6795 from xiaomi and another source for mt6753 from ulefone. They were almost identical and matched all the symbols in the blob from Vibeui 3.5 1631. Their contents were quite standard GPS HAL but a lot had to patch for Nougat. Mediatek already handled some transitions to GNSS which made things a bit easier. The last piece left was the prebuilt libepos. There was no 64 bit library from xioami but there were a few from other devices. Nougat already have the open source curl in extension so everything seemed ready. In a few attempts, I managed to build the new HAL out. I also include Mediatek's apk YGPS for testing. Another reason is the uniqueness of GPS from MTK. Common GPS utilities like GPS Status doesn't work. Anyway, GPS is working now! Cheers!
update (2017/2/17): I was using a mt6795 source from github. Turn out that it was already modified. There were something missing. I rewrote the HAL from the official mt6797 source. Now it is fully support GPS, GLONASS and BEIDOU. It is also backward compatible with the AOSP callbacks. Most of the GPS apks (eg. GPS status which was not working in stock MM) would be working now.
V. Audio recording
After checking with a few test apks, I'm pretty sure the issue was the audio recording solely not Mic. related. Actually, I already fixed it for some time but I didn't know. I was mislead by the pause feature in CM's SoundRecorder apk. That pause feature is unique to CM. MTK's (probably some others) HAL didn't handle it at all. Whenever I pressed the pause, it would call the missing function in HAL and hang. I wrote a test apk myself and found that it worked! I then looked into the source of AOSP. That cleared things up immediately. I don't want to modify CM's apk so I replace it with the one from AOSP.
The definition of the struct audio_stream_in has changed in Nougat. The function get_capture_position() was added. It's just for reporting the time stamp. The audio HAL blob from MTK didn't have this function. When the input streams were loaded, it would crash. I reinstated the untouched BOARD_USES_LEGACY_MTK_AV_BLOB flag and made a patch in AudioFlinger service. I added the specific header "audio.h" in device tree. Check the source on my github for detal.
One less thing on the list now! Cheers!
VI. MTK Codec
Codecs are essential for recording and playback. Although we can use the software codec from CM or Google for playback, video recording normally required the codec from the manufacturer. That's why I spent a lot of time to make them work first. Samsung and Qcom usually put the codecs in kernel drivers but Mediatek didn't. MTK's codec use the gralloc_ext, ui_ext and gui_ext helper libraries to "assist" the allocations and queries. Unfortunately, platform changes arose some issues in these blobs.
Naturally, to make it fully support Nougat, we have to rebuild it. I found the sources for mt6797. They are small but have some dependencies. Only minor changes would make them Nougat compliant. Both depend on the dpframework and gralloc.mt6752 which are proprietary. I already rebuilt gralloc_ext from source. Proved to be the right move now. I grouped them into the graphics folder in the hardware_lenovo_aio_otfp project. With these newly built libraries, all codecs are working now.
update (2017/3/18): Found the latest codecs for mt6750t. They work for YouTube, Snapchat, etc. Check the attachment in post #272.
TODO: WVM not handled
VII. Video Recording
For my Samsung devices and Tilapia, I can make the cameraservice work on both camera and camcorder. I don't need the HAL1 flag reverting to the mediaserver. After fixing the extra allocation libraries for my K3-note, I could record the video to a mp4 file. Audio was normal but video wasn't. I had studied for a while. I thought it was colour format related but I don't know why yet. There would be too much changes to the original source so I decided to put aside first.
Although it's not my wish, I compromised. Everything went well when I use the HAL1 flag switching back to mediaserver. It's time to move on to the WifiHAL now. Would return to this part later. Anyway, it's kinda fixed now!
TODO: Reinstate Cameraserver
VIII. WifiHAL for Nougat
WifiHAL has been introduced for a few generations. It provides another way to access the WiFi in the frameworks. MediaTek's WifiHAL was clearly derived from Broadcom's source. LP and MM only used a small portion of the callbacks. Nougat uses a few more and some of them haven't supported in the kernel driver yet. I used a patched Brodcom's WifiHAL in my earlier builds. It could only be loaded to pass the check. Things like hotspot, 5G, hidden ssid, etc. were not handled. I then ported the sources from mt6795 and mt6797 to my early WiFiHAL. It took me a few days to handle most of the Wifi commands and queries. Together with the minor patch in netd, hotspot is working. An immediate issue arose -- WiFi wouldn't auto reconnect. It required to turn on manually. Hidden ssid was still not working. When a network was manually added, Wifi would fail to scan for APs. Clearly it was another major problem I overlooked...
A long road to the solution
I enabled the logs in all the relevant java and c files. Nothing seemed special pop up. I then added my own extra logs. An odd scenario appeared on the channel list. I traced back to the callee in libwifi-service. It was a query message for the available channel lists for 2.4G, 5G and 5G DFS bands. However, the lists to all bands were the same. There were 3 channels all started with 2412 (which is the 1st channel of 2.4G band). The default log in GetChannelListCommand only dump the first entry so I add a for loop to dump all. I got 2412,2413 and 2414! That was definitely impossible. I realized there must be something wrong with this vendor data. Clearly the firmware and proprietary blobs won't be the problem. There were also nothing similar in the nvram. I then traced back to its source -- the kernel driver. Following the defconfig and Makefiles, I found the handler for this message in drivers/misc/mediatek/connectivity/conn_soc/drv_wlan/mt_wifi/wlan/os/linux/gl_cfg80211.c.
Looking at the function mtk_cfg80211_vendor_get_channel_list() made me cry and laugh...
kalMemZero(channels, sizeof(wifi_channel) * 4); /*rStatus = kalIoctl(prGlueInfo, wlanoidQueryStatistics, &channel, sizeof(channel), TRUE, TRUE, TRUE, FALSE, &u4BufLen); */ /* only for test */ num_channel = 3; channels = 2412; channels = 2413; channels = 2414; NLA_PUT_U32(skb, GSCAN_ATTRIBUTE_NUM_CHANNELS, num_channel); NLA_PUT(skb, GSCAN_ATTRIBUTE_CHANNEL_LIST, (sizeof(wifi_channel) * num_channel), channels);
The next and more annoyed issue was Hidden SSID. I'm pretty sure the WifiHAL was OK so I looked into the vendor lib lib_driver_cmd_mt66xx. Enabled a few MTK flags and started with a very limited code all resulted the same. Finally, the following line in log caught my attention:
E wpa_supplicant: wlan0: nl80211: Scan trigger failed: ret=-22 (Invalid argument)
TODO: p2p and WiFi-display not check
VIII. FM radio
update (2017/2/27): After taking another look, I found that the FM radio was already fixed. Probably after the audio recording fix. So everything is more or less working now. Cheers!
- With no doublt the LineageOS community should have most of the credits
- The CarbonROM, OmniROM, SlimROM and many other custom ROM teams who bring us the alternatives.
- Those who release the sources for MTK devices.
- And finally Lenovo who had released a castrated kernel source which motivate us to DIY.
Download: AFH LineageOS, AFH SlimRoms, AFH CarbonROM
(百度网盘)LineageOS , (百度网盘)SlimRoms, (百度网盘)CarbonROM
Check the patch for data access in SlimROM at Post #166
** backup your current /boot, /data and /system partition first **
TWRP 3.0.x is the only recovery tested. There are many TWRP using different device names out there. I already included a few but there would be more. If you got error (6) at flashing, you may change the line in updater-script or use different TWRP.
1. Do a factory reset (wipe /data and /cache)Step 1. is essential if you are from other ROMs, otherwise you might get ERROR:7.
2. Flash the ROM
3. Flash the Gapps
If you want root, you may flash SuperSU at this point
Reboot and hoping its not a night mare...
** Note: you don't need to wipe other partitions. Some important device specific data are in nvram and misc. **
2017/3/12 - SlimRoms build 0.15- The latest SlimRom 0.15 source, Android-7.1.1_r26 (NOF27C) - All patches/fixes in my 20170312 LineageOS-14.1 - Add long press AppSwitch (Recent) key for menu2017/3/12 - LineageOS 14.1- The latest LineageOS-14.1 source, Android-7.1.1_r25 (NOF27B) - The latest GPS HAL from CarbonROM - FMRadio: Fixed - Media Profiles: update to support common video formats. - Limit background processes to increase available RAM - Camera app: replaced with Snap by Lineage - Lineage is not ROOTED by default now2017/2/18 - CarbonRom CR-5.1- The latest Carbon CR-5.1 source, Android-7.1.1_r21 (NOF26W) - Initial beta release - New GPS HAL (fully working and faster) - Add long press AppSwitch (Recent) key for menu - All fixes from LineageOS-14.12017/2/14 - SlimRoms build 0.11- The latest SlimRom 0.11 source, Android-7.1.1_r21 (NOF26W) - Initial alpha test build 0.11 - All fixes from LineageOS-14.1 - Add prebuilt Chromium (from CarbonROM)2017/2/14 - LineageOS 14.1- The latest LineageOS-14.1 source, Android-7.1.1_r9 (NMF26V) - Libstagefright: Add fixes to media streaming (Don't know if it work or not) - LiveDisplay: Add support and settings - RIL: Handle a few more MTK specific messages (WIP) - GPS: Add backwards support for MM and earlier - GPS: Fill up the extra fields in GNSS struct - GPS: Add a override property to select GPS mode (GNSS/GPS) (Testing only) - Fixed Power off charging screen - kpoc_charger: keep the stock images (which are in the LOGO partition) - Fixed LED notifications - liblight: Rewrite the lights HAL for better support in Nougats and beyond (WIP) - liblight: Add dual color support (WIP)2017/1/17 - LineageOS 14.1- The latest LineageOS-14.1 source, Android-7.1.1_r9 (NMF26V) - RIL: Add a fix in custom RIL class (thx. @Moyster for the tip), don't know if it helps. - hostap: build from source - netd: a minor patch (wlan0 to ap0) for hotspot - WifiHAL: Rewrite a new one (almost from scratch...) - WifiHAL: Handle extra callbacks for Nougat - Kernel: Handle the mtk_cfg80211_vendor_get_channel_list() correctly - Kernel: Enabled 5G by default - wpa_supplicant: Fix a potential bug (which fixed MTK's hidden SSID issue)2017/1/5 - LineageOS 14.1- The latest LineageOS-14.1 source, Android-7.1.1_r6 (NMF26Q) - RIL: use the full set of blobs from S433 (newer than dev. 1631) - MTKCodec: Fix the storage issue in MTK Codec - MTKCodec: Include the complete set of MTK codec in dev. 1631 - media_profiles.xml: Include the vorbis decoder - libion: Rebuilt with open source libion - VideoRecording: Fix video recording - SEPolicy: update for codec and media server - kpoc_charger: rebuilt kpoc_charger from source [WIP]2016/12/17 - CM14.1- The latest CM-14.1 source, Android-7.1.1_r6 (NMF26Q) - Camera: Fix the flashlight in taking picture - AudioRecording:Fix audio recording - SoundRecorder: Replaced with the one from AOSP - SEPolicy: update for 7.1.1_r6 - media_profiles.xml: Rewrote with the values in stock log - AudioPolicy: Add extra profiles - OMX: Fix the crash of nativeHandle in video recording (WIP) - libhealthd: fix a few things (WIP) - WifiHAL: Rewrote a new one for Nougat (WIP)2016/12/11 - CM14.1- The latest CM-14.1 source, Android-7.1.1_r4 (NMF26O) - init.mt6752.rc: A few patches - gps.mt6752.so: Rebuilt the GPS HAL from source - libcurl.so: Use opensource curl - YGPS: Include MTK's test apk for GPS - SEPolicy: Further patches - Built-in VPN is working now - healthd: Add standard charging routines to (WIP)2016/12/6 - CM14.1- The latest DC-MTK kernel - Kernel: Enable UID CPU time - Kernel: Emulate the hardware random sysfs - Kernel: Treat all A/C chargers the same (same charging current and limits) - Kernel: rename led sysfs to AOSP defaults - Kernel: set default modem to 5 - Handle all extra SEPolicy - Fix the audio output - Add the RLOG override and xlog suppress properties - Fix RIL and mobile data - Fix all sensors - Add libhealthd.mt6752 to report battery status - Fix the camera
- Camera HDR mode (It use the Lenovo/MTK algorithm which is fuzzy and useless. Not worth to waste time on this.) - may be more...
Nougat and beyond?, ROM for the Lenovo K3 Note
Source Code: https://github.com/danielhk
ROM OS Version: 7.x Nougat
ROM Kernel: Linux 3.10.x
Based On: AOSP, LineageOS, SlimRoms, CarbonROM
Beta Release Date: 2017-03-12
Last Updated 2017-03-18