[DEV][19NOV] Native Mount DataOnEXT with DalvikOnNAND (Test #2)

• Post 1 - Android Boot Process, Space Restrictions in /data, Usual Workaround scripts that load from init.d
• Post 2 - The Permanent Solution for Internal space issues in /data, and how it is executed (Step by Step)
• Post 3 - Credits, Benefits, Note on Cache, Note to ROM Makers on ClockWorkMod Updater Script, FAQ
• Post 568 - Test ROM for Native Mount DataOnEXT with DalvikOnNAND
• Post 622 - Test #2 ROM for Native Mount DataOnEXT with DalvikOnNAND

• Have a system partition that is as big as I want in NAND, so that I can install any bloated ROM I want without chopping down anything, and yet have a /data partition for all my apps and their bloated data
• Avoid using internal (NAND) userdata partition completely, so that if the day comes where my /system is full of bad blocks, I will still be able to create a system partition in the ~900+ MB internal NAND my TMoUS HD2 has

• Device - TMoUS HD2
• Boot Loader - cLK 1.5.0.9
• Partitions (in order) and Sizes - Recovery - 10 MB, Boot - 10 MB, System - 500 MB, Userdata - <I didn't bother >, Misc - 1 MB, Cache - 2 MB.
• Recovery - CWM Based Touch Recovery B8
• SD Card - 16GB Transcend Class 6 with 4 GB EXT4 Partition (mounted as /data, of course!)

1. Bootloader – In HD2’s case, Magldr or cLK – loads the kernel based on how you have configured the phone.
   
2. Kernel – The kernel (zImage) is loaded into RAM along with an initial ramdisk (initrd.gz), which initializes various devices (IO, memory, GPU, etc.), interrupts, and mounts the root file system (/). After this, the first user-space process called init is started.
   
3. Init – this is a binary file that is contained within the initrd.gz. The init binary processes init.rc and init..rc , along with other .rc files that are called by these two .rc files. Some of the key functions (from this thread’s perspective) in the order of their initialization/ execution are:
   
   
   1. The init process follows the instructions in the init.rc and init.xyz.rc files and creates empty directories including /data. It then mounts the storage devices (partitions in the internal NAND (MTD)) to these empty directories. The NAND partition for system is mounted to /system, followed by the partitions specified for data, cache, etc. The directories for dalvik-cache (/data/dalvik-cache) are also created by the init process after mounting the specified device to /data.
      
   2. The init process then starts various services including adb, service manager, Volume Daemon (vold) for media like SD Card (FAT partition). Most importantly, the zygote service which initiates the Dalivk-Cache is loaded in this sequence.
      
   3. As we all know, Android is based on Linux. The boot sequence described above is common for all Linux machines – until the zygote stage. Core Android file like core, framework, services, IME, policy, etc. are executed from the Dalvik-Cache and hence Initialization of the Dalvik Cache is pretty much where Android comes into the picture
      
   4. The sysinit/ run-parts part, which runs scripts from the /system/etc/init.d later the Zygote stage. No matter how this is done, Android has already started loading by the time the boot process comes to executing scripts in /system/etc/init.d

ph03n!x said:

All - I just got a great news from @kokotas. Migrating to DataOnEXT has just become extremely easy - wait for it

Click to expand...

Click to collapse

kokotas said:

If you have a normal NAND installation(and an ext2/3/4 partition on your card) and want to give ph03n!x's DataOnEXT method a try,
BUT you think it is difficult THEN you can try the attached recovery package.
It follows all the steps of the described process AND at the end it copies all of your /data to the ext2/3/4 partition. <= This means that after rebooting, you'll have your system set up as before but with DataOnEXT!
And if you want to go back to your previous setup (that is a normal NAND installation without DataOnEXT), just flash the original kernel to your device.

Have in mind that I have only tested it on my phone. If you do try it, please leave a comment. Especially MAGLDR users! Cause it is not tested with MAGLDR.

Thank you ph03n!x for a great concept!

Click to expand...

Click to collapse

The Long Way (or this is how I started off doing this mod )

• Take a NANDROID backup using ClockWorkMod
• Back up all user apps and system app data using Titanium Backup
• Install Cygwin
• Install @dsixda’s Kitchen
• Install Notepad++ (or any other UNIX compatible text editor)

• You need a rom.zip file that has at least these three directories when you open the .zip – META-INF, system, boot (or boot.img)
• If your rom.zip file has a different structure, just extract the three directories alone from it and create a new zip. Essentially, when open the rom.zip you should see only these three directories

1. Back in the Kitchen Menu, select 0 for Advanced Options
   
   
   
   
   
2. Select 20 for Tools for boot image
   
   
   
   
   
3. Hit z for Extract kernel+ramdisk from NAND boot folder (or w for Extract kernel+ramdisk from boot.img)
   
   
   
   
   
4. Now navigate to the BOOT-EXTRACTED directory that got created under the directory where you installed the kitchen. You will see something similar to this:
   
   
   
   
   
5. Edit init.rc and init.htcleo.rc to find all mentions of sd-ext and comment them out with # in the begining. This step is to ensure there is no directory called sd-ext when your phone boots, avoiding any confusions
   
6. In init.rc, look for the on fs part. Under this part there will be a line that reads mount yaffs2 mtd@system /system noatime nodiratime. Below this will be another line that reads mount yaffs2 mtd@userdata /data nosuid nodev noatime nodiratime. Comment this line out with a # (#mount yaffs2 mtd@userdata /data nosuid nodev noatime nodiratime). Also, if you have mount yaffs2 mtd@system /system ro remount, comment that out too (make it #mount yaffs2 mtd@system /system ro remount)
   
7. Do all the following changes AFTER mount yaffs2 mtd@system /system noatime nodiratime, else the process will fail.
   
   
   Code:

   	# Wait to ensure SD Card is available, and scanning file system of EXT
   	wait /dev/block/mmcblk0p2
   	wait /dev/block/mmcblk0p2
   	wait /dev/block/mmcblk0p2
   	wait /dev/block/mmcblk0p2
   	exec /system/xbin/busybox sh -c "/system/bin/e2fsck -f -p -t -t -v /dev/block/mmcblk0p2 > /system/fsck.log"
   	
   # Native Mount DataOnEXT
   	mount ext4 /dev/block/mmcblk0p2 /data wait nosuid nodev noatime nodiratime barrier=0 data=writeback

   Leave everything else in this section as is - you will have another line to mount cache, do not remove or modify it.
   
   Next, scroll down to the section that reads on boot. Add the following like under it:
   
   Code:

   	# SD Card Cache
   	write /sys/devices/virtual/bdi/179:0/read_ahead_kb 2048

8. Build the NAND boot folder (for Magldr yboot) or boot.img (for cLK / Magldr rboot) as per your requirements
   
   
   
   
   
9. Head back to the Working folder, copy the boot directory or the boot.img and include it in your original ROM’s zip file (not the one you created for the kitchen, in case you had to). Check the init.d directory of the ROM’s zip to ensure there is no script that mounts the EXT partition to sd-ext directory, or has a move2sd/ app2sd/ int2sd script – delete such scripts!. Also make sure that your ROM's inatallation does not copy anything to /data during Installation. If it does, move those parts to /system. Or use the instructions I had given under Note to ROM Makers on ClockWorkMod Updater Script to mount the EXT partition as /data in Recovery.
10. Format your phone’s /system, /data, / sd-ext and /boot from clockworkmod and flash this modified rom.zip
11. Let your phone boot into Android and settle down.
12. Configure your Google Account
13. Download and install Titanium Backup
14. In Titanium Backup, restore all apps and their data from the backup you had taken earlier

• I have tested this method with CM7, CM9, CM10 (ParanoidAndroid JB) and Sense ROMs. There has also been feedback that this is working on other AOSP based ROMs like MIUI.
• The mount wait command is available in the init binary from CM7 onwards (not sure if it is there in CM6). If you are not sure (or cannot check the source code for init.c)

• All the Devs who are contributing their valuable time to get Android working on our phones
• Proz0r for implementing something very similar to what I had wanted. He has essentially implemented the same thing, I am implementing this in a slightly different way, that's all. (To be clear, I did not borrow the idea even though a few others have had the same brainwave that I thought I got first - I only borrowed the part where we have the phone wait until the sd-ext partition is ready for being mounted)
• Rick_1995 for the first hint to get this modification to work
• Securecrt for showing that this modification is indeed possible
• Xylograph for pointing me to another thread where this modification is already done
• marco.palumbi - for showing me that I need not have an elaborate script to ensure /dev/block/mmcblk0p2 is available, and a mount... wait option is available in the CM source for init. Also for doing away with the need for extprep script and including the e2fsck routine complete with a fsck.log in init.rc itself.
• kokotas - For the recovery-flashable version of this mod that simplifies migrating to DataOnEXT

• Space - Obvisiouly. The 1GB/ 2GB/ 4GB sd-ext partition will be mounted as internal memory, instead of the usual /data partition from NAND
• Reliability - EXT4 partition is arguably more reliable than YAFFS2
• Working around Bad Blocks - Most of our phones would be having a bunch of bad blocks by now (lucky you, Hynix NAND folks!). With this method, the /data partition is mounted directly to the EXT partition in your SD Card, and the internal userdata NAND partition is untouched. This means you can create bigger system partitions to work your way around bad blocks. Incidently all the bad blocks my phone has is in the userdata partition - this might be so because that is the area that is written to the most.
• Performance - A fast SD Card gives better transfer speeds than NAND (My ParanoidAndroid build is noticeably faster after this modification, with my Class 6 Transcend SD Card with 4GB sd-ext partition)
• Simplicity - Don't have to worry about hoards of the a2sd scripts and their variants, and carry-out trial and error as to which ones work reliably with your ROM of choice. /data is transparently and natively mounted to sd-ext, so no more low-sound or other odd issues
• /data/data can be in the sd-ext partition, and ROM updates will not corrupt it like in some a2sd scripts
• Hassle-free - Whether an EU HD2 or TMoUS HD2, you set a large size (445 MB?!!) for /system and forget about partitioning, as the internal userdata partition is not used at all and can be of whatever size. I am using 300 MB system in both my phones. This is close to native Android phones, where we do not keep changing the system partition size for every ROM that is installed.
• A proper multi-user setup - if you had to share the phone with someone else, all you have to do is swap out the SD Card you usually have in the phone to a fresh one that has the required FAT and EXT partitions. The other user will see a fresh Android install and configure whatever they want. When you get the phone back, all you have to do is put your usual SD Card back in, and voila! you have all your stuff just the way it was (Thanks to @teemoo for the tip!)
• Trying a new ROM? Make sure you do not botch up your data! Just remove the SD Card you usually use and place one for testing. You can always mod the initrd.gz of your new ROM, flash the boot image or place the initrd,gz in /boot and replace your usual SD Card when you want all your apps, data and settings!

1. Setting a cache size in the extprep script
2. Booting from the boot.img using fastboot boot command
3. Waiting for my phone to connect to Google servers
4. Opening Root Explorer and confirming the cache size
5. Running SD Tools, testing the SD Card Read/ Write speeds thrice in a row
6. Changing the cache size in extprep script, recreating a boot,img, and starting from fastboot boot...

package_extract_dir("work", "/tmp");
format("MTD", "system");
format("MTD", "boot");
mount("MTD", "system", "/system");
mount("ext4","/dev/block/mmcblk0p2","/data");
ui_print("Copying files to /system");
package_extract_dir("system", "/system");
...
...
ui_print("Copying files to /data");
package_extract_dir("data", "/data");

A: Mounting as EXT4 is backwards compatible with EXT2/3

mkdir /dalvik 0771 system system
mount yaffs2 mtd@userdata /dalvik nodev nosuid noatime nodiratime
symlink /dalvik /data/dalvik-cache