[KERNEL][MOD][08-03-2012] I/O Boost - Data2SD

K, time to give this a proper OP, if anyone wants any of the info that was here before you can look here

_________________________________________________________________________________________________

So, the whole idea here started with me reading an article on how part of the whole I/O problem with the transformer is partially caused by the hardware used as internal storage. I wanted to find out if this had any merit and I figured the best way to do it would be to "replace" the internal storage. I did this by mounting the /data partition to the exteral SD (which according to my research, my specific SD Card is better at writing speeds - allegedly the main problem with the transformer's internal storage hardware wise). Then I ran a bunch of benchmarks and have been running it that way for about 24 hours and so far it feels great. Anyone is welcome to give it a try, and hopefully with help, suggestions and feedback from the community, we can all take as much advantage of this idea as possible.

Before I go any further I want to give credit to those who helped me so far, because without them I would still be completely clueless, and not only have they helped be accomplishing what I got so far, but thanks to them I've also learned a bunch of things I didn't know before. So here it goes:

Rayman - For suggesting the method for mounting /data to the external SD.
lilstevie - For helping me get the new kernel flashed right.
Turge - For showing me how to properly repack the kernel.
Parastie - For suggesting doing the same thing to /cache (working on that now).
dagrim1 - For SQL patch and for suggesting a temporary remount (even though it didn't work it was a good thing to try).
_motley - For all his work on his awesome kernel.
_________________________________________________________________________________________________

Updates:

Update 1 (08-01-2012 File boot-data+cache+internal-AOKP6.1base.zip) :
Now both /data and /cache are moved to the external SD card. This means you need a third partition mmcblk1p3 in order to use this modification.
It will also mount the internal storage (previously inaccessible) to /mnt/sdcard_internal
It also attempts to (fail at this point) to mount the internal sd partition (what used to be /sdcard) to /sdcard/Internal_SD (which is why you will always see that folder get created but stay empty). If anybody knows how to make it work please advise.
Modified Lines:
init.cardhu.rc

Update 2 (08-01-2012 About backing up/restoring in recovery) : - READ UPDATE 5
One thing that worried me was that by using this mod people wouldn't be able to backup their data partition properly, but now I know that it's possible to do it. It will only work on TWRP though since it has basic terminal access and keyboard. To do it, go into Advanced > Terminal and in there type:

umount /dev/block/mmcblk0p8
mount /dev/block/mmcblk1p2 /data

And until you reboot, any backup/restore should use the external SD data partition instead of the internal. The same should be doable with the cache partition in case you want to backup/restore that.

Update 3 (08-02-2012 File flashme-kernel-motley305-aokp-data+cache2SD.zip) :

Put together a flashable zip that will install motley's 3.0.5 aokp kernel using this mod. Works like a charm so far though I only tried flashing on TWRP. Also, internal storage can be accessed in /data2 and internal sd can be accessed in /sdcardi . Current changes are as follow:


Update 4 (08-03-2012 File boot-cm10-unofficial-data+internal.zip) :

Running the unofficial CM10 (no cherrypicks one) using this mod and so far it's pretty amazing. The rom itself is pretty stable and even snappier with /data mounted to external SD. Benchmarks are at the bottom. Current modifications:

fstab.cardhu:

init.cardhu.rc:

Update 5 (08-03-2012 About backing up/restoring in recovery) :
So after doing some tests, and paying more attention to TWRP, I noticed something rather useful:
When you have this mod enabled, or whenever you have a mmcblk1p2 partiion, TWRP will have the sd-ext menu enabled. This means that to backup your data you can simply backup the sd-ext partition and to restore your data you can simply restore your sd-ext partition. No need to worry about manually switching the mount point for /data in recovery. I guess it was a whole lot easier than I thought.

Also congratulations and thanks to everyone who has contributed with this so far.

WE MADE IT TO FRONT PAGE ON XDA (08-03-2012)
_________________________________________________________________________________________________

Requirements:

There are a few things you will need to do in order for this to work right for you, and a couple of things you'll have to research before you even try it.

#1. Obviously, you have to be rooted/unlocked because you're not gonna be able to change much around otherwise.

#2. You MUST repartition your external SD. The kernel I've put together so far WILL ONLY mount /data to mmcblk1p2, which basically says "mount /data to the second partition in the external SD." also, the ramdisk expects that partition to be ext4, so essentially:
Make sure you have an external SD with at least two partitions and that the second partition is formatted to ext4. I personally use Gparted to repartition my stuff, but feel free to use whatever rocks your boat. Even if you're on windows you can still use gparted by using virtualbox, so I'm not gonna go look for a different windows solution.

#3. This is the research part... This will be beneficial or detrimental to each user depending on the SD card used. If you have a slow SD card this probably will do you no good. However, just because you have a class 10 SD card, that doesn't mean it will benefit you either. On my own research I have found that some class 6-10 SD Cards have extremely slow random write speeds, so if you happen to have one of those, even if it's a class 10, this might not be for you. This means that you're gonna have to do some research to find out if your SD Card will benefit you or not. You can always just give it a try, as far as I know this is entirely reversible, how easy or hard being just a matter of how bad you mess up on meeting the requirements and following the instructions.

#4. At this point (07-30-2012) I'm doing all this stuff using the AOKP milestone 6.1 kernel as base for my modified kernel, so if you're not using AOKP milestone 6.1, flashing my kernel might borke your system. You've been warned, feel free to proceed otherwise at your own peril.
_____________________________________________________________________________________________________________

Installation:

#1. Download attached file (boot-data2SD-AOKP6.1base.zip) and extract it to the root of your internal storage (/sdcard).

#2. Open a terminal.

#3. Type the following:

#4. Wait for it to complete.

#5. Reboot.

Upon rebooting you will know that it worked because it will look just as if you just flashed a new rom, that is, you'll get the device setup screen (assuming that the tablet booted at all lol). If you're planning to use TB to restore your apps, you'll probably want to copy the TB folder to your external SD's first partition so that you can copy it back once you're done with the device setup (at this point you will have no access - unless you manually mount it - to your internal storage).
_______________________________________________________________________________________

Reverting:

Follow the same exact steps for installation but use boot-default-AOKP6.1.zip instead.
_______________________________________________________________________________________

Optional:

#1. If you want to have access to everything you had on your data partition in the new data partition, you'll have to clone everything from one to the other. To do this, make sure that your new data partition (the one in your external SD) has enough storage space to fit everything you currently have in your data partition (the one in your internal storage). Then run the following command in your terminal.


BEWARE that if you have a lot of stuff this can take quite a while and even though I've read a way of getting the progress for this in Linux I'm not sure that you can check the progress on Android.
_______________________________________________________________________________________

Next steps in development:

#1. Move /cache as well.
#2. Find out what happens with recovery backups when the partitions are changed.
#3. Attempt to apply mod to motley's kernel.
#4. Create a script that is run on boot to eliminate need for replacing the kernel.
#5. With help from the community, find the best SD Card for this.
#6. Run the modified system for a while to have a good feel for performance benefits
#7. Come up with other interesting uses for this other than getting better I/O (maybe an easy - kinda easy - way to dual boot with ubuntu, maybe other stuff, dunno).
_______________________________________________________________________________________

How can I set my kernel to do this?

I didn't do a whole lot, and it's not like I want it to be a secret, so as I modify things I'll try to keep the steps listed here so that anyone modifying their own kernel who would like to try this modification can go ahead and do it.

You'll need to know how to unpack/repack a kernel. Turge has a SUPER EASY explanation here on how to do it on windows (I'll pack together the necessary binaries for linux later, maybe).

Mount /data to the second partition in your external SD (formatted as ext4 filesystem):
After unpacking the kernel navigate to the folder that has the ramdisk and open it
(DON'T USE ANY ASCII BASED TEXT EDITOR BECAUSE IT WILL PROBABLY MESS THINGS UP I USE NOTEPAD++)
Around line 26 change:
to
_______________________________________________________________________________________________________

SD Cards tested:

Samsung 32GB Class 10 MicroSDHC High Speed Memory Card - Very Good Results

SanDisk® microSDHCTM 8GB Memory Card - Very Good Results
"Either way, with this mod, the tablet feels like it should have right from the start. It's speedy and responsive, and apps being installed don't stall the system." - Turge - Post #59
_______________________________________________________________________________________________________

Benchmarks:

/data mounted to mmcblk0p8 (Internal Storage):






/data mounted to mmcblk1p2 (External SD):






RL Benchmark WITHOUT dagrim1's sql patch as per request:

/data mounted to mmcblk0p8 (Internal Storage):


/data mounted to mmcblk1p2 (External SD):


*Important thing to note: When running the benchmark with data in the internal SD it was about 220 seconds in the first run, then about 180, then about 130 and finally 118; whereas running it from the external SD was consistenly between 63 and 65 seconds every time. I think this more than proves that A) Asus used a cheap I/O storage, B) No matter what software changes are made, more than likely running the rw partitions from a better I/O storage, i.e. an external SD is a good idea.

As promised here are benchmarks on a JB rom (Unofficial CM10). Also sorry it took me a while to get these, I was going to use eos3 but I started getting random reboots. Then I decided to try cm10, and I messed up a flash and had to redo a bunch of things. Anyway, the only change here is the mod itself (no custom kernel or anything). Though one thing to note is that I moved /cache back to the internal partition after some thought that this allows /data and /cache to be written at the same time to different locations thus lowering the bottleneck.

/data mounted to mmcblk0p8 (internal storage):


/data mounted to mmcblk1p2 (external storage):


Now, as you can see, JB did bring a major improvement to I/O, bringing the benchmark down from about 115sec to 68 (almost reaching the modded ICS at 60 seconds). But as I expected, better software works better on better hardware and now the modded JB is running at 50 seconds instead of 60. Next I'm going to put dagrim1's sql patch and see how low the benchmark goes. Also will be posting the modded blob in just a little bit for anyone who wants to use it on CM10.

I'm using External SD with data2sd and ext4 partition.



To Disable Journaling in EXT4

Go into TWRP and then open a command prompt on your computer where adb is.
Type adb shell and enter it then type the following commands one at a time.

umount /data
umount /sdcard
e2fsck -f /dev/block/mmcblk1p2
tune2fs -O ^has_journal -c 1 -i 1d -m 0 /dev/block/mmcblk1p2
e2fsck -f /dev/block/mmcblk1p2

Watch out for the fourth line, after tune2fs - that is a capital O not a zero. The one further along the line is a zero.

To check if it has worked type the follwing after them

tune2fs -l /dev/block/mmcblk1p2 | grep features

If you don't see has_journal in the output all is good.

Here's a modified boot.blob from EOS nightly 35.

The only changes are in /fstab.cardhu:

• mounts 2nd partition of SD card to /data
• mount old data (/dev/block/mmcblk0p8) as /data1

and /init.rc:

Turge said:

I've repacked the boot.img for the Prime before to add init.d support so I'll post my method and the files needed in a few minutes once I get to work. It'll involve getting cygwin installed (with Perl support I believe) if you're on Windows.

Click to expand...

Click to collapse

Here are the steps for repacking the boot.img. Some involve running the commands via cygwin, others involve running them via the Windows Command Prompt.

The instructions for installing cygwin, extracting and repacking the boot.img were found here: http://www.freeyourandroid.com/guide/extract-edit-repack-boot-img-windows

Once you have setup cygwin, extract the attached files in a folder under your "home" folder in cygwin.

1. copy boot.blob to the same folder and run the following via the Windows Command Prompt to extract the boot.img from the boot.blob:
   
   
   Code:

   BlobUnpack.exe boot.blob
   ren boot.blob.LNX boot.img

2. From the cygwin bash terminal window, switch to the same folder and run the following to extract the ramdisk from the boot.img:
   
   
   Code:

   ./extractboot boot.img

   You now have an out/ramdisk folder that contains the files you want to edit.
   
3. Once done, repack the ramdisk and kernel into boot_new.img with the following command (via cygwin once again):
   
   
   Code:

   ./packboot

4. then from the Command Prompt repack boot_new.img into boot2.blob using the following:
   
   
   Code:

   blobpack -s boot2.blob LNX boot_new.img

5. You can now flash the boot.blob to the staging partition via a command in updater-script:
   
   
   Code:

   package_extract_file("/boot.blob", "/dev/block/mmcblk0p4");

   or by using adb while in recovery/android:
   
   
   Code:

   dd if=/sdcard/boot2.blob of=/dev/block/mmcblk0p4

Hi all. Attached is a boot.blob for Data2SD on Hairybean 1.2. Flash via fastboot.

(This is actually a blob, not an img - XDA won't accept a .blob file, so I renamed it to attach to this post. It shouldn't matter what it's called for flashing purposes though)