5,600,919 Members 41,147 Now Online
XDA Developers Android and Mobile Development Forum

[KERNEL][MOD][08-03-2012] I/O Boost - Data2SD

Tip us?
 
TweakerL
Old
(Last edited by TweakerL; 4th August 2012 at 12:03 PM.) Reason: Added more info
#1  
Senior Member - OP
Thanks Meter 309
Posts: 216
Join Date: Oct 2010
Location: Rosedale, MD

 
DONATE TO ME
Lightbulb [KERNEL][MOD][08-03-2012] I/O Boost - Data2SD

K, time to give this a proper OP, if anyone wants any of the info that was here before you can look here

__________________________________________________ _______________________________________________

So, the whole idea here started with me reading an article on how part of the whole I/O problem with the transformer is partially caused by the hardware used as internal storage. I wanted to find out if this had any merit and I figured the best way to do it would be to "replace" the internal storage. I did this by mounting the /data partition to the exteral SD (which according to my research, my specific SD Card is better at writing speeds - allegedly the main problem with the transformer's internal storage hardware wise). Then I ran a bunch of benchmarks and have been running it that way for about 24 hours and so far it feels great. Anyone is welcome to give it a try, and hopefully with help, suggestions and feedback from the community, we can all take as much advantage of this idea as possible.

Before I go any further I want to give credit to those who helped me so far, because without them I would still be completely clueless, and not only have they helped be accomplishing what I got so far, but thanks to them I've also learned a bunch of things I didn't know before. So here it goes:

Rayman - For suggesting the method for mounting /data to the external SD.
lilstevie - For helping me get the new kernel flashed right.
Turge - For showing me how to properly repack the kernel.
Parastie - For suggesting doing the same thing to /cache (working on that now).
dagrim1 - For SQL patch and for suggesting a temporary remount (even though it didn't work it was a good thing to try).
_motley - For all his work on his awesome kernel.
__________________________________________________ _______________________________________________

Updates:

Update 1 (08-01-2012 File boot-data+cache+internal-AOKP6.1base.zip) :
Now both /data and /cache are moved to the external SD card. This means you need a third partition mmcblk1p3 in order to use this modification.
It will also mount the internal storage (previously inaccessible) to /mnt/sdcard_internal
It also attempts to (fail at this point) to mount the internal sd partition (what used to be /sdcard) to /sdcard/Internal_SD (which is why you will always see that folder get created but stay empty). If anybody knows how to make it work please advise.
Modified Lines:
init.cardhu.rc
Code:
        # TweakerL MOD > original mount = mmcblk0p8 /data | mmcblk0p3 /cache 
        mount ext4 /dev/block/mmcblk1p2 /data wait noatime nodiratime nosuid nodev nodelalloc,errors=panic
        mount ext4 /dev/block/mmcblk1p3 /cache wait noatime nodiratime nosuid nodev nodelalloc,errors=panic
	# TweakerL MOD > added mounts for internal storage
	mkdir /mnt/sdcard_internal 0000 system system
	mount ext4 /dev/block/mmcblk0p8 /mnt/sdcard_internal wait noatime nodiratime nosuid nodev nodelalloc,errors=panic
	# TweakerL MOD > give access to internal SD from /sdcard
	mkdir /data/media/Internal_SD 0755 media_rw media_rw	
	mount /mnt/sdcard_internal/media /data/media/Internal_SD wait noatime nodiratime nosuid nodev nodelalloc,errors=panic
Update 2 (08-01-2012 About backing up/restoring in recovery) : - READ UPDATE 5
One thing that worried me was that by using this mod people wouldn't be able to backup their data partition properly, but now I know that it's possible to do it. It will only work on TWRP though since it has basic terminal access and keyboard. To do it, go into Advanced > Terminal and in there type:

umount /dev/block/mmcblk0p8
mount /dev/block/mmcblk1p2 /data

And until you reboot, any backup/restore should use the external SD data partition instead of the internal. The same should be doable with the cache partition in case you want to backup/restore that.

Update 3 (08-02-2012 File flashme-kernel-motley305-aokp-data+cache2SD.zip) :

Put together a flashable zip that will install motley's 3.0.5 aokp kernel using this mod. Works like a charm so far though I only tried flashing on TWRP. Also, internal storage can be accessed in /data2 and internal sd can be accessed in /sdcardi . Current changes are as follow:

Code:
        # TweakerL MOD > move /data and /cache to external SD card || original mount = mmcblk0p8 /data | mmcblk0p3 /cache 
        mount ext4 /dev/block/mmcblk1p2 /data wait noatime nodiratime nosuid nodev nodelalloc,errors=panic
        mount ext4 /dev/block/mmcblk1p3 /cache wait noatime nodiratime nosuid nodev nodelalloc,errors=panic
	# TweakerL MOD > create mount for internal storage
	mkdir /data2 0000 system system
	mount ext4 /dev/block/mmcblk0p8 /data2 wait noatime nodiratime nosuid nodev nodelalloc,errors=panic
	# TweakerL MOD > symlink internal sd to a couple of easily accessible locations
	symlink /data2/media /mnt/sdcard_internal
	symlink /data2/media /sdcardi
Update 4 (08-03-2012 File boot-cm10-unofficial-data+internal.zip) :

Running the unofficial CM10 (no cherrypicks one) using this mod and so far it's pretty amazing. The rom itself is pretty stable and even snappier with /data mounted to external SD. Benchmarks are at the bottom. Current modifications:

fstab.cardhu:
Code:
#TweakerL MOD > Move /data to external SD and internal /data to /data2
/dev/block/mmcblk1p2           /data               ext4      rw,nosuid,nodev,noatime,user_xattr,acl,barrier=1,journal_async_commit,nodelalloc,data=writeback    wait
/dev/block/mmcblk0p8           /data2              ext4      rw,nosuid,nodev,noatime,user_xattr,acl,barrier=1,journal_async_commit,nodelalloc,data=writeback    wait
init.cardhu.rc:
Code:
        # TweakerL MOD > create mount for internal storage
	mkdir /data2 0000 system system
        mount_all /fstab.cardhu
	# TweakerL MOD > symlink internal sd to a couple of easily accessible locations
	symlink /data2/media /mnt/sdcard_internal
	symlink /data2/media /sdcardi
Update 5 (08-03-2012 About backing up/restoring in recovery) :
So after doing some tests, and paying more attention to TWRP, I noticed something rather useful:
When you have this mod enabled, or whenever you have a mmcblk1p2 partiion, TWRP will have the sd-ext menu enabled. This means that to backup your data you can simply backup the sd-ext partition and to restore your data you can simply restore your sd-ext partition. No need to worry about manually switching the mount point for /data in recovery. I guess it was a whole lot easier than I thought.

Also congratulations and thanks to everyone who has contributed with this so far.

WE MADE IT TO FRONT PAGE ON XDA (08-03-2012)
__________________________________________________ _______________________________________________

Requirements:

There are a few things you will need to do in order for this to work right for you, and a couple of things you'll have to research before you even try it.

#1. Obviously, you have to be rooted/unlocked because you're not gonna be able to change much around otherwise.

#2. You MUST repartition your external SD. The kernel I've put together so far WILL ONLY mount /data to mmcblk1p2, which basically says "mount /data to the second partition in the external SD." also, the ramdisk expects that partition to be ext4, so essentially:
Make sure you have an external SD with at least two partitions and that the second partition is formatted to ext4. I personally use Gparted to repartition my stuff, but feel free to use whatever rocks your boat. Even if you're on windows you can still use gparted by using virtualbox, so I'm not gonna go look for a different windows solution.

#3. This is the research part... This will be beneficial or detrimental to each user depending on the SD card used. If you have a slow SD card this probably will do you no good. However, just because you have a class 10 SD card, that doesn't mean it will benefit you either. On my own research I have found that some class 6-10 SD Cards have extremely slow random write speeds, so if you happen to have one of those, even if it's a class 10, this might not be for you. This means that you're gonna have to do some research to find out if your SD Card will benefit you or not. You can always just give it a try, as far as I know this is entirely reversible, how easy or hard being just a matter of how bad you mess up on meeting the requirements and following the instructions.

#4. At this point (07-30-2012) I'm doing all this stuff using the AOKP milestone 6.1 kernel as base for my modified kernel, so if you're not using AOKP milestone 6.1, flashing my kernel might borke your system. You've been warned, feel free to proceed otherwise at your own peril.
__________________________________________________ __________________________________________________ _________

Installation:

#1. Download attached file (boot-data2SD-AOKP6.1base.zip) and extract it to the root of your internal storage (/sdcard).

#2. Open a terminal.

#3. Type the following:
Code:
su
dd if=/sdcard/boot.blob of=dev/block/mmcblk0p4 bs=1
#4. Wait for it to complete.

#5. Reboot.

Upon rebooting you will know that it worked because it will look just as if you just flashed a new rom, that is, you'll get the device setup screen (assuming that the tablet booted at all lol). If you're planning to use TB to restore your apps, you'll probably want to copy the TB folder to your external SD's first partition so that you can copy it back once you're done with the device setup (at this point you will have no access - unless you manually mount it - to your internal storage).
__________________________________________________ _____________________________________

Reverting:

Follow the same exact steps for installation but use boot-default-AOKP6.1.zip instead.
__________________________________________________ _____________________________________

Optional:

#1. If you want to have access to everything you had on your data partition in the new data partition, you'll have to clone everything from one to the other. To do this, make sure that your new data partition (the one in your external SD) has enough storage space to fit everything you currently have in your data partition (the one in your internal storage). Then run the following command in your terminal.

Code:
dd if=dev/block/mmcblk0p8 of=dev/block/mmcblk1p2 bs=4096 conv=notrunc,noerror
BEWARE that if you have a lot of stuff this can take quite a while and even though I've read a way of getting the progress for this in Linux I'm not sure that you can check the progress on Android.
__________________________________________________ _____________________________________

Next steps in development:

#1. Move /cache as well.
#2. Find out what happens with recovery backups when the partitions are changed.
#3. Attempt to apply mod to motley's kernel.
#4. Create a script that is run on boot to eliminate need for replacing the kernel.
#5. With help from the community, find the best SD Card for this.
#6. Run the modified system for a while to have a good feel for performance benefits
#7. Come up with other interesting uses for this other than getting better I/O (maybe an easy - kinda easy - way to dual boot with ubuntu, maybe other stuff, dunno).
__________________________________________________ _____________________________________

How can I set my kernel to do this?

I didn't do a whole lot, and it's not like I want it to be a secret, so as I modify things I'll try to keep the steps listed here so that anyone modifying their own kernel who would like to try this modification can go ahead and do it.

You'll need to know how to unpack/repack a kernel. Turge has a SUPER EASY explanation here on how to do it on windows (I'll pack together the necessary binaries for linux later, maybe).

Mount /data to the second partition in your external SD (formatted as ext4 filesystem):
After unpacking the kernel navigate to the folder that has the ramdisk and open it
(DON'T USE ANY ASCII BASED TEXT EDITOR BECAUSE IT WILL PROBABLY MESS THINGS UP I USE NOTEPAD++)
Around line 26 change:
Code:
mount ext4 /dev/block/mmcblk0p8 /data wait noatime nodiratime nosuid nodev nodelalloc,errors=panic
to
Code:
mount ext4 /dev/block/mmcblk1p2 /data wait noatime nodiratime nosuid nodev nodelalloc,errors=panic
__________________________________________________ __________________________________________________ ___

SD Cards tested:

Samsung 32GB Class 10 MicroSDHC High Speed Memory Card - Very Good Results

SanDiskŪ microSDHCTM 8GB Memory Card - Very Good Results
"Either way, with this mod, the tablet feels like it should have right from the start. It's speedy and responsive, and apps being installed don't stall the system." - Turge - Post #59
__________________________________________________ __________________________________________________ ___

Benchmarks:

/data mounted to mmcblk0p8 (Internal Storage):






/data mounted to mmcblk1p2 (External SD):






RL Benchmark WITHOUT dagrim1's sql patch as per request:

/data mounted to mmcblk0p8 (Internal Storage):


/data mounted to mmcblk1p2 (External SD):


*Important thing to note: When running the benchmark with data in the internal SD it was about 220 seconds in the first run, then about 180, then about 130 and finally 118; whereas running it from the external SD was consistenly between 63 and 65 seconds every time. I think this more than proves that A) Asus used a cheap I/O storage, B) No matter what software changes are made, more than likely running the rw partitions from a better I/O storage, i.e. an external SD is a good idea.

As promised here are benchmarks on a JB rom (Unofficial CM10). Also sorry it took me a while to get these, I was going to use eos3 but I started getting random reboots. Then I decided to try cm10, and I messed up a flash and had to redo a bunch of things. Anyway, the only change here is the mod itself (no custom kernel or anything). Though one thing to note is that I moved /cache back to the internal partition after some thought that this allows /data and /cache to be written at the same time to different locations thus lowering the bottleneck.

/data mounted to mmcblk0p8 (internal storage):


/data mounted to mmcblk1p2 (external storage):


Now, as you can see, JB did bring a major improvement to I/O, bringing the benchmark down from about 115sec to 68 (almost reaching the modded ICS at 60 seconds). But as I expected, better software works better on better hardware and now the modded JB is running at 50 seconds instead of 60. Next I'm going to put dagrim1's sql patch and see how low the benchmark goes. Also will be posting the modded blob in just a little bit for anyone who wants to use it on CM10.
Attached Thumbnails
Click image for larger version

Name:	Screenshot_2012-08-01-22-17-37.jpg
Views:	31900
Size:	21.8 KB
ID:	1236469   Click image for larger version

Name:	Screenshot_2012-08-01-22-32-24.jpg
Views:	31965
Size:	22.0 KB
ID:	1236470   Click image for larger version

Name:	Screenshot_2012-08-02-23-21-29.jpg
Views:	31415
Size:	20.1 KB
ID:	1239251   Click image for larger version

Name:	Screenshot_2012-08-03-09-05-31.jpg
Views:	31374
Size:	20.2 KB
ID:	1239252  
Attached Files
File Type: zip boot-data2SD-AOKP6.1base.zip - [Click for QR Code] (3.87 MB, 362 views)
File Type: zip boot-default-AOKP6.1.zip - [Click for QR Code] (3.86 MB, 152 views)
File Type: zip flashme-kernel-motley305-aokp-data+cache2SD.zip - [Click for QR Code] (3.99 MB, 173 views)
File Type: zip boot-cm10-unofficial-data+internal.zip - [Click for QR Code] (3.88 MB, 264 views)
The Following 26 Users Say Thank You to TweakerL For This Useful Post: [ Click to Expand ]
 
Striatum_bdr
Old
(Last edited by Striatum_bdr; 29th July 2012 at 08:23 PM.)
#2  
Striatum_bdr's Avatar
Senior Member
Thanks Meter 878
Posts: 3,040
Join Date: May 2011
Location: Marseille
We don't have sd-ext it's an old trick when phones had very little /data partitions, you have the possibility to create a sd-ext partition on an external sdcard and mounting it as a secondary data. (like opt partition on Linux).

To see what block device is data just run 'mount' command in terminal emulator. I don't have my device here.


sdcard cache is already set to 2048 if I'm correct.

Your script would mean creating a sd-ext partition on an external sdcard, modify fstab to have it correctly mounted then applying the script.

Not really easy for common users.

I would rather look at kernel drivers (not I/O schedulers but drivers handling with file system format) but it's quite a hard work.
 
TweakerL
Old
#3  
Senior Member - OP
Thanks Meter 309
Posts: 216
Join Date: Oct 2010
Location: Rosedale, MD

 
DONATE TO ME
Default Hmmm...

Was talking to Rayman and it doesn't actually seem that hard to do... Just gotta change the init.cardhu.rc in the ramdisk to mount /data to /dev/block/mmcblk1p2 instead of /dev/block/mmcblk0p8

The thing is, that while I know every step that I have to take to get it done, I haven't used linux in forever and quite honestly I couldn't even compile blobtools right now if I wanted to to extract the ramdisk from the boot blob to make the necessary change... so yea anyone who knows how to edit a kernel should be able to do it, and then just repack it as a blob... I'll probably look into it later, but if anyone wants to type the terminal commands for to to get/compile blobtools I'll appreciate it...
 
dagrim1
Old
#4  
Senior Member
Thanks Meter 184
Posts: 1,226
Join Date: Sep 2008
As in just a mount of data to mmcblk1p2?

Would a temp solution (just to check if it works) be to remount data manually? (Tried it, to mmcblk1p1 btw since 1p2 didn't seem to exist for me, but it still mounts to mmcblk0p8.

Using:

mount -o remount,rw -t ext4 /dev/block/mmcblk1p1 /data

(as su in terminal)
 
TweakerL
Old
#5  
Senior Member - OP
Thanks Meter 309
Posts: 216
Join Date: Oct 2010
Location: Rosedale, MD

 
DONATE TO ME
Quote:
Originally Posted by dagrim1 View Post
As in just a mount of data to mmcblk1p2?

Would a temp solution (just to check if it works) be to remount data manually? (Tried it, to mmcblk1p1 btw since 1p2 didn't seem to exist for me, but it still mounts to mmcblk0p8.

Using:

mount -o remount,rw -t ext4 /dev/block/mmcblk1p1 /data

(as su in terminal)
Reason why mmcblk1p2 didn't work is because you have to repartition the sd card to have an ext4 partition... personally what I did was take my 32gb sd card and have the first partition as a fat32 partition for storage and set the rest to an ext4 partition... also, you have to do that because the /data partition is already expected to be an ext4 partition on most of the current ROMs... Trying to set it without doing that most likely won't work.

Also, another thing that's important is that for this to be beneficial you have to have an SD Card with higher random write speed than your internal storage speed... my internal storage speed is about .25 mb/s and my sdcard is about 1.5mb/s so there should be a big difference... Oh and if you happen to have a class 10 sdcard that doesn't necessary mean that it has high random write speed... you actually have to go look up the specs or run benchmarks on it.
 
dagrim1
Old
#6  
Senior Member
Thanks Meter 184
Posts: 1,226
Join Date: Sep 2008
Quote:
Originally Posted by TweakerL View Post
Reason why mmcblk1p2 didn't work is because you have to repartition the sd card to have an ext4 partition... personally what I did was take my 32gb sd card and have the first partition as a fat32 partition for storage and set the rest to an ext4 partition... also, you have to do that because the /data partition is already expected to be an ext4 partition on most of the current ROMs... Trying to set it without doing that most likely won't work.

Also, another thing that's important is that for this to be beneficial you have to have an SD Card with higher random write speed than your internal storage speed... my internal storage speed is about .25 mb/s and my sdcard is about 1.5mb/s so there should be a big difference... Oh and if you happen to have a class 10 sdcard that doesn't necessary mean that it has high random write speed... you actually have to go look up the specs or run benchmarks on it.
Thanks, makes sense yeah... will have to check if it's worth the hassle for now. Not at this moment anyway, but interesting concept...
 
TweakerL
Old
#7  
Senior Member - OP
Thanks Meter 309
Posts: 216
Join Date: Oct 2010
Location: Rosedale, MD

 
DONATE TO ME
Default Stuck... again...

So I figured how to unpack the boot.blob, then unpack the boot.blob.LNX, then decompress the ramdisk... made the necessary change to init.cardhu.rc... compressed the ramdisk to the same format it was before, repacked the boot.blob.LNX, repacked the boot.blob... dd if=blob of=dev/block/mmcblk0p4 seek=28 bs=1 ... and ... nothing... reboot and where you're supposed to get the quick progress bar nothing seems to happen... I'm assuming I messed up on the recompressing ramdisk/packing the boot.blob... but I'm not sure how...

Anyway... I'll post exactly how I did it tomorrow so maybe someone with more experience can help me figure out where I messed up...

But so far, I wanna say thanks to rayman and lilstevie for all the help they've given me so far with this idea.
 
kokopuphz
Old
#8  
Member
Thanks Meter 63
Posts: 91
Join Date: Nov 2010
Quote:
Originally Posted by TweakerL View Post
... compressed the ramdisk to the same format it was before, repacked the boot.blob.LNX, repacked the boot.blob... dd if=blob of=dev/block/mmcblk0p4 seek=28 bs=1 ...
Just wondering, why did you choose to use seek=28?
I believe the TFP will need the first 28 header signature to be there in order to flash through the staging paritition (p4).

Your other option would be to flash using fastboot:
1. If you've installed the AndroidRoot.mobi bootloader (if you have nvflash), then you can directly flash the boot.blob.LNX file, as this is a raw image.
fastboot -i 0x0b05 flash boot boot.blob.LNX
2. If you don't have AndroidRoot.mobi bootloader, then I suggest you get NVFlash working first and get a backup... if not, you can use the following to flash the blob:
fastboot -i 0x0b05 flash boot blobfileyou'vecreated
3. Use fastboot to flash to the staging partition:
fastboot -i 0x0b05 flash staging blobfileyou'vecreated
 
Turge
Old
#9  
Turge's Avatar
Recognized Developer
Thanks Meter 18287
Posts: 4,638
Join Date: Sep 2008
Location: Hamilton

 
DONATE TO ME
Quote:
Originally Posted by TweakerL View Post
So I figured how to unpack the boot.blob, then unpack the boot.blob.LNX, then decompress the ramdisk... made the necessary change to init.cardhu.rc... compressed the ramdisk to the same format it was before, repacked the boot.blob.LNX, repacked the boot.blob... dd if=blob of=dev/block/mmcblk0p4 seek=28 bs=1 ... and ... nothing... reboot and where you're supposed to get the quick progress bar nothing seems to happen... I'm assuming I messed up on the recompressing ramdisk/packing the boot.blob... but I'm not sure how...

Anyway... I'll post exactly how I did it tomorrow so maybe someone with more experience can help me figure out where I messed up...

But so far, I wanna say thanks to rayman and lilstevie for all the help they've given me so far with this idea.
I've repacked the boot.img for the Prime before to add init.d support so I'll post my method and the files needed in a few minutes once I get to work. It'll involve getting cygwin installed (with Perl support I believe) if you're on Windows.


Want to give something in return? Follow our project here.
 
Parastie
Old
#10  
Senior Member
Thanks Meter 94
Posts: 802
Join Date: Apr 2010
Location: Lodz, Poland

 
DONATE TO ME
Would it be better to move cache and maybe dalvik cache (assuming the SD random read/write is faster then internal memory) ? Since you're only moving data and leaving cache on internal, that'll still hit the issues of having bad IO. Moving cache (which I believe would have more random access) I think would be better.

Thoughts?
Give me a thanks! It's like Karma!

Devices:
Sony Xperia Ion l28at sim unlocked/bootloader sadly locked by AT&T.
Asus Transformer Prime Wookied Out.
HTC Desire with CM10.1! (RETIRED)
(If you have a prime, you're doing yourself a disservice by NOT unlocking.)
Google Users! Make sure you're using dual authentication. Especially if you have credit cards with Google.

The Following User Says Thank You to Parastie For This Useful Post: [ Click to Expand ]
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes