[KERNEL][MOD][08-03-2012] I/O Boost - Data2SD

Search This thread

TweakerL

Senior Member
Oct 6, 2010
280
364
34
Rosedale, MD
K, time to give this a proper OP, if anyone wants any of the info that was here before you can look here

_________________________________________________________________________________________________

So, the whole idea here started with me reading an article on how part of the whole I/O problem with the transformer is partially caused by the hardware used as internal storage. I wanted to find out if this had any merit and I figured the best way to do it would be to "replace" the internal storage. I did this by mounting the /data partition to the exteral SD (which according to my research, my specific SD Card is better at writing speeds - allegedly the main problem with the transformer's internal storage hardware wise). Then I ran a bunch of benchmarks and have been running it that way for about 24 hours and so far it feels great. Anyone is welcome to give it a try, and hopefully with help, suggestions and feedback from the community, we can all take as much advantage of this idea as possible.

Before I go any further I want to give credit to those who helped me so far, because without them I would still be completely clueless, and not only have they helped be accomplishing what I got so far, but thanks to them I've also learned a bunch of things I didn't know before. So here it goes:

Rayman - For suggesting the method for mounting /data to the external SD.
lilstevie - For helping me get the new kernel flashed right.
Turge - For showing me how to properly repack the kernel.
Parastie - For suggesting doing the same thing to /cache (working on that now).
dagrim1 - For SQL patch and for suggesting a temporary remount (even though it didn't work it was a good thing to try).
_motley - For all his work on his awesome kernel.
_________________________________________________________________________________________________

Updates:

Update 1 (08-01-2012 File boot-data+cache+internal-AOKP6.1base.zip) :
Now both /data and /cache are moved to the external SD card. This means you need a third partition mmcblk1p3 in order to use this modification.
It will also mount the internal storage (previously inaccessible) to /mnt/sdcard_internal
It also attempts to (fail at this point) to mount the internal sd partition (what used to be /sdcard) to /sdcard/Internal_SD (which is why you will always see that folder get created but stay empty). If anybody knows how to make it work please advise.
Modified Lines:
init.cardhu.rc
Code:
        # TweakerL MOD > original mount = mmcblk0p8 /data | mmcblk0p3 /cache 
        mount ext4 /dev/block/mmcblk1p2 /data wait noatime nodiratime nosuid nodev nodelalloc,errors=panic
        mount ext4 /dev/block/mmcblk1p3 /cache wait noatime nodiratime nosuid nodev nodelalloc,errors=panic
	# TweakerL MOD > added mounts for internal storage
	mkdir /mnt/sdcard_internal 0000 system system
	mount ext4 /dev/block/mmcblk0p8 /mnt/sdcard_internal wait noatime nodiratime nosuid nodev nodelalloc,errors=panic
	# TweakerL MOD > give access to internal SD from /sdcard
	mkdir /data/media/Internal_SD 0755 media_rw media_rw	
	mount /mnt/sdcard_internal/media /data/media/Internal_SD wait noatime nodiratime nosuid nodev nodelalloc,errors=panic

Update 2 (08-01-2012 About backing up/restoring in recovery) : - READ UPDATE 5
One thing that worried me was that by using this mod people wouldn't be able to backup their data partition properly, but now I know that it's possible to do it. It will only work on TWRP though since it has basic terminal access and keyboard. To do it, go into Advanced > Terminal and in there type:

umount /dev/block/mmcblk0p8
mount /dev/block/mmcblk1p2 /data

And until you reboot, any backup/restore should use the external SD data partition instead of the internal. The same should be doable with the cache partition in case you want to backup/restore that.

Update 3 (08-02-2012 File flashme-kernel-motley305-aokp-data+cache2SD.zip) :

Put together a flashable zip that will install motley's 3.0.5 aokp kernel using this mod. Works like a charm so far though I only tried flashing on TWRP. Also, internal storage can be accessed in /data2 and internal sd can be accessed in /sdcardi . Current changes are as follow:

Code:
        # TweakerL MOD > move /data and /cache to external SD card || original mount = mmcblk0p8 /data | mmcblk0p3 /cache 
        mount ext4 /dev/block/mmcblk1p2 /data wait noatime nodiratime nosuid nodev nodelalloc,errors=panic
        mount ext4 /dev/block/mmcblk1p3 /cache wait noatime nodiratime nosuid nodev nodelalloc,errors=panic
	# TweakerL MOD > create mount for internal storage
	mkdir /data2 0000 system system
	mount ext4 /dev/block/mmcblk0p8 /data2 wait noatime nodiratime nosuid nodev nodelalloc,errors=panic
	# TweakerL MOD > symlink internal sd to a couple of easily accessible locations
	symlink /data2/media /mnt/sdcard_internal
	symlink /data2/media /sdcardi

Update 4 (08-03-2012 File boot-cm10-unofficial-data+internal.zip) :

Running the unofficial CM10 (no cherrypicks one) using this mod and so far it's pretty amazing. The rom itself is pretty stable and even snappier with /data mounted to external SD. Benchmarks are at the bottom. Current modifications:

fstab.cardhu:
Code:
#TweakerL MOD > Move /data to external SD and internal /data to /data2
/dev/block/mmcblk1p2           /data               ext4      rw,nosuid,nodev,noatime,user_xattr,acl,barrier=1,journal_async_commit,nodelalloc,data=writeback    wait
/dev/block/mmcblk0p8           /data2              ext4      rw,nosuid,nodev,noatime,user_xattr,acl,barrier=1,journal_async_commit,nodelalloc,data=writeback    wait

init.cardhu.rc:
Code:
        # TweakerL MOD > create mount for internal storage
	mkdir /data2 0000 system system
        mount_all /fstab.cardhu
	# TweakerL MOD > symlink internal sd to a couple of easily accessible locations
	symlink /data2/media /mnt/sdcard_internal
	symlink /data2/media /sdcardi

Update 5 (08-03-2012 About backing up/restoring in recovery) :
So after doing some tests, and paying more attention to TWRP, I noticed something rather useful:
When you have this mod enabled, or whenever you have a mmcblk1p2 partiion, TWRP will have the sd-ext menu enabled. This means that to backup your data you can simply backup the sd-ext partition and to restore your data you can simply restore your sd-ext partition. No need to worry about manually switching the mount point for /data in recovery. I guess it was a whole lot easier than I thought.

Also congratulations and thanks to everyone who has contributed with this so far.

WE MADE IT TO FRONT PAGE ON XDA (08-03-2012)
_________________________________________________________________________________________________

Requirements:

There are a few things you will need to do in order for this to work right for you, and a couple of things you'll have to research before you even try it.

#1. Obviously, you have to be rooted/unlocked because you're not gonna be able to change much around otherwise.

#2. You MUST repartition your external SD. The kernel I've put together so far WILL ONLY mount /data to mmcblk1p2, which basically says "mount /data to the second partition in the external SD." also, the ramdisk expects that partition to be ext4, so essentially:
Make sure you have an external SD with at least two partitions and that the second partition is formatted to ext4. I personally use Gparted to repartition my stuff, but feel free to use whatever rocks your boat. Even if you're on windows you can still use gparted by using virtualbox, so I'm not gonna go look for a different windows solution.

#3. This is the research part... This will be beneficial or detrimental to each user depending on the SD card used. If you have a slow SD card this probably will do you no good. However, just because you have a class 10 SD card, that doesn't mean it will benefit you either. On my own research I have found that some class 6-10 SD Cards have extremely slow random write speeds, so if you happen to have one of those, even if it's a class 10, this might not be for you. This means that you're gonna have to do some research to find out if your SD Card will benefit you or not. You can always just give it a try, as far as I know this is entirely reversible, how easy or hard being just a matter of how bad you mess up on meeting the requirements and following the instructions.

#4. At this point (07-30-2012) I'm doing all this stuff using the AOKP milestone 6.1 kernel as base for my modified kernel, so if you're not using AOKP milestone 6.1, flashing my kernel might borke your system. You've been warned, feel free to proceed otherwise at your own peril.
_____________________________________________________________________________________________________________

Installation:

#1. Download attached file (boot-data2SD-AOKP6.1base.zip) and extract it to the root of your internal storage (/sdcard).

#2. Open a terminal.

#3. Type the following:
Code:
su
dd if=/sdcard/boot.blob of=dev/block/mmcblk0p4 bs=1

#4. Wait for it to complete.

#5. Reboot.

Upon rebooting you will know that it worked because it will look just as if you just flashed a new rom, that is, you'll get the device setup screen (assuming that the tablet booted at all lol). If you're planning to use TB to restore your apps, you'll probably want to copy the TB folder to your external SD's first partition so that you can copy it back once you're done with the device setup (at this point you will have no access - unless you manually mount it - to your internal storage).
_______________________________________________________________________________________

Reverting:

Follow the same exact steps for installation but use boot-default-AOKP6.1.zip instead.
_______________________________________________________________________________________

Optional:

#1. If you want to have access to everything you had on your data partition in the new data partition, you'll have to clone everything from one to the other. To do this, make sure that your new data partition (the one in your external SD) has enough storage space to fit everything you currently have in your data partition (the one in your internal storage). Then run the following command in your terminal.

Code:
dd if=dev/block/mmcblk0p8 of=dev/block/mmcblk1p2 bs=4096 conv=notrunc,noerror

BEWARE that if you have a lot of stuff this can take quite a while and even though I've read a way of getting the progress for this in Linux I'm not sure that you can check the progress on Android.
_______________________________________________________________________________________

Next steps in development:

#1. Move /cache as well.
#2. Find out what happens with recovery backups when the partitions are changed.
#3. Attempt to apply mod to motley's kernel.
#4. Create a script that is run on boot to eliminate need for replacing the kernel.
#5. With help from the community, find the best SD Card for this.
#6. Run the modified system for a while to have a good feel for performance benefits
#7. Come up with other interesting uses for this other than getting better I/O (maybe an easy - kinda easy - way to dual boot with ubuntu, maybe other stuff, dunno).
_______________________________________________________________________________________

How can I set my kernel to do this?

I didn't do a whole lot, and it's not like I want it to be a secret, so as I modify things I'll try to keep the steps listed here so that anyone modifying their own kernel who would like to try this modification can go ahead and do it.

You'll need to know how to unpack/repack a kernel. Turge has a SUPER EASY explanation here on how to do it on windows (I'll pack together the necessary binaries for linux later, maybe).

Mount /data to the second partition in your external SD (formatted as ext4 filesystem):
After unpacking the kernel navigate to the folder that has the ramdisk and open it
(DON'T USE ANY ASCII BASED TEXT EDITOR BECAUSE IT WILL PROBABLY MESS THINGS UP I USE NOTEPAD++)
Around line 26 change:
Code:
mount ext4 /dev/block/mmcblk0p8 /data wait noatime nodiratime nosuid nodev nodelalloc,errors=panic
to
Code:
mount ext4 /dev/block/mmcblk1p2 /data wait noatime nodiratime nosuid nodev nodelalloc,errors=panic
_______________________________________________________________________________________________________

SD Cards tested:

Samsung 32GB Class 10 MicroSDHC High Speed Memory Card - Very Good Results

SanDisk® microSDHCTM 8GB Memory Card - Very Good Results
"Either way, with this mod, the tablet feels like it should have right from the start. It's speedy and responsive, and apps being installed don't stall the system." - Turge - Post #59
_______________________________________________________________________________________________________

Benchmarks:

/data mounted to mmcblk0p8 (Internal Storage):

attachment.php

attachment.php

attachment.php

attachment.php


/data mounted to mmcblk1p2 (External SD):

attachment.php

attachment.php

attachment.php

attachment.php


RL Benchmark WITHOUT dagrim1's sql patch as per request:

/data mounted to mmcblk0p8 (Internal Storage):
attachment.php


/data mounted to mmcblk1p2 (External SD):
attachment.php


*Important thing to note: When running the benchmark with data in the internal SD it was about 220 seconds in the first run, then about 180, then about 130 and finally 118; whereas running it from the external SD was consistenly between 63 and 65 seconds every time. I think this more than proves that A) Asus used a cheap I/O storage, B) No matter what software changes are made, more than likely running the rw partitions from a better I/O storage, i.e. an external SD is a good idea.

As promised here are benchmarks on a JB rom (Unofficial CM10). Also sorry it took me a while to get these, I was going to use eos3 but I started getting random reboots. Then I decided to try cm10, and I messed up a flash and had to redo a bunch of things. Anyway, the only change here is the mod itself (no custom kernel or anything). Though one thing to note is that I moved /cache back to the internal partition after some thought that this allows /data and /cache to be written at the same time to different locations thus lowering the bottleneck.

/data mounted to mmcblk0p8 (internal storage):
attachment.php


/data mounted to mmcblk1p2 (external storage):
attachment.php


Now, as you can see, JB did bring a major improvement to I/O, bringing the benchmark down from about 115sec to 68 (almost reaching the modded ICS at 60 seconds). But as I expected, better software works better on better hardware and now the modded JB is running at 50 seconds instead of 60. Next I'm going to put dagrim1's sql patch and see how low the benchmark goes. Also will be posting the modded blob in just a little bit for anyone who wants to use it on CM10.
 

Attachments

  • boot-data2SD-AOKP6.1base.zip
    3.9 MB · Views: 722
  • boot-default-AOKP6.1.zip
    3.9 MB · Views: 342
  • Screenshot_2012-08-01-22-17-37.jpg
    Screenshot_2012-08-01-22-17-37.jpg
    21.8 KB · Views: 36,443
  • Screenshot_2012-08-01-22-32-24.jpg
    Screenshot_2012-08-01-22-32-24.jpg
    22 KB · Views: 36,541
  • flashme-kernel-motley305-aokp-data+cache2SD.zip
    4 MB · Views: 246
  • Screenshot_2012-08-02-23-21-29.jpg
    Screenshot_2012-08-02-23-21-29.jpg
    20.1 KB · Views: 35,988
  • Screenshot_2012-08-03-09-05-31.jpg
    Screenshot_2012-08-03-09-05-31.jpg
    20.2 KB · Views: 35,914
  • boot-cm10-unofficial-data+internal.zip
    3.9 MB · Views: 341
Last edited:

Striatum_bdr

Senior Member
May 29, 2011
4,650
2,176
Marseille
We don't have sd-ext it's an old trick when phones had very little /data partitions, you have the possibility to create a sd-ext partition on an external sdcard and mounting it as a secondary data. (like opt partition on Linux).

To see what block device is data just run 'mount' command in terminal emulator. I don't have my device here.


sdcard cache is already set to 2048 if I'm correct.

Your script would mean creating a sd-ext partition on an external sdcard, modify fstab to have it correctly mounted then applying the script.

Not really easy for common users.

I would rather look at kernel drivers (not I/O schedulers but drivers handling with file system format) but it's quite a hard work.
 
Last edited:

TweakerL

Senior Member
Oct 6, 2010
280
364
34
Rosedale, MD
Hmmm...

Was talking to Rayman and it doesn't actually seem that hard to do... Just gotta change the init.cardhu.rc in the ramdisk to mount /data to /dev/block/mmcblk1p2 instead of /dev/block/mmcblk0p8

The thing is, that while I know every step that I have to take to get it done, I haven't used linux in forever and quite honestly I couldn't even compile blobtools right now if I wanted to to extract the ramdisk from the boot blob to make the necessary change... so yea anyone who knows how to edit a kernel should be able to do it, and then just repack it as a blob... I'll probably look into it later, but if anyone wants to type the terminal commands for to to get/compile blobtools I'll appreciate it...
 

dagrim1

Senior Member
Sep 1, 2008
1,706
365
As in just a mount of data to mmcblk1p2?

Would a temp solution (just to check if it works) be to remount data manually? (Tried it, to mmcblk1p1 btw since 1p2 didn't seem to exist for me, but it still mounts to mmcblk0p8.

Using:

mount -o remount,rw -t ext4 /dev/block/mmcblk1p1 /data

(as su in terminal)
 

TweakerL

Senior Member
Oct 6, 2010
280
364
34
Rosedale, MD
As in just a mount of data to mmcblk1p2?

Would a temp solution (just to check if it works) be to remount data manually? (Tried it, to mmcblk1p1 btw since 1p2 didn't seem to exist for me, but it still mounts to mmcblk0p8.

Using:

mount -o remount,rw -t ext4 /dev/block/mmcblk1p1 /data

(as su in terminal)

Reason why mmcblk1p2 didn't work is because you have to repartition the sd card to have an ext4 partition... personally what I did was take my 32gb sd card and have the first partition as a fat32 partition for storage and set the rest to an ext4 partition... also, you have to do that because the /data partition is already expected to be an ext4 partition on most of the current ROMs... Trying to set it without doing that most likely won't work.

Also, another thing that's important is that for this to be beneficial you have to have an SD Card with higher random write speed than your internal storage speed... my internal storage speed is about .25 mb/s and my sdcard is about 1.5mb/s so there should be a big difference... Oh and if you happen to have a class 10 sdcard that doesn't necessary mean that it has high random write speed... you actually have to go look up the specs or run benchmarks on it.
 

dagrim1

Senior Member
Sep 1, 2008
1,706
365
Reason why mmcblk1p2 didn't work is because you have to repartition the sd card to have an ext4 partition... personally what I did was take my 32gb sd card and have the first partition as a fat32 partition for storage and set the rest to an ext4 partition... also, you have to do that because the /data partition is already expected to be an ext4 partition on most of the current ROMs... Trying to set it without doing that most likely won't work.

Also, another thing that's important is that for this to be beneficial you have to have an SD Card with higher random write speed than your internal storage speed... my internal storage speed is about .25 mb/s and my sdcard is about 1.5mb/s so there should be a big difference... Oh and if you happen to have a class 10 sdcard that doesn't necessary mean that it has high random write speed... you actually have to go look up the specs or run benchmarks on it.

Thanks, makes sense yeah... will have to check if it's worth the hassle for now. Not at this moment anyway, but interesting concept...
 

TweakerL

Senior Member
Oct 6, 2010
280
364
34
Rosedale, MD
Stuck... again...

So I figured how to unpack the boot.blob, then unpack the boot.blob.LNX, then decompress the ramdisk... made the necessary change to init.cardhu.rc... compressed the ramdisk to the same format it was before, repacked the boot.blob.LNX, repacked the boot.blob... dd if=blob of=dev/block/mmcblk0p4 seek=28 bs=1 ... and ... nothing... reboot and where you're supposed to get the quick progress bar nothing seems to happen... I'm assuming I messed up on the recompressing ramdisk/packing the boot.blob... but I'm not sure how...

Anyway... I'll post exactly how I did it tomorrow so maybe someone with more experience can help me figure out where I messed up...

But so far, I wanna say thanks to rayman and lilstevie for all the help they've given me so far with this idea.
 

kokopuphz

Senior Member
Nov 10, 2010
129
84
... compressed the ramdisk to the same format it was before, repacked the boot.blob.LNX, repacked the boot.blob... dd if=blob of=dev/block/mmcblk0p4 seek=28 bs=1 ...
Just wondering, why did you choose to use seek=28?
I believe the TFP will need the first 28 header signature to be there in order to flash through the staging paritition (p4).

Your other option would be to flash using fastboot:
1. If you've installed the AndroidRoot.mobi bootloader (if you have nvflash), then you can directly flash the boot.blob.LNX file, as this is a raw image.
fastboot -i 0x0b05 flash boot boot.blob.LNX
2. If you don't have AndroidRoot.mobi bootloader, then I suggest you get NVFlash working first and get a backup... if not, you can use the following to flash the blob:
fastboot -i 0x0b05 flash boot blobfileyou'vecreated
3. Use fastboot to flash to the staging partition:
fastboot -i 0x0b05 flash staging blobfileyou'vecreated
 

Turge

Inactive Recognized Developer
Sep 20, 2008
4,792
20,768
Hamilton
venomroms.com
So I figured how to unpack the boot.blob, then unpack the boot.blob.LNX, then decompress the ramdisk... made the necessary change to init.cardhu.rc... compressed the ramdisk to the same format it was before, repacked the boot.blob.LNX, repacked the boot.blob... dd if=blob of=dev/block/mmcblk0p4 seek=28 bs=1 ... and ... nothing... reboot and where you're supposed to get the quick progress bar nothing seems to happen... I'm assuming I messed up on the recompressing ramdisk/packing the boot.blob... but I'm not sure how...

Anyway... I'll post exactly how I did it tomorrow so maybe someone with more experience can help me figure out where I messed up...

But so far, I wanna say thanks to rayman and lilstevie for all the help they've given me so far with this idea.

I've repacked the boot.img for the Prime before to add init.d support so I'll post my method and the files needed in a few minutes once I get to work. It'll involve getting cygwin installed (with Perl support I believe) if you're on Windows.
 

Parastie

Senior Member
Apr 27, 2010
825
95
Malta
Would it be better to move cache and maybe dalvik cache (assuming the SD random read/write is faster then internal memory) ? Since you're only moving data and leaving cache on internal, that'll still hit the issues of having bad IO. Moving cache (which I believe would have more random access) I think would be better.

Thoughts?
 
  • Like
Reactions: Moe5508

TweakerL

Senior Member
Oct 6, 2010
280
364
34
Rosedale, MD
As in just a mount of data to mmcblk1p2?

Would a temp solution (just to check if it works) be to remount data manually? (Tried it, to mmcblk1p1 btw since 1p2 didn't seem to exist for me, but it still mounts to mmcblk0p8.

Using:

mount -o remount,rw -t ext4 /dev/block/mmcblk1p1 /data

(as su in terminal)

Thanks for the idea, I tried to do that but nothing seemed to happen (checking on file manager /data partition is still taking the same amount of space as it did before). It would've been a really good way of testing this whole thing though :)
 

TweakerL

Senior Member
Oct 6, 2010
280
364
34
Rosedale, MD
Just wondering, why did you choose to use seek=28?
I believe the TFP will need the first 28 header signature to be there in order to flash through the staging paritition (p4).

Your other option would be to flash using fastboot:
1. If you've installed the AndroidRoot.mobi bootloader (if you have nvflash), then you can directly flash the boot.blob.LNX file, as this is a raw image.
fastboot -i 0x0b05 flash boot boot.blob.LNX
2. If you don't have AndroidRoot.mobi bootloader, then I suggest you get NVFlash working first and get a backup... if not, you can use the following to flash the blob:
fastboot -i 0x0b05 flash boot blobfileyou'vecreated
3. Use fastboot to flash to the staging partition:
fastboot -i 0x0b05 flash staging blobfileyou'vecreated

I used seek=28 out of despair and by lilstevie's suggestions... dd with or without seek had the same exact result, the staging just ignoring the whole thing lol...

Sounds like a plan (flashing with fastboot)... I've got one dumb question though before I do that (and yea i've got nvflash setup and all the backups and stuff). How do I actually go about restoring the system with NVFLASH if I go and borke the system ? XD
 
Last edited:

TweakerL

Senior Member
Oct 6, 2010
280
364
34
Rosedale, MD
I've repacked the boot.img for the Prime before to add init.d support so I'll post my method and the files needed in a few minutes once I get to work. It'll involve getting cygwin installed (with Perl support I believe) if you're on Windows.

That would be much appreciated, I don't mind steps for windows or linux, I'll go either way :)
 

TweakerL

Senior Member
Oct 6, 2010
280
364
34
Rosedale, MD
Would it be better to move cache and maybe dalvik cache (assuming the SD random read/write is faster then internal memory) ? Since you're only moving data and leaving cache on internal, that'll still hit the issues of having bad IO. Moving cache (which I believe would have more random access) I think would be better.

Thoughts?

I agree that moving the cache is a good idea, one of the main reasons why I'm testing with /data though is that it will be much easier to have solid evidence of whether this works or not that way since all the benchmark apps seem to benchmark on the /data partition. I know benchmarks aren't real world results, but if I can run benchmarks on the same partition and it's 5 times faster on the SD card than on the internal memory, I think it should mean something. After that, if there are positive results, I'm thinking of moving both /data and /cache partitions and run that way for a while to see how well it performs, and then to run with just the /cache moved and see how that performs.
 

Turge

Inactive Recognized Developer
Sep 20, 2008
4,792
20,768
Hamilton
venomroms.com
I've repacked the boot.img for the Prime before to add init.d support so I'll post my method and the files needed in a few minutes once I get to work. It'll involve getting cygwin installed (with Perl support I believe) if you're on Windows.

Here are the steps for repacking the boot.img. Some involve running the commands via cygwin, others involve running them via the Windows Command Prompt.

The instructions for installing cygwin, extracting and repacking the boot.img were found here: http://www.freeyourandroid.com/guide/extract-edit-repack-boot-img-windows

Once you have setup cygwin, extract the attached files in a folder under your "home" folder in cygwin.

  1. copy boot.blob to the same folder and run the following via the Windows Command Prompt to extract the boot.img from the boot.blob:

    Code:
    BlobUnpack.exe boot.blob
    ren boot.blob.LNX boot.img
  2. From the cygwin bash terminal window, switch to the same folder and run the following to extract the ramdisk from the boot.img:

    Code:
    ./extractboot boot.img
    You now have an out/ramdisk folder that contains the files you want to edit.
  3. Once done, repack the ramdisk and kernel into boot_new.img with the following command (via cygwin once again):

    Code:
    ./packboot
  4. then from the Command Prompt repack boot_new.img into boot2.blob using the following:

    Code:
    blobpack -s boot2.blob LNX boot_new.img
  5. You can now flash the boot.blob to the staging partition via a command in updater-script:

    Code:
    package_extract_file("/boot.blob", "/dev/block/mmcblk0p4");
    or by using adb while in recovery/android:

    Code:
    dd if=/sdcard/boot2.blob of=/dev/block/mmcblk0p4
 

Attachments

  • PrimeBootImgFiles.rar
    22.8 KB · Views: 90
Last edited:

tshoulihane

Senior Member
Jun 16, 2010
578
108
Did anyone think of running iotop? If we know what part of /data is contributing to the stalls, maybe an interesting idea would be to just mount that part of the tree on SD?
 

TweakerL

Senior Member
Oct 6, 2010
280
364
34
Rosedale, MD
what's iotop?

Though regardless, the problem I'm trying to deal with is the fact that apparently, the storage hardware in the Prime has limited I/O capabilities, namely random write speeds, regardless of software. Because of this the stalls are at least partially caused by the "where" the /data is rather than the "content" in the /data.

Sent from my Transformer Prime TF201 using Xparent SkyBlue Tapatalk 2
 

TweakerL

Senior Member
Oct 6, 2010
280
364
34
Rosedale, MD
ffs. Why do I bother?

You know, I can just google it, but the fact that you care enough to post your opinion but not enough to explain it is the kind of mentality that keeps people who could potentially contribute to the community from doing so because they have to go research all over the internet, possibly going through bad information, for something that might be very simple. Read a few posts up and you'll see the right kind of mindset. Turge could've just*given some halfassed response and sent me on a wild goose chase but instead he took the time to explain in a way that anyone with any amount of knowledge could understand...

And I hope that since you can't bother to give an useful response, that you can't bother wasting you "precious time" justifying and complaining about how people ask questions that they could just look for elsewhere...
 

Top Liked Posts

  • There are no posts matching your filters.
  • 26
    K, time to give this a proper OP, if anyone wants any of the info that was here before you can look here

    _________________________________________________________________________________________________

    So, the whole idea here started with me reading an article on how part of the whole I/O problem with the transformer is partially caused by the hardware used as internal storage. I wanted to find out if this had any merit and I figured the best way to do it would be to "replace" the internal storage. I did this by mounting the /data partition to the exteral SD (which according to my research, my specific SD Card is better at writing speeds - allegedly the main problem with the transformer's internal storage hardware wise). Then I ran a bunch of benchmarks and have been running it that way for about 24 hours and so far it feels great. Anyone is welcome to give it a try, and hopefully with help, suggestions and feedback from the community, we can all take as much advantage of this idea as possible.

    Before I go any further I want to give credit to those who helped me so far, because without them I would still be completely clueless, and not only have they helped be accomplishing what I got so far, but thanks to them I've also learned a bunch of things I didn't know before. So here it goes:

    Rayman - For suggesting the method for mounting /data to the external SD.
    lilstevie - For helping me get the new kernel flashed right.
    Turge - For showing me how to properly repack the kernel.
    Parastie - For suggesting doing the same thing to /cache (working on that now).
    dagrim1 - For SQL patch and for suggesting a temporary remount (even though it didn't work it was a good thing to try).
    _motley - For all his work on his awesome kernel.
    _________________________________________________________________________________________________

    Updates:

    Update 1 (08-01-2012 File boot-data+cache+internal-AOKP6.1base.zip) :
    Now both /data and /cache are moved to the external SD card. This means you need a third partition mmcblk1p3 in order to use this modification.
    It will also mount the internal storage (previously inaccessible) to /mnt/sdcard_internal
    It also attempts to (fail at this point) to mount the internal sd partition (what used to be /sdcard) to /sdcard/Internal_SD (which is why you will always see that folder get created but stay empty). If anybody knows how to make it work please advise.
    Modified Lines:
    init.cardhu.rc
    Code:
            # TweakerL MOD > original mount = mmcblk0p8 /data | mmcblk0p3 /cache 
            mount ext4 /dev/block/mmcblk1p2 /data wait noatime nodiratime nosuid nodev nodelalloc,errors=panic
            mount ext4 /dev/block/mmcblk1p3 /cache wait noatime nodiratime nosuid nodev nodelalloc,errors=panic
    	# TweakerL MOD > added mounts for internal storage
    	mkdir /mnt/sdcard_internal 0000 system system
    	mount ext4 /dev/block/mmcblk0p8 /mnt/sdcard_internal wait noatime nodiratime nosuid nodev nodelalloc,errors=panic
    	# TweakerL MOD > give access to internal SD from /sdcard
    	mkdir /data/media/Internal_SD 0755 media_rw media_rw	
    	mount /mnt/sdcard_internal/media /data/media/Internal_SD wait noatime nodiratime nosuid nodev nodelalloc,errors=panic

    Update 2 (08-01-2012 About backing up/restoring in recovery) : - READ UPDATE 5
    One thing that worried me was that by using this mod people wouldn't be able to backup their data partition properly, but now I know that it's possible to do it. It will only work on TWRP though since it has basic terminal access and keyboard. To do it, go into Advanced > Terminal and in there type:

    umount /dev/block/mmcblk0p8
    mount /dev/block/mmcblk1p2 /data

    And until you reboot, any backup/restore should use the external SD data partition instead of the internal. The same should be doable with the cache partition in case you want to backup/restore that.

    Update 3 (08-02-2012 File flashme-kernel-motley305-aokp-data+cache2SD.zip) :

    Put together a flashable zip that will install motley's 3.0.5 aokp kernel using this mod. Works like a charm so far though I only tried flashing on TWRP. Also, internal storage can be accessed in /data2 and internal sd can be accessed in /sdcardi . Current changes are as follow:

    Code:
            # TweakerL MOD > move /data and /cache to external SD card || original mount = mmcblk0p8 /data | mmcblk0p3 /cache 
            mount ext4 /dev/block/mmcblk1p2 /data wait noatime nodiratime nosuid nodev nodelalloc,errors=panic
            mount ext4 /dev/block/mmcblk1p3 /cache wait noatime nodiratime nosuid nodev nodelalloc,errors=panic
    	# TweakerL MOD > create mount for internal storage
    	mkdir /data2 0000 system system
    	mount ext4 /dev/block/mmcblk0p8 /data2 wait noatime nodiratime nosuid nodev nodelalloc,errors=panic
    	# TweakerL MOD > symlink internal sd to a couple of easily accessible locations
    	symlink /data2/media /mnt/sdcard_internal
    	symlink /data2/media /sdcardi

    Update 4 (08-03-2012 File boot-cm10-unofficial-data+internal.zip) :

    Running the unofficial CM10 (no cherrypicks one) using this mod and so far it's pretty amazing. The rom itself is pretty stable and even snappier with /data mounted to external SD. Benchmarks are at the bottom. Current modifications:

    fstab.cardhu:
    Code:
    #TweakerL MOD > Move /data to external SD and internal /data to /data2
    /dev/block/mmcblk1p2           /data               ext4      rw,nosuid,nodev,noatime,user_xattr,acl,barrier=1,journal_async_commit,nodelalloc,data=writeback    wait
    /dev/block/mmcblk0p8           /data2              ext4      rw,nosuid,nodev,noatime,user_xattr,acl,barrier=1,journal_async_commit,nodelalloc,data=writeback    wait

    init.cardhu.rc:
    Code:
            # TweakerL MOD > create mount for internal storage
    	mkdir /data2 0000 system system
            mount_all /fstab.cardhu
    	# TweakerL MOD > symlink internal sd to a couple of easily accessible locations
    	symlink /data2/media /mnt/sdcard_internal
    	symlink /data2/media /sdcardi

    Update 5 (08-03-2012 About backing up/restoring in recovery) :
    So after doing some tests, and paying more attention to TWRP, I noticed something rather useful:
    When you have this mod enabled, or whenever you have a mmcblk1p2 partiion, TWRP will have the sd-ext menu enabled. This means that to backup your data you can simply backup the sd-ext partition and to restore your data you can simply restore your sd-ext partition. No need to worry about manually switching the mount point for /data in recovery. I guess it was a whole lot easier than I thought.

    Also congratulations and thanks to everyone who has contributed with this so far.

    WE MADE IT TO FRONT PAGE ON XDA (08-03-2012)
    _________________________________________________________________________________________________

    Requirements:

    There are a few things you will need to do in order for this to work right for you, and a couple of things you'll have to research before you even try it.

    #1. Obviously, you have to be rooted/unlocked because you're not gonna be able to change much around otherwise.

    #2. You MUST repartition your external SD. The kernel I've put together so far WILL ONLY mount /data to mmcblk1p2, which basically says "mount /data to the second partition in the external SD." also, the ramdisk expects that partition to be ext4, so essentially:
    Make sure you have an external SD with at least two partitions and that the second partition is formatted to ext4. I personally use Gparted to repartition my stuff, but feel free to use whatever rocks your boat. Even if you're on windows you can still use gparted by using virtualbox, so I'm not gonna go look for a different windows solution.

    #3. This is the research part... This will be beneficial or detrimental to each user depending on the SD card used. If you have a slow SD card this probably will do you no good. However, just because you have a class 10 SD card, that doesn't mean it will benefit you either. On my own research I have found that some class 6-10 SD Cards have extremely slow random write speeds, so if you happen to have one of those, even if it's a class 10, this might not be for you. This means that you're gonna have to do some research to find out if your SD Card will benefit you or not. You can always just give it a try, as far as I know this is entirely reversible, how easy or hard being just a matter of how bad you mess up on meeting the requirements and following the instructions.

    #4. At this point (07-30-2012) I'm doing all this stuff using the AOKP milestone 6.1 kernel as base for my modified kernel, so if you're not using AOKP milestone 6.1, flashing my kernel might borke your system. You've been warned, feel free to proceed otherwise at your own peril.
    _____________________________________________________________________________________________________________

    Installation:

    #1. Download attached file (boot-data2SD-AOKP6.1base.zip) and extract it to the root of your internal storage (/sdcard).

    #2. Open a terminal.

    #3. Type the following:
    Code:
    su
    dd if=/sdcard/boot.blob of=dev/block/mmcblk0p4 bs=1

    #4. Wait for it to complete.

    #5. Reboot.

    Upon rebooting you will know that it worked because it will look just as if you just flashed a new rom, that is, you'll get the device setup screen (assuming that the tablet booted at all lol). If you're planning to use TB to restore your apps, you'll probably want to copy the TB folder to your external SD's first partition so that you can copy it back once you're done with the device setup (at this point you will have no access - unless you manually mount it - to your internal storage).
    _______________________________________________________________________________________

    Reverting:

    Follow the same exact steps for installation but use boot-default-AOKP6.1.zip instead.
    _______________________________________________________________________________________

    Optional:

    #1. If you want to have access to everything you had on your data partition in the new data partition, you'll have to clone everything from one to the other. To do this, make sure that your new data partition (the one in your external SD) has enough storage space to fit everything you currently have in your data partition (the one in your internal storage). Then run the following command in your terminal.

    Code:
    dd if=dev/block/mmcblk0p8 of=dev/block/mmcblk1p2 bs=4096 conv=notrunc,noerror

    BEWARE that if you have a lot of stuff this can take quite a while and even though I've read a way of getting the progress for this in Linux I'm not sure that you can check the progress on Android.
    _______________________________________________________________________________________

    Next steps in development:

    #1. Move /cache as well.
    #2. Find out what happens with recovery backups when the partitions are changed.
    #3. Attempt to apply mod to motley's kernel.
    #4. Create a script that is run on boot to eliminate need for replacing the kernel.
    #5. With help from the community, find the best SD Card for this.
    #6. Run the modified system for a while to have a good feel for performance benefits
    #7. Come up with other interesting uses for this other than getting better I/O (maybe an easy - kinda easy - way to dual boot with ubuntu, maybe other stuff, dunno).
    _______________________________________________________________________________________

    How can I set my kernel to do this?

    I didn't do a whole lot, and it's not like I want it to be a secret, so as I modify things I'll try to keep the steps listed here so that anyone modifying their own kernel who would like to try this modification can go ahead and do it.

    You'll need to know how to unpack/repack a kernel. Turge has a SUPER EASY explanation here on how to do it on windows (I'll pack together the necessary binaries for linux later, maybe).

    Mount /data to the second partition in your external SD (formatted as ext4 filesystem):
    After unpacking the kernel navigate to the folder that has the ramdisk and open it
    (DON'T USE ANY ASCII BASED TEXT EDITOR BECAUSE IT WILL PROBABLY MESS THINGS UP I USE NOTEPAD++)
    Around line 26 change:
    Code:
    mount ext4 /dev/block/mmcblk0p8 /data wait noatime nodiratime nosuid nodev nodelalloc,errors=panic
    to
    Code:
    mount ext4 /dev/block/mmcblk1p2 /data wait noatime nodiratime nosuid nodev nodelalloc,errors=panic
    _______________________________________________________________________________________________________

    SD Cards tested:

    Samsung 32GB Class 10 MicroSDHC High Speed Memory Card - Very Good Results

    SanDisk® microSDHCTM 8GB Memory Card - Very Good Results
    "Either way, with this mod, the tablet feels like it should have right from the start. It's speedy and responsive, and apps being installed don't stall the system." - Turge - Post #59
    _______________________________________________________________________________________________________

    Benchmarks:

    /data mounted to mmcblk0p8 (Internal Storage):

    attachment.php

    attachment.php

    attachment.php

    attachment.php


    /data mounted to mmcblk1p2 (External SD):

    attachment.php

    attachment.php

    attachment.php

    attachment.php


    RL Benchmark WITHOUT dagrim1's sql patch as per request:

    /data mounted to mmcblk0p8 (Internal Storage):
    attachment.php


    /data mounted to mmcblk1p2 (External SD):
    attachment.php


    *Important thing to note: When running the benchmark with data in the internal SD it was about 220 seconds in the first run, then about 180, then about 130 and finally 118; whereas running it from the external SD was consistenly between 63 and 65 seconds every time. I think this more than proves that A) Asus used a cheap I/O storage, B) No matter what software changes are made, more than likely running the rw partitions from a better I/O storage, i.e. an external SD is a good idea.

    As promised here are benchmarks on a JB rom (Unofficial CM10). Also sorry it took me a while to get these, I was going to use eos3 but I started getting random reboots. Then I decided to try cm10, and I messed up a flash and had to redo a bunch of things. Anyway, the only change here is the mod itself (no custom kernel or anything). Though one thing to note is that I moved /cache back to the internal partition after some thought that this allows /data and /cache to be written at the same time to different locations thus lowering the bottleneck.

    /data mounted to mmcblk0p8 (internal storage):
    attachment.php


    /data mounted to mmcblk1p2 (external storage):
    attachment.php


    Now, as you can see, JB did bring a major improvement to I/O, bringing the benchmark down from about 115sec to 68 (almost reaching the modded ICS at 60 seconds). But as I expected, better software works better on better hardware and now the modded JB is running at 50 seconds instead of 60. Next I'm going to put dagrim1's sql patch and see how low the benchmark goes. Also will be posting the modded blob in just a little bit for anyone who wants to use it on CM10.
    5
    I'm using External SD with data2sd and ext4 partition.



    To Disable Journaling in EXT4

    Go into TWRP and then open a command prompt on your computer where adb is.
    Type adb shell and enter it then type the following commands one at a time.

    umount /data
    umount /sdcard
    e2fsck -f /dev/block/mmcblk1p2
    tune2fs -O ^has_journal -c 1 -i 1d -m 0 /dev/block/mmcblk1p2
    e2fsck -f /dev/block/mmcblk1p2

    Watch out for the fourth line, after tune2fs - that is a capital O not a zero. The one further along the line is a zero.

    To check if it has worked type the follwing after them

    tune2fs -l /dev/block/mmcblk1p2 | grep features

    If you don't see has_journal in the output all is good.
    3
    Here's a modified boot.blob from EOS nightly 35.

    The only changes are in /fstab.cardhu:

    • mounts 2nd partition of SD card to /data
    • mount old data (/dev/block/mmcblk0p8) as /data1

    Code:
    /dev/block/mmcblk1p2        /data               ext4      noatime,nosuid,nodev,nomblk_io_submit,errors=panic    wait,encryptable=footer
    /dev/block/mmcblk0p8        /data1              ext4      noatime,nosuid,nodev,nomblk_io_submit,errors=panic    wait,encryptable=footer

    and /init.rc:
    Code:
    mkdir /data1 0771 system system
    3
    I've repacked the boot.img for the Prime before to add init.d support so I'll post my method and the files needed in a few minutes once I get to work. It'll involve getting cygwin installed (with Perl support I believe) if you're on Windows.

    Here are the steps for repacking the boot.img. Some involve running the commands via cygwin, others involve running them via the Windows Command Prompt.

    The instructions for installing cygwin, extracting and repacking the boot.img were found here: http://www.freeyourandroid.com/guide/extract-edit-repack-boot-img-windows

    Once you have setup cygwin, extract the attached files in a folder under your "home" folder in cygwin.

    1. copy boot.blob to the same folder and run the following via the Windows Command Prompt to extract the boot.img from the boot.blob:

      Code:
      BlobUnpack.exe boot.blob
      ren boot.blob.LNX boot.img
    2. From the cygwin bash terminal window, switch to the same folder and run the following to extract the ramdisk from the boot.img:

      Code:
      ./extractboot boot.img
      You now have an out/ramdisk folder that contains the files you want to edit.
    3. Once done, repack the ramdisk and kernel into boot_new.img with the following command (via cygwin once again):

      Code:
      ./packboot
    4. then from the Command Prompt repack boot_new.img into boot2.blob using the following:

      Code:
      blobpack -s boot2.blob LNX boot_new.img
    5. You can now flash the boot.blob to the staging partition via a command in updater-script:

      Code:
      package_extract_file("/boot.blob", "/dev/block/mmcblk0p4");
      or by using adb while in recovery/android:

      Code:
      dd if=/sdcard/boot2.blob of=/dev/block/mmcblk0p4
    3
    Hi all. Attached is a boot.blob for Data2SD on Hairybean 1.2. Flash via fastboot.

    (This is actually a blob, not an img - XDA won't accept a .blob file, so I renamed it to attach to this post. It shouldn't matter what it's called for flashing purposes though)
Our Apps
Get our official app!
The best way to access XDA on your phone
Nav Gestures
Add swipe gestures to any Android
One Handed Mode
Eases uses one hand with your phone