Post Reply

[DATA2EXT] [11/Dec] FULL BENCHMARK, V4, Improve the efficiency of IO

OP ownhere

3rd December 2010, 06:08 AM   |  #1  
OP Senior Member
Flag Beijing
Thanks Meter: 288
 
213 posts
Join Date:Joined: Jun 2010
More
This script only for ROM Developers!

EDIT:2010/12/13 update explain:
1. Why three partition?
Answer:for get best balance of stabilty and performance.

2. Why don't use two partition and loopfile on 2nd partition(my script V1)?
Answer:Loopfile on 2nd partition need enable journaling on 2nd partittion file system to get stability after Unexpected loss of power or Unsafe reboot(without umount). And, enable journaling on a large partition is slower than small partition. In my phone, 1.3G /data/app don't need journal, only 300M /data/data need journal, so I use a small parition for safe and performance in V4.

3. Why not use ext4 directly at 3rd partition?
Answer:for speed up sqlite3, ext4 loopfile on ext4 device, make a huge improve for sqlite3 insert.

4.Why not NILFS2 or BRTFS?
Answer: I have already tried nilfs2 before I use ext4. nilfs2 crash my system. Brtfs marked 'EXPERIMENTAL' in kernel source(2.6.32.15), I like stability more than speed.


__________________________________________________ ______________________________________
EDIT:2010/12/12 update V4 script:
/data/:ext4, disable journaling
/data/data:ext4 loopfile on ext4 device, enable journaling on ext4 device, disable journaling on ext4 loopfile, for fast and stable sqlite3 access.
really smooth this time. please trying...
__________________________________________________ ______________________________________
EDIT:2010/12/11 FULL BENCHMARK: (all test use my sandisk C2 16G TF card, run in SuSE Linux 11.2 desktop)
I did a full benchmark for ext2/ext4, with loop or without loop, write, read, sqlite insert, this is result:

I do follow tests, umount after every test to prevent from kernel cache.
Sqlite3 db benchmark is insert 2000 records to db.

500 50k-2M files(random size) to 2G ext4 loopfile on ext4 device, enable journaling on ext4, sync after every file write.
write: 355 seconds
read: 52 seconds
db:76 seconds

500 50k-2M files(random size) to 2G ext4 loopfile on ext4 device, disable journaling on ext4, sync after every file write.
write: 212 seconds
read: 52 seconds
db:3 seconds

500 50k-2M files(random size) to 2G ext2 loopfile on ext4 device, with sync-in-mount-option, sync after every file write.
write: 380 seconds
read: 45 seconds
db: 3 seconds

500 50k-2M files(random size) to 2G ext2 loopfile on ext4 device, without sync-in-mount-option, sync after every file write.
write: 390 seconds
read: 45 seconds
db: 3 seconds

500 50k-2M files(random size) to ext4 device, enable journaling, sync after every file write.
write: 208 seconds
read: 47 seconds
db: 95 seconds

500 50k-2M files(random size) to ext4 device, disable journaling, sync after every file write.
write: 191 seconds
read: 48 seconds
db: 50 seconds

500 50k-2M files(random size) to ext4 device mount with loop option, enable journaling, sync after every file write.
write: 198 seconds
read: 51 seconds
db: 72 seconds


500 50k-2M files(random size) to ext4 device mount with loop option, disable journaling, sync after every file write.
write: 191 seconds
read: 50 seconds
db: 2 seconds


As you see, the best speed is: mount real ext4 device with loop option, disable journaling. The best balance of performance and safe is mount real ext4 device with loop option, enable journaling! We don't need a loopfile but just change mount option.
I attach my testsuit file below.
__________________________________________________ _________________
EDIT:2010/12/10 V3 changelog:
1.change loopback device to real partition for powersave and better write performance.
2.disable second partition's journaling.
3.You need 3rd partition for /data/data. But even without the third partition of this script can also work(not stable). My partition settings:second partition:1.5G, thrid partition:0.5G

Explain:
1.about NAND and SD: Desire have poor NAND memory for userdata, just 150M(this can be enlarged by MTD or HBOOT, but still small for me). This MTD partition need store app, dalvik-cache, app-data, it is too small to fill many apps.So someone think may be someting can be symbol link to SD, this is original Cyanogen A2SD, in this case , the /data/data directory is not moved to SD, because the battery reason and the speed need. This is what the HDROM 2NAND version do. So, the original A2SD is totally same as 2NAND. I don't known why the 2NAND be named. The NAND is not fast because it is yaffs2(journaling) filesystem, so if install many apps the phone will be slow(every app's sqlite have write demand)

2.about ext2/ext3/ext4: Ext2 have no journal feature, so lost power or just normal reboot will be crashed it, ext3 is journaling version of ext2. Ext4 is diffrent from ext2/ext3, it included Extends feautre, which provide better performance for large file(large than block size:4K). I analyze android system files, 4/5 is large than 4k, so the Extends feature is very useful. all EXTX system is base on Block, and the default block size is 1024 for small partition. Larger block size has better performance, so I force block size to 4096 when format.

3.about loopback device: The original Data2SD script make a huge loopback file for /data parition, then get better IO score. The reason is loopback device provide better Read performance. But the Write performance is wrose, so the phone got lag when install app or copy file.And the lag will occur suddenly. If system want to write data to loopback device, it is not real write, just write to kernel buffer/cache, the data will be flush to real disk when real filesystem(where loopback device stay on) got sync or commit command. So the real filesystem must enable journaling or do sync frequently。

4.about my old data2ext script: I try to splite directory to diffrent FS for the balance of performance and lag. Why I formated partition to EXT4? the reason can be found at item 2. Why create a small loopback device? the reason is item 3. But I found some bug with this case, any data write to loopback device need twice kernel IO command, one virtual one real, so the cpu usage will be high. Real EXT4 partition don't need journaling if no loopback device on it. For data integrity on virtual EXT4 loopback device, I had to enable journal on it.So I design new disk layout for better performance, it is DATA2EXT V3.
__________________________________________________ ________________
V2 change log:
1.change loopback device FS to EXT4 for journal, more stable
2.change mount option, add commit=large Prime number, reduce real IO, avoid concentration of commit. change journaling scheduler from Ordered to WriteBack to improve performance.


I would like to share my Data2EXT script, a collection of data2sd(Thanks to sibere) and the advantages of ext4, you can let the phone run at peak efficiency.

I use EXT4 for whole SD 2nd partition, and create a ext2 loopback device with 1/5 size of 2nd partition.

The files in /data/app,/data/.systemapp it not updated frequently. so store them in EXT4.

In /data/data and /data/app-private, a lot of small files and sqlite database, is updated frequently. so store thenm in EXT2 loopback device.

/data/dalvik-cache is stored in phone memory /mnt/asec/mtddata for distributed IO pressure.

/data/misc, /data/system, /data/property is stored in phone memory /mnt/asec/mtddata for system settings restore(or you will lost your timezone/language settings after reboot).

use this script, I can get quandrant score:2000+, IO:4000+, my phone is smooth and quickly.
Attached Files
File Type: zip data2ext.zip - [Click for QR Code] (2.07 MB, 12696 views)
File Type: txt 01data2extV2.txt - [Click for QR Code] (5.3 KB, 2740 views)
File Type: rar 01data2extV3.rar - [Click for QR Code] (1.3 KB, 2574 views)
File Type: zip testsuite.zip - [Click for QR Code] (241.3 KB, 1703 views)
File Type: txt 01data2extV4.txt - [Click for QR Code] (6.4 KB, 4798 views)
Last edited by ownhere; 13th December 2010 at 07:22 AM.
The Following 50 Users Say Thank You to ownhere For This Useful Post: [ View ]
3rd December 2010, 06:29 AM   |  #2  
Junior Member
Thanks Meter: 1
 
6 posts
Join Date:Joined: Jun 2010
More
this is great job

thx ownhere
3rd December 2010, 06:47 AM   |  #3  
kecinzer's Avatar
Senior Member
Flag Decin
Thanks Meter: 1,265
 
1,599 posts
Join Date:Joined: Jul 2007
Donate to Me
More
If I have Data2SD already installed, I have to do it again, or simply copy this file to /system/etc/init.d and reboot?
3rd December 2010, 07:47 AM   |  #4  
Lo_RaT_PeNaT's Avatar
Senior Member
Flag Valencia
Thanks Meter: 189
 
1,135 posts
Join Date:Joined: Aug 2008
More
I do not understand that I do, only copy the script and reboot and automatically makes the whole process or do I have any other operation, thanks and apologies.

PD. I have read and believed the other post, which is to automatically process, the only problem was the FC, which was corrected by manager rom also lose the wifi, this script solves these problems?
Last edited by Lo_RaT_PeNaT; 3rd December 2010 at 07:50 AM.
3rd December 2010, 07:49 AM   |  #5  
Senior Member
Thanks Meter: 123
 
1,536 posts
Join Date:Joined: Jun 2010
Donate to Me
More
Do you need to use any and on this and can u use an ext2 partition
Sent from my HTC Desire using XDA App
3rd December 2010, 07:50 AM   |  #6  
Senior Member
Thanks Meter: 25
 
258 posts
Join Date:Joined: Aug 2010
noob question.. what do i do with the script?
3rd December 2010, 07:53 AM   |  #7  
Lo_RaT_PeNaT's Avatar
Senior Member
Flag Valencia
Thanks Meter: 189
 
1,135 posts
Join Date:Joined: Aug 2008
More
http://forum.xda-developers.com/show...&postcount=477
The Following User Says Thank You to Lo_RaT_PeNaT For This Useful Post: [ View ]
3rd December 2010, 08:07 AM   |  #8  
Bengaal's Avatar
Senior Member
Thanks Meter: 51
 
450 posts
Join Date:Joined: Oct 2008
More
Quote:
Originally Posted by ownhere

I use EXT4 for whole SD 2nd partition, and create a ext2 loopback device with 1/5 size of 2nd partition.

I don't understand this......Can you explain it?
How partition the micro_SD card?
3rd December 2010, 08:16 AM   |  #9  
d3sm0nd's Avatar
Senior Member
Flag Sofia
Thanks Meter: 14
 
547 posts
Join Date:Joined: Dec 2007
More
Can someone explain how to apply the script? Do we have to create the partitions ourself or will the script create them?
Does it work on DHD Roms?
3rd December 2010, 08:34 AM   |  #10  
MacCarron's Avatar
Senior Member
Flag Treviso
Thanks Meter: 145
 
518 posts
Join Date:Joined: Dec 2008
More
Guys there's a thread only god knows how many pages long that explains every possible variation on EXT* loopback devices..how about scrolling just a little next time?

http://forum.xda-developers.com/showthread.php?t=852219

The Following User Says Thank You to MacCarron For This Useful Post: [ View ]
Post Reply Subscribe to Thread
Previous Thread Next Thread
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes


Top Threads in Desire Android Development by ThreadRank