Attend XDA's Second Annual Developer Conference, XDA:DevCon 2014!
5,731,698 Members 49,411 Now Online
XDA Developers Android and Mobile Development Forum

[DEV] data2"XYZ" general filesystem discussion (Updated Dec 20 - Benchmarking)

Tip us?
 
melethron
Old
(Last edited by melethron; 20th December 2010 at 05:35 PM.)
#1  
Senior Member - OP
Thanks Meter 192
Posts: 854
Join Date: Sep 2010
Post [DEV] data2"XYZ" general filesystem discussion (Updated Dec 20 - Benchmarking)

I will rewrite this posting soon. As for now please check the scheduler Benchmark:

http://forum.xda-developers.com/show...&postcount=257



================================================== ==============
data2btrfs (this is DEV status so don't ask how to install - there is (little) linux expirience required)

Update about btrfs dec 15: Still testing btrfs and im REALLY amazed. Battery life has no difference to ext4 but performance is FAR better then ext4. I do still get lags in some cases (like fast! downloads) but this is simply a hardware issue which can not be addressed. Im using a class 2 sd only and i dont think that this can get ANY better. I think btrfs should run like HELL on a class 6+ card.

Update dec 14: Atm i'm using btrfs with "ssd_spread" (for ssd harddisk with older controllers. The controllers in SD should suck compared to some modern SSD so this will effect SD-Cards even more than "ssd" optimization (more info). And im amazed. It is even more smooth than nilfs (maybe less cpu) and so far the smoothest expirence i had with data2sd/data2ext. Even compared to EXT4 without journaling.

Why btrfs?

1. Btrfs is "copy-on-write" filesystem and i think that "copy-on-write" or logbased (like nilfs) will be the future.

2. Btrfs and nilfs are both better for "old" or "cheap" SSD-Drives because they are optimized for it. While this will decrease the performance of a modern SSD-Harddisk (since they are getting better and better in emulating a rotating device like a classic HD) it will INCREASE the general performance on ssd like an sd-card because their controllers is a lot worse than one of a state-of-the art SSD-Harddisk. So these optimization.

3. MeGoo (a linux made for smartphones) already has chosen it as default even though its still beta and i think they have some good reasons for this.

4. Btrfs has the same (and theoratically it should be even less - tiny) battery consumption. Other then nilfs wich had horrible battery (over 1%/h on its own even when phone was idle). Thats why i moved to btrfs from nilfs.

EDIT: Removed something really stupid xD



Safety concerns about data2btrfs


script: Considering the script i addressed ALL safety conserns that there are at present. This is based on the findings of the SGS devs and mounting directly on /data is even more save then cyanogen's app2sd+ (at least for the desire which unmounts /data - SGS doesn't). Only dalvik-cache is symlinked and a data corruption there is unimportant and can be helped by a simple wipe of the cache. So no concerns about this. (But note: the script is still "young" and there might be still errors - thats why i also don't provide a zip for ext4 yet)

btrfs is indeed still in an experimental state. But speaking FOR MYSELF ONLY (everyone should estimate this on its own) i don't have any concerns. The reasons are:
-experimental state is almost over and i doubt that there are any major bugs present.
-many Linux already use it and its default for MeGoo
-I have backups for all sensible data on my computer (like license keys - this can be lost if you backup already corrupted data with titanium/etc)


How to use it

(experienced linux users who know what they do ONLY)

- (recommended for ~ +20% speed) Delete your old ext partition create a new primary partition if it is not aligned yet. Gparted has a "align to mb" option wich makes alignement increadibly easy. Since 1024kb alignement is enough for sd there isn't anything else you need to care about.
- Make a btrfs partition with your favorite linux distribution (im still a ubuntu noob ) with btrfs utils.
- Make sure you flash a Rom that doesn't need sd-ext for installation (check the META-INF/../update-script for "copy_package XYZ SD-EXT:XYZ" - delete or format sd-ext there isn't a problem you will get an error (like on leedroid) on install but this doesn't matter)
-make sure you have Kernel with btrfs support (see below for this)


Kernels with btrfs support (SENSE ONLY for now)


-NeoPhyTe.x360 's (ext2, ext3, ext4, reiserfs, xfs, jfs, btrfs, nilfs2)

-Coutts99 Kernels (chose the btrfs-test-kernel ofc ;) )

Known Issues

no btrfs support in recovery this means:

-no backup of the sd-ext partition (it isn't ext anymore ;) )
-wont work with ROMs that need sd-ext for installation (some HD roms)

Thanks

- cyanogen who made app2sd+ in the first place.
- sibere and ownhere who started all the data2sd.
- supercurio and Unhelpful of SGS' Voodoolagfix (#Project-Voodoo on freenode) for their helpfull script tipps.
- Coutts99 and Neophyte for their Kernels.
- woti23 for all those test's in this thread.
- All devs of XDA in generall for all this nice stuff that is done here

Further Plans for the script
- Changing scheduler settings depending on results in the btrfs thread to get the last out of your SD while assuring safety (thanks woti23 for all the testing).
- Trying to add support to delete the ext partition and create it new with an 1024k alignement (~ +20% more speed)
- Mounting /app-cache to /cache or internal data partition (8 MB more RAM since its mounted on RAM)
- Changes for the mount options of the internal YAFFS2 Partition to optimize the speed of the "dalvik-cache" (have to look into YAFFS2 what can be done)
- Adding some lines of code to clear up any present symlinks on first boot to make it ROM upgrade compatible
- Adding e2fsck with check on every boot (script is quick edit of my script for btrfs - no e2fsck yet)
- Other changes based on the results in this or other threads about data2ext/data2sd
- Remove the errors i made ^^
---------------------------------------------------------------------------------------
Remark: i moved the data2ext4 script to my other script and set the dev tag again. I also added UBIFS discussion to the title. I also removed the nilfs utils and the data2ext4. The data2ext script can be found here. If someone wants to test nilfs you can PM me (but battery SUCKS incredibly )
---------------------------------------------------------------------------------------


Changelog:
:
(dec 15): One minor change in assigning a variable. Doesn't change ANYTHING in actual behavior but code it's more "logical" (assigning the output of "busybox ls" to the variable instead of the command ^^ d'oh)
(dec 15) Fixed a stupid error that could lead to exit code (2) ....... and uploaded the proper file for btrfs (not the ext one). Sorry.... -.-
(dec 16) Fixed the fix ....
(dec 16) the scripts create now the folder it needs if it is not present yet (sorry i also forgot this in required ... was only there in the ext4 thread ^^)

================================================== =======================
DISCLAIMER: BTRFS is still under development and thus an EXPERIMENTAL filesystem. Please refer to the btrfs wiki for more information.
Attached Files
File Type: txt 40data2btrfs.txt - [Click for QR Code] (1.8 KB, 99 views)
The Following 17 Users Say Thank You to melethron For This Useful Post: [ Click to Expand ]
 
coutts99
Old
#2  
Senior Member
Thanks Meter 263
Posts: 1,040
Join Date: Nov 2010
Location: Sunderland

 
DONATE TO ME
Quote:
Originally Posted by melethron View Post
Quick Info: NILFS2 - data2nilf2 will be much faster then data2ext and won't have Lags.


Long info:
After some thought's how to make things better for data2ext i started to think about how a flash based memory actually works.

Your Sd-card isn't a HDD (as you may have noticed ^^). It is actually a NAND based flash memory wich is divided into pages of 2-4kb and they are grouped to blocks of 128kb-512kb (!). But you will never see these. And you cant never change these (mkfs -b as much as you like ^^). This is done in hardware by a controller. All you will see is - you may call it an emulation - the LBA provided by the "flash translation Layer" (FTL). While the FTL can read/and write per page (2-4kb) it can't overwrite it. It needs to delete the data first AND THIS CAN ONLY BE DONE PER BLOCK !!! (Thats why read mb/s is higher then write mb/s).

What this means. Your filessystem should be aligned on the Blocks and should also avoid the need to erase blocks.

So forget every filesystem that my rock on your HDD (it may prefer e rewriteble sector if it's close f/e. If you "emulate" a filesystem through the controller it should respect the "true nature" of the hardware. A logbased filesystem would be far better since it will more likely take advantage of erased blocks. Also you may want that the blocks of the FS fit on the blocks of the NAND (alignement - but this mathes for every fs).

... Well long story short ... you need NILFS2

Links

http://www.usenix.org/event/lsf08/tech/shin_SSD.pdf

and this:
http://forum.xda-developers.com/show...4&postcount=21

(For germans: http://www.hardwareluxx.de/community...l#post10373042)

Btw: In the XDA thread linked use it for the internal sd in samsung galaxy S for a LAGFIX. Wasn't lag the reason data2ext sucks atm.


My 2nd partition would already use this by now but there are 2 things needed:
-Kernel Support
-mount binary

Im to noobish to provide both ... so .... help ... anyone .... pls ?!
Ill try and sort a kernel
The Following 2 Users Say Thank You to coutts99 For This Useful Post: [ Click to Expand ]
 
coutts99
Old
#3  
Senior Member
Thanks Meter 263
Posts: 1,040
Join Date: Nov 2010
Location: Sunderland

 
DONATE TO ME
http://mirror.couttstech.com/android...est_kernel.zip

Usual HAVS@925 kernel with NILFS built-in

As for mount binary. busybox mount might work. I've just been playing with aufs and normal mount worked for that.
The Following 7 Users Say Thank You to coutts99 For This Useful Post: [ Click to Expand ]
 
melethron
Old
#4  
Senior Member - OP
Thanks Meter 192
Posts: 854
Join Date: Sep 2010
Quote:
Originally Posted by coutts99 View Post
Ill try and sort a kernel
NICE! looks far more promissing than btrfs (at least for flash memory)
 
coutts99
Old
#5  
Senior Member
Thanks Meter 263
Posts: 1,040
Join Date: Nov 2010
Location: Sunderland

 
DONATE TO ME
Quote:
Originally Posted by melethron View Post
NICE! looks far more promissing than btrfs (at least for flash memory)
I was going to have a play with this but forgot to bring home my spare sd card and my sd card adaptor

Sent from my HTC Desire using XDA App
The Following User Says Thank You to coutts99 For This Useful Post: [ Click to Expand ]
 
NeoPhyTe.x360
Old
#6  
NeoPhyTe.x360's Avatar
Retired Recognized Developer
Thanks Meter 1804
Posts: 1,645
Join Date: Aug 2009
Location: oυτ o иoωнєяє

 
DONATE TO ME
Another One Nilfs2+Reiserfs


http://neophyte.vipme.com/ftp/_Extra...iserfs_v70.zip
The Following 5 Users Say Thank You to NeoPhyTe.x360 For This Useful Post: [ Click to Expand ]
 
melethron
Old
#7  
Senior Member - OP
Thanks Meter 192
Posts: 854
Join Date: Sep 2010
Quote:
Originally Posted by coutts99 View Post
http://mirror.couttstech.com/android...est_kernel.zip

Usual HAVS@925 kernel with NILFS built-in

As for mount binary. busybox mount might work. I've just been playing with aufs and normal mount worked for that.
Thanks x 3 times.

About binary

"Mount -t auto" didn't work with nilfs2 with ubuntu. "Mount -t nilfs2" calls mount.nilfs2 i think.


EDIT: Did you add nilfs or nilfs2?
 
coutts99
Old
#8  
Senior Member
Thanks Meter 263
Posts: 1,040
Join Date: Nov 2010
Location: Sunderland

 
DONATE TO ME
Quote:
Originally Posted by melethron View Post
Thanks x 3 times.

About binary

"Mount -t auto" didn't work with nilfs2 with ubuntu. "Mount -t nilfs2" calls mount.nilfs2 i think.


EDIT: Did you add nilfs or nilfs2?

Nilfs2

Sent from my HTC Desire using XDA App
 
coutts99
Old
#9  
Senior Member
Thanks Meter 263
Posts: 1,040
Join Date: Nov 2010
Location: Sunderland

 
DONATE TO ME
Quote:
Originally Posted by NeoPhyTe.x360 View Post
Neo, have you got mount binary cross compiled for arm?

Sent from my HTC Desire using XDA App
The Following User Says Thank You to coutts99 For This Useful Post: [ Click to Expand ]
 
NeoPhyTe.x360
Old
(Last edited by NeoPhyTe.x360; 10th December 2010 at 08:09 PM.)
#10  
NeoPhyTe.x360's Avatar
Retired Recognized Developer
Thanks Meter 1804
Posts: 1,645
Join Date: Aug 2009
Location: oυτ o иoωнєяє

 
DONATE TO ME
Quote:
Originally Posted by coutts99 View Post
Neo, have you got mount binary cross compiled for arm?

Sent from my HTC Desire using XDA App


Not, we need this, but I havent got source of this.

need mkfs.reiserfs, and mkfs.nilfs2

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes