FORUMS
Remove All Ads from XDA

[HOWTO][exFAT][WORK IN PROGRESS] Mount exFAT formatted drives and cards

161 posts
Thanks Meter: 115
 
By smitna, Senior Member on 1st December 2011, 06:08 PM
Post Reply Email Thread
Reserved...
The Following 2 Users Say Thank You to smitna For This Useful Post: [ View ] Gift smitna Ad-Free
 
 
1st December 2011, 06:08 PM |#2  
OP Senior Member
Thanks Meter: 115
 
Donate to Me
More
[HOWTO][exFAT][WORK IN PROGRESS] Mount exFAT formatted drives and cards
I have successful compiled the exFAT userspace driver from http://code.google.com/p/exfat/ together with libfuse.
So we are theoretically able to mount every exFAT formatted drive (connected via OTG and also the external sdcard, BUT THIS IS NOT YET FULL TESTED).
This howto is far away from being perfect. Also my english isn't perfect - sorry. Feel free to send me corrections.

@Mods: I think it's a development-related thread. If this is not your mind, please move it to the right section - thank you very much (and also for your continuously work in the "background").

* For updates please have a look in the footer of this post, I forgot to reserve a second "post space" *

Please be very careful! I am not responsible for any damage or lost data on your phone or storage. I have tested this on my phone with a connected card reader and with the "external" sdcard
(Galaxy S2, usually mounted at /sdcard/external_sd).

ATTENTION:
I have discovered one "big" problem that must be solved before all other things and I NEED PERHAPS SOME HELP!
Binaries attached read update no. 2:

After every reboot the exFAT partition will be damaged WITHOUT modification of the vold.fstab config, so don't use a card or drive with important data on it.
This has to do with the automount function of the "New Volume Manager Daemon" Vold.
I suspect that the daemon wants to mount the exFAT volume as FAT32 read/write and overwrites the first bytes of the the first block. I will check this.
It doesn't matter if it's the "external" card or a connected drive/sdcard reader.

The problem:
dd if=/dev/block/mmcblk1p1 of=/sdcard/exfat_ok.bin count=1 bs=1024
hexdump -C /sdcard/exfat_ok.bin

the correct hex:
00000000h: EB 76 90 45 58 46 41 54 20 20 20 (three bytes and then the string EXFAT plus three spaces. This is the recognition string for the exfat-utils (exactly 8 bytes)

dd if=/dev/block/mmcblk1p1 of=/sdcard/exfat_not_ok.bin count=1 bs=1024
hexdump -C /sdcard/exfat_not_ok.bin

after a reboot of the phone the string changes to
00000000h: 52 52 61 41 58 46 41 54 20 20 20 (four new bytes at the beginning -> AXFAT, the recognition of the partition fails) I think no more changes are made.

This is not in relation to the exFAT tools or the FUSE library, the "damage" happens before!

Beside that, the exFAT card seems to be usable and after manually mount to /sdcard/external_sd the card can be activated (under settings - memory) - Sorry I have a german GUI...

STOP READING HERE IF YOU DON'T WANT TO PLAY WITH YOUR DATA ON THE EXFAT PARTITION!

There is no long term experience with this. Don't try it if you don't have some skills with Linux and Android. Make a full backup!
You have been warned...

[Q] Why exFAT and not using ext2/3/4 or any other file system?
[A] I don't know. It's your decision. exFAT is developed mainly for flash memory and could be used "out of the box" with newer windows versions and Mac OS X in contrast to ext2/3/4.
Sadly there is no good extX windows driver out available.
exFAT also supports XDHC card from 32 GB upwards. See http://en.wikipedia.org/wiki/ExFAT for more details.
XDHC cards are reported to work with the SG2.
This makes it interesting for micro sdcards greater than 32 GB (if your are lucky and have one) and for large files (greater than 4 GB), like video files.
Note: FAT32 is of course able to address more than 32 GB, but not "official".
There are many disadvantages as well, read the article.

[Q] Is it free and/or free to use?
[A] Once again, I don't know exactly. Tuxera http://www.tuxera.com/ has an agreement (licence program) with Microsoft and announced an exFAT driver for Android, but I couldn't find a free downloadable (source) package.
The driver is probably free to use but not free to distribute and until now not released.

The exFAT project on Google Project Hosting is licensed under GNU GPL v3, so we will and can use it free. But it's still in beta stage.

[Q] What do I need?
[A] A rooted Android phone with a suitable kernel and fuse support, take siyah (because it a good kernel). I have only a Samsung Galaxy S2, that's because the thread is here.
Enough free memory. Installed busybox. Access via adb shell or a ssh-terminal connection (QuickSSHd or SSHDroid from the market).
An other option is to use a terminal window on the phone.

[Q] What could be possible?
[A]
  • automatically mount the OTG drive with util-linux-ng or vold or something like that (needs support from kernel or ROM)
  • replace the FAT32 partition of the "external" sdcard (external_sd) with exFAT (needs support from both kernel and ROM I think)

OK, let's start...

First option, the harder way (you need a linux machine for this): Compiling the driver and utils

  1. Download the CodeSourcery Toolchain/Crosscompiler for ARM EABI for Linux from https://sourcery.mentor.com/sgpp/lit...@template=lite (tested with 2011.03-41, I saw a newer untested version Sourcery G++ Lite 2011.03-42)
  2. Install the compiler on your linux box with (for example) sh ./arm-2011.03-41-arm-none-linux-gnueabi.bin, the installer will asking you a few questions, it should be easy.
  3. Download latest stable fuse (fuse-2.8.6.tar.gz) from http://fuse.sourceforge.net/
  4. Download fuse-exfat (exfat-utils-0.9.5.tar.gz and fuse-exfat-0.9.5.tar.gz) from http://code.google.com/p/exfat/ or use my prepared packages with the Makefiles
  5. Prepare the cross compiler, this is my example script, please change the path (CROSS_PATH) to the CodeSourcery binaries and the CROSS_ROOT path


    Make a directory (CROSS_ROOT) for the libraries and the headers, ex.:
    /android/src/cross/lib
    /android/src/cross/include

    ---- File prepare_codesourcery.sh
    #!/bin/bash
    export CROSS_PATH=/android/CodeSourcery
    export CROSS_ROOT=/android/src/cross

    export ARCH=arm
    export PATH="$CROSS_PATH:$PATH"

    # version 2011.03-41-arm-none-linux-gnueabi
    export CROSS_COMPILE="$CROSS_PATH/bin/arm-none-linux-gnueabi-"
    export CFLAGS=' -I$CROSS_ROOT/include -g -O2 -static -march=armv6 -mfpu=neon -mfloat-abi=softfp'
    export LDFLAGS=' -L$CROSS_PATH/arm-none-linux-gnueabi/libc/lib -L$CROSS_ROOT/lib -Wl,--whole-archive -lpthread -lrt -ldl -Wl,--no-whole-archive'
    export CC="$CROSS_PATH/bin/arm-none-linux-gnueabi-gcc"
    ----

  6. Export the setup:
    . ./prepare_codesourcery.sh

    Check the path with
    arm-none-linux-gnueabi-gcc -v


  7. Compile libfuse
    1. ./configure --host=arm-linux --enable-util --enable-lib --disable-mtab --enable-static=yes --enable-shared=no
    2. make
    3. copy the static libraries libfuse.a and libulockmgr.a from fuse-2.8.6/lib/.libs to $CROSS_ROOT/lib
    4. copy the headers (.h-files) from fuse-2.8.6/include to $CROSS_ROOT/include
    c) and d) is not really necessary, but is used for the flags to find the headers and libs, see the file prepare_codesourcery.sh above.


  8. Compile fuse-exfat
    I had no luck with SCons (a substitution for make) to cross compile for ARM, so I created some Makefiles to build fuse-exfat and exfat-utils, see attachement

    Note: the next step is not necessary, libexfat is also included in exfat-utils if you use the attached source package:

    Use your downloaded sources package and copy the Makefiles from my packages to every directory or use my source packages
    In fuse-exfat/fuse-exfat-0.9.5 run make


  9. Compile exfat-utils
    In fuse-exfat/exfat run make


    Note: If you get errors like strip: "Unable to recognise the format of the input file" then you have to symlink arm strip to strip temporary with
    ln -s $CROSS_PATH/bin/arm-none-linux-gnueabi-strip $CROSS_PATH/bin/strip
    so arm-strip is used instead of strip from your linux dist



    Second option: Download the binaries


    1.-9. Don't care about it...

  10. Copy all binaries to your phone. They are big but "portable" because of the static build. You can use adb or any other method. The files must be executable, so place them for example in /system/xbin or /data/ and chmod them 755
  11. Connect an empty hard drive, empty pen drive or a card reader with an empty sdcard to the phone with an OTG cable. The drive should only contain a prepared partition (don't care about the file system).
    But you can also create a partition with fdisk on the phone, if your busybox installation is useable.
    1. Check the connection of the USB devices with
      lsusb
      or something like that
    2. Check the partitions with
      cat /proc/partitions
      You have to see a new partition like sdc1. The partition is visible under /dev/block/platform/s3c_otghcd/sdc1
    3. Create a new exFAT partition with
      mkexfatfs /dev/block/platform/s3c_otghcd/sdc1
      Check the type of partition with
      fdisk -l /dev/block/platform/s3c_otghcd/sdc
      (you should see it as "HPFS/NTFS")
    4. Make a new directory ex.
      mkdir /data/exfat
      for the mount point
    5. Mount the new exFAT partition read/write with
      mount.exfat-fuse -o rw /dev/block/platform/s3c_otghcd/sdc1 /data/exfat
      to mount point /data/exfat or any other path
      or for testing with
      mount.exfat-fuse -o ro /dev/block/platform/s3c_otghcd/sdc1 /data/exfat
      readonly
    6. To unmount the device use
      sync
      umount /data/exfat

Thanks to gokhanmoral for his great kernel, tolis626 and olifee (members of this forum) to give me the idea of doing this, unknown devs from http://repository.timesys.com/ for a example Makefile to bypass
the unwieldy "SCons". I wasn't able to use it for cross compiling because of tons of parameters and variables, my shame...


Links:
exFAT (GPL): http://code.google.com/p/exfat/wiki/QuckStartGuide (it's not a typo)
Some informations: http://en.wikipedia.org/wiki/ExFAT
SiyahKernel: http://forum.xda-developers.com/show....php?t=1263838
exFAT Makefile: http://repository.timesys.com/builds...9.5-make.patch
CodeSourcery: https://sourcery.mentor.com/sgpp/lit...@template=lite
FUSE (Filesystem in Userspace): http://fuse.sourceforge.net/


Update no. 1 | 01/12/2011
I think I made one step forward: It's vold as I can see.
I have commented out the block for the external_sd in /system/etc/vold.fstab
# external sdcard
#{
# ums_path = /sys/devices/platform/usb_mass_storage/lun1/file
# asec = enable
#}
#dev_mount sdcard1 /mnt/sdcard/external_sd auto /devices/platform/s3c-sdhci.2/mmc_host/mmc1

Now after new rebooting the partition is not damaged and I was able to mount it as expected at /sdcard/external_sd.
The disadvantage is now is that the system cannot recognize the card as a regular sdcard and the memory part in settings is greyed out.
It's like the card is not insert for the ROM, nevertheless the directories are shown in a file explorer like "root explorer" (with free/used values and I could edit a text file with a build-in editor)

Update no. 2 | 05/12/2011
I am now sure after some (more) tests, it's the vold daemon.
To mount a exfat volume, the configuration /system/etc/vold.fstab needs modification.

DO NOT MOUNT A VOLUME WITHOUT MODIFICATION

vold (version 2) is locked to VFAT/FAT32 volumes. Earlier versions had support for ext(2/3/4 ???) volumes too, this was removed by Google and/or Samsung (don't know).

Sadly I can't find a documentation for vold2 and I am stuck here. Because for replacing the "external_sd" from FAT32 to exFAT it's also necessary that vold2 recognizes the card correctly. (Because of the "asec" mounts for Apps2SD).
Perhaps it's possible to map this mounts to the internal sdcard (setting asec = enable in vold.fstab), but I haven't tried this yet.

Conclusion: It's possible to mount such exFAT volume with some restrictions and with modification of the vold.fstab.



Specs:
/data/bin/dumpexfat /dev/block/mmcblk1p1
dumpexfat 0.9.5
Volume label
Volume serial number 0xb965fe93
FS version 1.0
Sector size 512
Cluster size 32768
Sectors count 25173456
Free sectors 25169728
Clusters count 393284
Free clusters 393277
First sector 0
FAT first sector 128
FAT sectors count 3136
First cluster sector 3264
Root directory cluster 5
Volume state 0x0000
FATs count 1
Drive number 0x80
Allocated space 0%

Please no questions about the values, there is a second ext4 partition on the card...

So the configuration in vold.fstab and perhaps some other files have to be changed. I have nearly no knowledge with "void". Is a expert out there?
From command line a short speed test shows this result (no other GUI test possible in the moment):

/data/bin/hdparm -tT /dev/block/mmcblk1p1

/dev/block/mmcblk1p1:
Timing cached reads: 228 MB in 2.01 seconds = 113.27 MB/sec
Timing buffered disk reads: 36 MB in 3.02 seconds = 11.92 MB/sec

Card: Patriot 16 GB Class 10, no OC
The Following 15 Users Say Thank You to smitna For This Useful Post: [ View ] Gift smitna Ad-Free
1st December 2011, 07:52 PM |#3  
z3r0n3's Avatar
Member
Thanks Meter: 31
 
More
Did you test the overhead of a FUSE filesystem on Android? As far I know the performance may be sub-optimal because of the overhead of using a filesystem on userspace mode. The Tuxera driver uses kernel mode and is very optimized, but as far I know it's only for OEM's that want to license the module for their devices.

Anyway, very interesting, mainly because exFAT is the default filesystem for SDXC.
1st December 2011, 08:15 PM |#4  
OP Senior Member
Thanks Meter: 115
 
Donate to Me
More
Quote:
Originally Posted by z3r0n3

Did you test the overhead of a FUSE filesystem on Android? As far I know the performance may be sub-optimal because of the overhead of using a filesystem on userspace mode. The Tuxera driver uses kernel mode and is very optimized, but as far I know it's only for OEM's that want to license the module for their devices.

Anyway, very interesting, mainly because exFAT is the default filesystem for SDXC.

No, I have not tested this. It will be one of the next steps if there is a solution for the problem I wrote about. In the moment I don't know if it's kernel related, rom/vold related or anything other. But I think it should have less overhead than NTFS in userspace, surely more than FAT32. Please read the comments about speed at http://code.google.com/p/exfat/updates/list And yes, the Tuxera driver is not for us "end users". Perhaps Samsung will give us a present in the next official ROM release
2nd December 2011, 12:13 AM |#5  
Senior Member
Thanks Meter: 102
 
More
Quote:
Originally Posted by z3r0n3

Did you test the overhead of a FUSE filesystem on Android? As far I know the performance may be sub-optimal because of the overhead of using a filesystem on userspace mode. The Tuxera driver uses kernel mode and is very optimized, but as far I know it's only for OEM's that want to license the module for their devices.

Anyway, very interesting, mainly because exFAT is the default filesystem for SDXC.

I also did not test it . However, it should be comparable to a desktop machine (taking the slower CPU into account). My NTFS-3G experience for several years has shown it is actually pretty fast, but takes up a lot of CPU time if high fragmentation is present. Nevertheless, I think the throughput (with our devices CPU) will still be much higher than writing to SD-card in most cases.

And yes, although Linus said FUSE-filesystems are just toys, http://www.spinics.net/lists/linux-f.../msg46078.html, they are very fast and stable toys in my experience.

I think we will not see an open-source kernel-module for exFAT / NTFS-3G in the near future. For one, there is the licensing-issue (which will be much more of a problem if it is included in the kernel-sources / can be built against them), and on the other hand, it always took some YEARS time until a new filesystem was reliable enough to warrant an accepted kernel module. After all, btrfs is just becoming widely accepted and stable after 4 years of development (and a shorter time in-kernel). And this is expected to be the next-gen filesystem for linux, and as such the focus of development. With the correct mount-options, it should also be nice to SD-cards . Maybe there will be some time to try it when kernel 3.1 (with the 'stable' version) comes to our phones.

So for the next years, the FUSE-solution is the best we can get, and for Android, the most compatible one across kernels and devices (it only needs a kernel-dependent kernel-module in addition to whats cooking here, after all). The perfomance graphs by Tuxera on their site even show that their fuse-exFAT is faster than in-kernel FAT, so I guess we should not worry about performance even with the open-source beta implementation. Maybe battery life could be an issue if CPU-usage spikes when copying large files, that might be worth some testing.

I'm personally not switching to exFAT in the near future, but will watch this thread and might do some experiments in some weeks when I have more time .

Thanks for the good work, smitna!
2nd December 2011, 01:11 AM |#6  
z3r0n3's Avatar
Member
Thanks Meter: 31
 
More
Quote:
Originally Posted by olifee

I also did not test it . However, it should be comparable to a desktop machine (taking the slower CPU into account). My NTFS-3G experience for several years has shown it is actually pretty fast, but takes up a lot of CPU time if high fragmentation is present. Nevertheless, I think the throughput (with our devices CPU) will still be much higher than writing to SD-card in most cases.

And yes, although Linus said FUSE-filesystems are just toys, http://www.spinics.net/lists/linux-f.../msg46078.html, they are very fast and stable toys in my experience.

I think we will not see an open-source kernel-module for exFAT / NTFS-3G in the near future. For one, there is the licensing-issue (which will be much more of a problem if it is included in the kernel-sources / can be built against them), and on the other hand, it always took some YEARS time until a new filesystem was reliable enough to warrant an accepted kernel module. After all, btrfs is just becoming widely accepted and stable after 4 years of development (and a shorter time in-kernel). And this is expected to be the next-gen filesystem for linux, and as such the focus of development. With the correct mount-options, it should also be nice to SD-cards . Maybe there will be some time to try it when kernel 3.1 (with the 'stable' version) comes to our phones.

So for the next years, the FUSE-solution is the best we can get, and for Android, the most compatible one across kernels and devices (it only needs a kernel-dependent kernel-module in addition to whats cooking here, after all). The perfomance graphs by Tuxera on their site even show that their fuse-exFAT is faster than in-kernel FAT, so I guess we should not worry about performance even with the open-source beta implementation. Maybe battery life could be an issue if CPU-usage spikes when copying large files, that might be worth some testing.

I'm personally not switching to exFAT in the near future, but will watch this thread and might do some experiments in some weeks when I have more time .

Thanks for the good work, smitna!

I have some bad experiences with NTFS-3g on my netbook. Trying to transfer a large number of files from my netbook (running Arch Linux) to my external HDD (that is NTFS) and the transfer was slow and my CPU time are always on 100%. That's why I asked if it was tested, because I don't really know if FUSE is suitable for embedded devices. But yeah, the only way to know is to test, and it's still too early for that .

But licensing is really a issue? I know that Linux have module to read a NTFS partition (but not write, this is why we have NTFS-3g) and there was some work for a read-only module for exFAT too (sadly, it didn't get much attention).

Anyway, I'm not switching for exFAT too anyway and I don't know how they aprove exFAT as the default filesystem on SDXC cards, but it's important anyway and will be critical somewhere in the future to have support to this filesystem.
25th January 2012, 03:36 AM |#7  
Member
Thanks Meter: 8
 
More
First of all, thanks for your work on this. I was shocked to discover that my SGS2 wouldn't support any filesystems that support large files on external sdcards. I was able to get your solution working in the sense that I could format an sdcard with exfat, mount it, write to it, and unmount it. Awesome! I am having one serious problem though: After editing the vold.fstab and rebooting, I get constant FCs after trying to install any APKs. It doesn't matter if I have my external_sd mounted or not. And these are not apps that are trying to install to the sdcard. Any ideas?

I've got the AT&T version of the SGS2 (i777), but I'm running Siyah's latest kernel. The FCs just say it's the media process.
30th January 2012, 05:40 AM |#8  
OP Senior Member
Thanks Meter: 115
 
Donate to Me
More
Quote:
Originally Posted by dildano

First of all, thanks for your work on this. I was shocked to discover that my SGS2 wouldn't support any filesystems that support large files on external sdcards. I was able to get your solution working in the sense that I could format an sdcard with exfat, mount it, write to it, and unmount it. Awesome! I am having one serious problem though: After editing the vold.fstab and rebooting, I get constant FCs after trying to install any APKs. It doesn't matter if I have my external_sd mounted or not. And these are not apps that are trying to install to the sdcard. Any ideas?

I've got the AT&T version of the SGS2 (i777), but I'm running Siyah's latest kernel. The FCs just say it's the media process.

Sorry for my later answer!

This should be nevertheless a problem with the app2sd service and I have no solution for this. My thread here is only a howto, nothing for the "daily to use"...
I also don't know why there is no other selectable alternative file system for "us users" to use for the external card by default and not FAT32.

The media service depends on the vold daemon to my knowledge. So I cannot recomment this for the default external sdcard, because of the media service scans. If you want to store larger files (e.g. video files), you should better split your sdcard in one FAT32 partition and an additional ext2/3/4 partition. On this partition there is no 4 GB limit. The "media scanner" will not scan the files (videos etc.) on the partition, but you can choose the videos from your favorite player with the file chooser.
Hope this helps you a little bit.
31st January 2012, 03:52 AM |#9  
Member
Thanks Meter: 8
 
More
Thanks for the response. I was actually trying out multiple partitions over the weekend to no avail. Granted, I was trying a combination of FAT32 and NTFS. It would actually work for a while until the NTFS partition would appear to get corrupted. I thought about ext2, but my understanding is that Samsung somehow disabled ext* support for external SD cards. Is that not the case? Anyway, I'm surprised that more folks haven't caught onto your work here because storing large files seems to be a fairly common issue for SGS2 users.
31st January 2012, 11:48 AM |#10  
OP Senior Member
Thanks Meter: 115
 
Donate to Me
More
Quote:
Originally Posted by dildano

Thanks for the response. I was actually trying out multiple partitions over the weekend to no avail. Granted, I was trying a combination of FAT32 and NTFS. It would actually work for a while until the NTFS partition would appear to get corrupted. I thought about ext2, but my understanding is that Samsung somehow disabled ext* support for external SD cards. Is that not the case? Anyway, I'm surprised that more folks haven't caught onto your work here because storing large files seems to be a fairly common issue for SGS2 users.

I am using an ext3 partition with deactivated journalling and noatime option on the sdcard since months without problems. Of course I have also a FAT32 partition on the card to stay compatible with vold and media scanner. With a start script it is mounted after every phone restart.
The Following User Says Thank You to smitna For This Useful Post: [ View ] Gift smitna Ad-Free
2nd February 2012, 03:25 PM |#11  
Member
Thanks Meter: 8
 
More
OK, I must have misunderstood. So is it just that vold will not allow us to automatically mount ext* partitions? I'll try it with a script as soon as I can get some time. Thanks.
Post Reply Subscribe to Thread

Tags
drive, exfat, mount, sdcard

Guest Quick Reply (no urls or BBcode)
Message:
Previous Thread Next Thread
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes