[MOD] 1-Click Flashable exFAT + NTFS + EXT4 support for CM10.1/AOSP + Partitioned SD!

[lordazoroth]

Just flash the zip. It handles everything and has the NTFS binaries included.

The update-script automatically removes "old" versions of my scripts and places the new ones in their place. It deletes the following extraneous files from the original release, since they don't work; if you actually want the exfat utils (even though they all fail saying "cannot get size"), you have to add them back in manually.

delete("/system/etc/init.d/50exfat-mountsd"); <- Replaced with  50fuse-mountsd
delete("/system/xbin/exfat-mountsd.sh"); <- Replaced with fuse-mountsd.sh

delete("/system/xbin/dumpexfat"); <- None of these work anyway
delete("/system/xbin/exfatlabel");
delete("/system/xbin/exfatfsck");
delete("/system/xbin/mkexfatfs");

Thank you so much!

 

[Captain_Throwback]

Re: [MOD] 1-Click Flashable exFAT + NTFS + EXT4 support for CM10.1 / AOSP (external S

This is an interesting mod. I'm trying the script on my EVO 4G LTE, running CM10.1, Android 4.2.2. The script doesn't detect that I'm running as root, as my Terminal simply identifies me as user 0 (still root). But the script won't continue. Does it matter that I use SuperSU to gain root? Maybe I should try adjusting the script to allow for identifying user 0 as root (how would I do that?)

I only decided to try this because TWRP recovery recently enabled support for exFAT, and the card reads fine there after reformatting. I know the block is set properly, so that's not it. Just the root user recognition appears to be the issue. Can you help?

 

[HMkX2]

This is an interesting mod. I'm trying the script on my EVO 4G LTE, running CM10.1, Android 4.2.2. The script doesn't detect that I'm running as root, as my Terminal simply identifies me as user 0 (still root). But the script won't continue. Does it matter that I use SuperSU to gain root? Maybe I should try adjusting the script to allow for identifying user 0 as root (how would I do that?

I did a pretty basic check for root - it runs "whoami" and looks for the string "root". I have tested my script against Chainfire's SU 1.3 and SuperUser 3.3 - the only difference I noted was Chainfire was a bit slower/laggier due to the popups, and didn't allow ADB shell root access by default - I had to click "ok". Also, the command parameters for an internal command I use (adb shell su -c "blah") were a bit glitchy, and didn't handle escaped strings well.

Could you please run ("whoami" and "id") command and tell me what string it reports? I'll edit the checking in for you and so others can use it. I will look into the proper method to detect root -- but busybox has limited utilities to do so.

 

[Captain_Throwback]

Re: [MOD] 1-Click Flashable exFAT + NTFS + EXT4 support for CM10.1 / AOSP (external S

I did a pretty basic check for root - it runs "whoami" and looks for the string "root". I have tested my script against Chainfire's SU 1.3 and SuperUser 3.3 - the only difference I noted was Chainfire was a bit slower/laggier due to the popups, and didn't allow ADB shell root access by default - I had to click "ok". Also, the command parameters for an internal command I use (adb shell su -c "blah") were a bit glitchy, and didn't handle escaped strings well.

Could you please run ("whoami" and "id") command and tell me what string it reports? I'll edit the checking in for you and so others can use it. I will look into the proper method to detect root -- but busybox has limited utilities to do so.

Turns out your script is fine. I had a poorly compiled version of CM10.1. I used a more reliable version, and now everything works as intended.

So I'm able to use MTP to view the files in Windows (or mount as Mass Storage in recovery), and now the device is able to read from the external SD with no issues.

The only thing I can't do (this being an HTC device) is use the external SD card to update firmware via the bootloader, because the HTC bootloader can't read the card. However there are ways around that, and I'd rather have the card formatted as it was intended.

Thanks for your work on this! You can add the HTC EVO 4G LTE to your list of working devices (at least on CM10.1/Android 4.2.2).

 

[HMkX2]

The only thing I can't do (this being an HTC device) is use the external SD card to update firmware via the bootloader, because the HTC bootloader can't read the card. However there are ways around that, and I'd rather have the card formatted as it was intended.

You'll be happy with my 5.0 release. I'm currently coding multiple-partition support, and user-customizable dynamic mountpoints.

 

[spearoid]

Re: [MOD] 1-Click Flashable exFAT + NTFS + EXT4 support for CM10.1 / AOSP (external S

I have a long question I want to ask you that is related to this thread but sort of off topic still. I tried to PM you but got a message that either you can't receive PM's yet or you've chosen not to receive them. Shall I post it here?

 

[HMkX2]

I tried to PM you but got a message that either you can't receive PM's yet or you've chosen not to receive them.

PM enabled.

For future reference, I do not check PMs... ever. Please do not consider it an insult if I pretend like I never read it. Chances are I haven't.

 

[HMkX2]

Major Update: Version 5.0 is out.

Multiple partitions are supported. Also no more probing back and forth for Block Device, it's all automatic. (Word of warning: it blacklists mmcblk0, but does "explore" a little bit. Please inform me if it lists any inappropriate mounts. I may switch it back to manually specifying device.)

 

[Captain_Throwback]

Major Update: Version 5.0 is out.

Multiple partitions are supported. Also no more probing back and forth for Block Device, it's all automatic. (Word of warning: it blacklists mmcblk0, but does "explore" a little bit. Please inform me if it lists any inappropriate mounts. I may switch it back to manually specifying device.)

Wow. My head a-splode.

Okay, so in order for my bootloader to have a partition to read, I'm going to need to create a FAT32 partition on my SD card in addition to the current EXT4 & exFAT partitions. If I do it that way, will they all mount automatically? I'm wondering how/if TWRP will be able to see all of them, since it can read the external storage as exFAT with no issues, as well as the EXT4 partition. I assume the .mounthere files will only affect the OS while booted, so it can't really affect what the bootloader sees, forcing me to create multiple partitions. What about when mounting to PC via MTP? Will both the FAT32 and exFAT partitions be mounted (in addition to my internal storage)? Currently I get internal storage and SD card. I would expect a third entry there for the FAT32 partition?

This might be getting more complicated than I can handle .

EDIT: Okay, upon re-reading the OP, I think I get it. I need to add a FAT32 partition at the beginning of the card for the bootloader to read. Then, on my exFAT partition, I need to add a .mounthere file that tells the OS to mount my large storage partition at /storage/sdcard1. My FAT32 partition will automatically be mounted at /storage/mmcblk1 (or I can create a folder somewhere and a .mounthere file to have it mounted elsewhere). That part I think I now understand.

What I don't know is how/if TWRP will handle the additional partition. I may have to specify the block location of the new partition in the recovery.fstab and compile a custom version myself to get it to mount. Which is fine, if I can get my buildbox running again. But I don't know if TWRP can handle all those partitions by default, and if it can, how/where they would be mounted.

 

[HMkX2]

Okay, so in order for my bootloader to have a partition to read, I'm going to need to create a FAT32 partition on my SD card in addition to the current EXT4 & exFAT partitions. If I do it that way, will they all mount automatically?

Yes, but only with my script. Create Fat32 (p1) and exFAT+ext4 (p2 and p3).

"Dumb" programs that don't support multiple partitions (Windows [!!], bootloader, CM10.1/AOSP roms) will happily look at the FAT32 (p1) as if it is the only one on the drive. (Although, Windows at least recognizes other partitions exist -- and you can toggle some flags using BOOTICE.exe to make them visible.) My script will come along when you run it (or automatically at boot) and mount things as necessary.

I'm wondering how/if TWRP will be able to see all of them, since it can read the external storage as exFAT with no issues, as well as the EXT4 partition. I assume the .mounthere files will only affect the OS while booted, so it can't really affect what the bootloader sees, forcing me to create multiple partitions.​

Bingo. Dumb programs (bootloader/CM10.1) = only see partition1, smarter programs (TWRP/my script) = see everything. ("But it's too late!")

What about when mounting to PC via MTP? Will both the FAT32 and exFAT partitions be mounted (in addition to my internal storage)? Currently I get internal storage and SD card. I would expect a third entry there for the FAT32 partition?​

In short, no. (Not without some serious hacks to the ROM/stuff I don't know how to do.) Whatever is mounted under "/storage/sdcard1" and "/storage/sdcard0 (more properly, "/data/media") will be visible in MTP. Obviously, you can't mount two things there at once -- only two partitions can be visible.

However, people use a trick where they use "mount -o bind" (Directory Binds, see that program) to make a "Portal" (ala Valve) to another part of the filesystem inside a visible one. However, because of a technical bug current utilities will not work. (Why I was so emphatic on my script being able to work around it.) So yes: it is entirely possible, no: existing utilities cannot do it for you because of a bug (e.g. DirectoryBind + 4.2) , and no: I haven't coded for it (yet).

EDIT: Okay, upon re-reading the OP, I think I get it. I need to add a FAT32 partition at the beginning of the card for the bootloader to read. Then, on my exFAT partition, I need to add a .mounthere file that tells the OS to mount my large storage partition at /storage/sdcard1. My FAT32 partition will automatically be mounted at /storage/mmcblk1 (or I can create a folder somewhere and a .mounthere file to have it mounted elsewhere). That part I think I now understand.​

Bingo, that is 100% correct.

What I don't know is how/if TWRP will handle the additional partition. I may have to specify the block location of the new partition in the recovery.fstab and compile a custom version myself to get it to mount. Which is fine, if I can get my buildbox running again. But I don't know if TWRP can handle all those partitions by default, and if it can, how/where they would be mounted.​

I have absolutely no idea. It depends on how robustly TWRP is coded.

You have to remember I wrote this all in *bash/busybox script*. I am pretty much pushing the limits of what Bash is capable of. I know how to code, but not at the level of "unintelligibility" of the CM10.1 codebase -- if I could compile and knew more about kernel programming, gerrit etiquette, and the "politics" behind code review, I just would have have coded exFAT/ext4/NTFS/btfs support for CM10.1 myself by now.

However, Busybox is near-universal, the binaries are portable, and my script will work on any phone you throw it at.

 

[HMkX2]

So... I was digging through the Android SDK and I found out it is apparently possible to have multiple MTP connections. (i.e.have custom partitions show).

https://android.googlesource.com/platform/docs/source.android.com/+/master/src/tech/storage/index.md

Docs are above. Need to recompile framework-res.apk and add a couple lines to an XML. I have no way to do that or test.

 

[Captain_Throwback]

So . . . I have my 3 partitions set up:

FAT32
exFAT
ext4

And the .mounthere file is working marvelously on the second two . . . but the FAT32 partition (@ /dev/block/mmcblk1p3 according to /proc/partitions) isn't getting mounted by default under /storage. Any idea why?

And for the record, it seems TWRP overlooks the initial FAT32 partition and just reads the exFAT, which is perfect. I haven't tested bootloader yet, as I only have access to all 3 partitions in Ubuntu currently.

 

[HMkX2]

...the .mounthere file is working marvelously on the second two . . . but the FAT32 partition (@ /dev/block/mmcblk1p3 according to /proc/partitions) isn't getting mounted by default under /storage. Any idea why?

Can you please post a debug.txt using the command posted in the OP? I have a suspicion it will say "device busy", since Vold auto-mounts fat32. (Although, it would be the first time I've heard of it mounting partition #3!).

fuse-mountsd.sh > /sdcard/debug.txt 2>&1​

I coded in something to try to unmount it if it did that, but it seems that isn't working. I need to find out why.

 

[Captain_Throwback]

Can you please post a debug.txt using the command posted in the OP? I have a suspicion it will say "device busy", since Vold auto-mounts fat32. (Although, it would be the first time I've heard of it mounting partition #3!).

fuse-mountsd.sh > /sdcard/debug.txt 2>&1​

I coded in something to try to unmount it if it did that, but it seems that isn't working. I need to find out why.

Looks like the script isn't recognizing the partition type as FAT32 for blk3. Attaching log momentarily.

 

[Captain_Throwback]

And now, after mounting my internal storage to my computer in Mass Storage mode, I get a "Damaged SD card" error, and it seems my exFAT partition is corrupted (again). Looks like I'll be reformatting and restoring data again.

I do still like this idea, even though I've ended up reformatting my card several times already (once by forgetting to install the script after a system wipe). I can give you the log from the failed repair attempt too, if you'd like.

 

[HMkX2]

Ah, cripes. There's an even bigger problem than that. You see that giant error at the top saying to report it, that it couldn't determine the NO-GO zone? Those DM- devices shouldn't be mounting! I may pull 5.0 until I can fix that problem.

I can give you the log from the failed repair attempt too, if you'd like.​

And yes, the more logs the merrier. I only have 1 Android device to test on -- I need to see what is wigging out on other devices.

And now, after mounting my internal storage to my computer in Mass Storage mode, I get a "Damaged SD card" error, and it seems my exFAT partition is corrupted (again). Looks like I'll be reformatting and restoring data again.
..
I do still like this idea, even though I've ended up reformatting my card several times already (once by forgetting to install the script after a system wipe).

Wait.... what? You formatted your *internal* storage as exFAT?? (Not your SD card/external storage?) My script was never intended to mount anything on the INTERNAL memory!

(Technical explanation I explicitly blacklisted anything on mmcblk0 to forestall the possibility of people mounting their modems/system/whatever by accident. I could add that support IN, but to me the risks would be too great without knowing EXACTLY how each+every person's partition table is set up.

Request:

Can you run these commands and give me the outputs?

dd if=/dev/block/mmcblk1p3 bs=512 count=1 of=/sdcard/dd_mmcblk1p3.bin.rar
-- Will create dd_mmcblk1p3.bin.rar in your SD card, please upload that.
-- This will tell me what the header signature looks like, I'll have to check it

cat /proc/mounts > /sdcard/mounts.txt
cat /proc/partitions > /sdcard/partitions.txt
-- Will create 2 .txt files on your SD card, I can use those to find out what is mounted where on your phone

I need to know more before specifying anything further.

 

[Captain_Throwback]

Ah, cripes. There's an even bigger problem than that. You see that giant error at the top saying to report it, that it couldn't determine the NO-GO zone? Those DM- devices shouldn't be mounting!

I may pull 5.0 until I can fix that problem.

And yes, the more logs the merrier. I only have 1 Android device to test on -- I need to see what is wigging out on other devices.

I did notice that. Good thing I reported it, then (kind of ). And here's my other log, from the partition that couldn't be repaired.

Is it possible, on the first successful mount, to take a snapshot of the partition information, and then use that to restore the damage later, if necessary? I'm not sure how you're doing it now, but I assume it tries to replace the damaged portions with some standard information, and I was wondering if taking the info from the actual device somehow would be better (if it's even possible). Just spitballin' here, of course. I won't purport to know how this works .

 

[HMkX2]

Is it possible, on the first successful mount, to take a snapshot of the partition information, and then use that to restore the damage later, if necessary? I'm not sure how you're doing it now, but I assume it tries to replace the damaged portions with some standard information, and I was wondering if taking the info from the actual device somehow would be better (if it's even possible). Just spitballin' here, of course. I won't purport to know how this works .

Not spitballin', that's exactly what I thought at one point. I even mention around Page 2 that I had an option of that and another way, and I chose the other way.

There are a couple problems with "saving" the header from the first time it works. (1) There are large chunks of hex of dubious function (2) Where would I store the header? The card? The phone itself? Chicken and the egg. (3) What would happen if you plugged in a DIFFERENT card? It would overwrite it's header with the wrong one!

May I ask what you are using to format the card exFAT? If you get it to the point exFAT is working can you run this command to dump the header of the exFAT partition? OR flash version 5.2 of my script and give another debug.txt, I turned on full logging -- much, much less legible output for the user, but easier for me to find specific problems.

dd if=/dev/block/mmcblk1p1 bs=512 count=1 of=/sdcard/exfat_header.bin.rar​

PS: You do not need to reformat your SD Card and restore data every time. Simply plug the card into Windows and run Chkdsk /f on the partition. You can toggle which partition is visible using BOOTICE.exe. Run the program, click "Parts Manage", click "ReFormat USB Disk" (ignore its name, it is a poor choice of words), and see if "USB-HDD mode" is selected. (It should be.)

Then use the dropdown box, select which partition you want visible, and click "Set Accessible". Click Cancel button to quit. Windows should see that partition for Chkdsk /f now.

 

[HMkX2]

And if I didn't say it already, thank you for the logs and testing! This is what I need to try to make the script work for everyone, even the people who lurk and don't comment.

 

[Captain_Throwback]

Wait.... what? You formatted your *internal* storage as exFAT?? (Not your SD card/external storage?) My script was never intended to mount anything on the INTERNAL memory!

(Technical explanation I explicitly blacklisted anything on mmcblk0 to forestall the possibility of people mounting their modems/system/whatever by accident. I could add that support IN, but to me the risks would be too great without knowing EXACTLY how each+every person's partition table is set up.

Request:

Can you run these commands and give me the outputs?

dd if=/dev/block/mmcblk1p3 bs=512 count=1 of=/sdcard/dd_mmcblk1p3.bin.rar
-- Will create dd_mmcblk1p3.bin.rar in your SD card, please upload that.
-- This will tell me what the header signature looks like, I'll have to check it

cat /proc/mounts > /sdcard/mounts.txt
cat /proc/partitions > /sdcard/partitions.txt
-- Will create 2 .txt files on your SD card, I can use those to find out what is mounted where on your phone

I need to know more before specifying anything further.

Oh, no - sorry for the confusion. It just so happens that after mounting my internal card to the computer, that the EXTERNAL card showed as damaged. I wouldn't dream of formatting my phone's internal memory to a non-native format.

I'll pull this information for you as soon as I can. I'm at work right now, so my options are limited. Oh yeah, and the card is at home.

Not spitballin', that's exactly what I thought at one point. I even mention around Page 2 that I had an option of that and another way, and I chose the other way.

There are a couple problems with "saving" the header from the first time it works. (1) There are large chunks of hex of dubious function (2) Where would I store the header? The card? The phone itself? Chicken and the egg. (3) What would happen if you plugged in a DIFFERENT card? It would overwrite it's header with the wrong one!

May I ask what you are using to format the card exFAT? If you get it to the point exFAT is working can you run this command to dump the header of the exFAT partition? OR flash version 5.2 of my script and give another debug.txt, I turned on full logging -- much, much less legible output for the user, but easier for me to find specific problems.

dd if=/dev/block/mmcblk1p1 bs=512 count=1 of=/sdcard/exfat_header.bin.rar​

PS: You do not need to reformat your SD Card and restore data every time. Simply plug the card into Windows and run Chkdsk /f on the partition. You can toggle which partition is visible using BOOTICE.exe. Run the program, click "Parts Manage", click "ReFormat USB Disk" (ignore its name, it is a poor choice of words), and see if "USB-HDD mode" is selected. (It should be.)

Then use the dropdown box, select which partition you want visible, and click "Set Accessible". Click Cancel button to quit. Windows should see that partition for Chkdsk /f now.

I won't be able to finish formatting the card until I get home, but when I do, I'll dump the header for you.

I'm using the plain old Windows Disk Management tool to format as exFAT. None of my other programs will do it (MiniTool or Paragon).

Great tip on the reformatting thing. Since I have a FAT32 partition at the beginning of the card, Windows wasn't giving me any options to do anything with it. I'd prefer to not have to reformat the card anymore, so hopefully chkdsk /f works (or we find a way to avoid the corruption at all, which would be ideal).

On the header thing - I understand the issues there. Isn't there any check that can be run to confirm the identity of the card inserted into the device? That would be a way of ensuring that the wrong card's header doesn't overwrite a different card. When I insert the card in Ubuntu it seems to have some number by which it mounts the partition as /media, and it seems to be different for every card. But I have no idea where it's getting that from. It's an 8-digit code I think with a dash in the middle, comprised of letters and numbers.

And in the case of where - how about the root of internal storage, or providing an option to the user on where to store it? I know some devices have limited space, so that would be the most flexible option, I suppose.