[MOD] Switch from non-emulated to emulated internal SD card

[Lanchon]

I start this thread due to popular demand. This issue is brought up every once in a while in regards to the Galaxy S2 family of devices, but may be of interest to owners of other similarly old hardware. I have been sitting on this info for over a year without time to document it and develop it further, and finally decided to publish what i know so others can implement what is needed.

NOTE: This information applies to Android 6. In Android 7+ things could have changed.
EDIT: confirmed working on Android 7.1.1 (CM14.1) by the.gangster!


What is this?

Old hardware that originally shipped with pre-ICS Android (such as the Galaxy S2 and friends) that provides an "internal" SD card, do so by means of a dedicated FAT partition in the eMMC that is separate from the EXT4 /data partition. On the other hand, newer hardware comes with a single big EXT4 /data partition and "emulates" an SD card (or several) by storing their contents in the /data/media folder. This is called "emulated storage".


Advantages of emulated storage

Emulated storage has many advantages over non-emulated storage:

- EXT4 is a modern journaling file system that is much safer than FAT.
- Due to increased safety, mount-time full file system check is not necessary in EXT4; so it mounts much faster than FAT.
- FAT is limited to files that are less than 4GB in size.
- Emulated storage can be encrypted.
- Only devices with emulated storage can support multiple users.


Conventional wisdom

All over the net you can find discussions regarding switching to emulated storage. Basically a two step process:

- Repartition your device to a huge /data and a vestigial or deleted /sdcard.
- Build a modified Android from source for your device that uses emulated storage.

Discussions center on the fact that ROM maintainers do not want to switch to emulated storage to avoid forcing everyone to repartition and wipe their phones.


The big misconception

The reality is that emulated storage can be used with standard ROMs built for non-emulated devices. The ROMs do not even need to be patched and can be flashed and used as-is. The repeated discussions over whether CM should switch have always been unnecessary.


How does it work then?

Surprisingly, Android stores the flag determining whether storage emulation is enabled or not in /data, presumably to allow upgrades from non-emulated to emulated modes. The flag is stored in /data/system/storage.xml: non-emulated storage is signaled by the presence of XML attribute primaryStorageUuid="primary_physical". To enable emulated storage: just edit this file, remove said attribute, and reboot. It is that simple.


What happens if I upgrade my ROM?

Everything keeps on working as it should; the above change only affects /data.


What happens if I wipe /data?

Your device will return to non-emulated mode. If the /sdcard partition is missing, it might not be able to boot.


Can wipe be 'fixed'?

Yes, but it requires a change in the ROM. This change has to be reapplied somehow every time you upgrade your ROM.

After a wipe, Android will initialize /data/system/storage.xml based on the value of system property ro.vold.primary_physical. If set to 1, Android will use non-emulated storage. To fix wipe, set this property to 0.


Is that all there is to it?

Nope! Recovery will still think you are on non-emulated storage and format /data when you choose to wipe it. This means that "wipe /data" will also wipe your emulated SD card!!! This is very dangerous.

I recommend that someone like Arnab builds a TWRP image for emulated storage. Maybe a second official build could be configured: something like device 'i9100_emu'.

I made an on-device patcher for TWRP before, and a year ago I tried to adapt it to mod TWRP for emulated storage use. Unfortunately I seem to remember that it is not possible to mod a TWRP image as the emu/non-emu distinction apparently gets baked into the code during compilation. Nonetheless, i found a file called 'lanchon-emulated-sd-twrp-3.0.2-0-i9100.img' lying around in my hard drive, and i have no idea what it is. Maybe a failed attempt? Could it work? I don't know, i don't remember what it is; guess not, but i could post it.


Moving forward

A year ago I wanted to make some scripts to do these changes, then run them though Flashize and sign them to convert them to flashable scripts. I don't have time for any of this now, but if someone wants to follow up, here is what could be done:

- A script to convert to emulated storage.
- A script to mod the ROM to use emulated storage after a /data wipe.
- A script like the previous one that uses the 'addon' ROM flashing mechanism to auto-run after each ROM flash, just like GAPPS do.

Of course a modded TWRP image would also be a great thing to have.


What else?

There is the question of what happens to the partition that had been /sdcard before the conversion. I suppose it gets mounted as a secondary SD card. In the case of the S2, that has an actual SD card slot, you can probably have two secondary SD cards mounted simultaneously.

The vestigial internal SD card is useless and it would be preferable to have it disappear. Methods to try:

- Wipe the /sdcard partition area so that the file system is gone and it cannot be mounted. This might produce the desired outcome, or Android might just fail to boot.
- Mod the ROM so that it does not look for this partition.
- Configure Android to swap SD cards 0 and 1, so that at least /sdcard0 is the external, useful one. (There are several threads covering this.)


How to repartition

REPIT cannot move data from one partition to another. So unless you manually backup the data in /sdcard, you will loose it.

On the i9100, to enlarge /data and reduce /sdcard to the minimum, you can use this configuration after backing up the /sdcard:

lanchon-repit-XXXXXXXX-system=same-data=max-sdcard=min+wipe-preload=min+wipe-i9100.zip​

If you want to try rendering /sdcard unmountable to see what happens, you can use this configuration instead:

lanchon-repit-XXXXXXXX-system=same-data=max-sdcard=min+wipe+swap-preload=min+wipe-i9100.zip​

NOTE: REPIT does not work on encrypted phones! If your phone is encrypted, you will have to back up /data to an external SD card and then format it to get rid of encryption before using REPIT.


And that's all folks!

I believe this procedure is too involved for the average user. So IMHO the standard builds should not move to emulated storage, given that advanced users can convert by themselves. The big problem is that wiping /sdcard is unavoidable due to lack of spare space to hold the data, unless really smart scripts that reduce and enlarge the partitions incrementally while moving files are made. A TWRP build that is able to backup and restore both the emulated and non-emulated sdcards to the external sdcard would be a big hit for this operation. The other big issue is handling encrypted phones; TWRP backups can help in that regard too.

Please share your experiences on this thread. I cannot test any of this, but with your help a full solution can be found.

 

[Lanchon]

(reserved)

 

[Namoi]

Thanks for these insights. I do hope someone with the needed skills will carry on ?

 

[Lanchon]

Thanks for these insights. I do hope someone with the needed skills will carry on ?

well, you can carry on! just use a file manager in root mode, create a backup of /data/system/storage.xml, edit the live file, and remove the attribute. that's all there is to it. your sdcard will be /data/media/0 after a reboot. you can copy your current sdcard files there (better do it after the first reboot) using adb shell in TWRP. some (but not all) files can also be copied using a root file manager.

 

[the.gangster]

NOTE: This information applies to Android 6. In Android 7+ things could have changed.

I dared to try it.

Just to confirm:
Modifying the storage.xml still seems to do the job on Android 7.1.1 (CM14.1).

After extinguishing the mentioned attribute (and nothing else yet! ) in storage.xml:
- an emulated storage was created in /data/media/0
- default folder structure got created in there
- settings->storage shows "internal share storage" as well as the well-known "sdcard0" and "sdcard1"
- file manager (where I had bookmarks for sdcard0 + 1) now shows bookmarks to Internal shared storage + sdcard1
- camera app (set to store pictures on phone) automatically switched to the new folderstructure
- Total Commander still showed sdcard0+1 until deleting its app-data -> now it references SD-card to emulated/0 and USB to sdcard1
- After connecting the phone to a PC and switching USB connection from "charging" to "Transfer files" even the PC only shows "Internal shared storage" and "sdcard1"

So all in all, android seems to really kind of automatically hide the existing non-emulated one in most apps, while still allowing access to it thru /mnt/media_rw/UUID if needed.

That's it for today from my side.
I'll add some screenshots.

 

[Lanchon]

I dared to try it.

Just to confirm:
Modifying the storage.xml still seems to do the job on Android 7.1.1 (CM14.1).

After extinguishing the mentioned attribute (and nothing else yet! ) in storage.xml:
- an emulated storage was created in /data/media/0
- default folder structure got created in there
- settings->storage shows "internal share storage" as well as the well-known "sdcard0" and "sdcard1"
- file manager (where I had bookmarks for sdcard0 + 1) now shows bookmarks to Internal shared storage + sdcard1
- camera app (set to store pictures on phone) automatically switched to the new folderstructure
- Total Commander still showed sdcard0+1 until deleting its app-data -> now it references SD-card to emulated/0 and USB to sdcard1
- After connecting the phone to a PC and switching USB connection from "charging" to "Transfer files" even the PC only shows "Internal shared storage" and "sdcard1"

So all in all, android seems to really kind of automatically hide the existing non-emulated one in most apps, while still allowing access to it thru /mnt/media_rw/UUID if needed.

That's it for today from my side.
I'll add some screenshots.

so it seems everything works!!

strange that the system seems to show the size of sdcard0 as used spaced in internal storage...

so will you backup sdcard0 and REPIT? or go back to non-emulated? i'm curious about what would happen with the +wipe+swap options in sdcard=. will the phone boot or not? who knows...

 

[ale5000]

Does someone have any link that explain the "'addon' ROM flashing mechanism"?

PS: Does using emulated SD affect read/write performance?

 

[the.gangster]

so it seems everything works!!
strange that the system seems to show the size of sdcard0 as used spaced in internal storage...
so will you backup sdcard0 and REPIT? or go back to non-emulated? i'm curious about what would happen with the +wipe+swap options in sdcard=. will the phone boot or not? who knows...

Free space: Yes, it looks like the size calculation simply shows the used space out of the whole emmc area (mmcblk0). So all partitions except /data (mmcblk0p10) are "used" space (even if they are free) and only the actual free space in the /data partition (1.5gb of 4.5gb) is shown as "not used".
Repit: Haven't made my mind up, yet. As I only started testing it out of curiosity. I wasn't even a friend of the idea to break up with all the well-known tools and reinventing everything from scratch. But as it turns out now, it might not be the big step that it was expected to be. Meanwhile I would rather consider preserving a small internalsd, not to break with the partition layout.
Whether formatted or not shouldn't make a big difference. There have been so many people here, who used the odin-with-pitfile-method to change their partitions and who where not instructed properly to format the internalSD in the recovery right away, that I can recall the only thing happend was, the s2 booted up and if an App wanted to access that storage it resulted in "sdcard0 is corrupted"-errors.
But maybe I'll also manage to test the rest within the next week.
@ale5000
just take a look into your /system/addon.d folder and you'll find some scripts that are executed automatically upon ROM upgrades. They should serve as good examples already.

 

[Lanchon]

Free space: Yes, it looks like the size calculation simply shows the used space out of the whole emmc area (mmcblk0). So all partitions except /data (mmcblk0p10) are "used" space (even if they are free) and only the actual free space in the /data partition (1.5gb of 4.5gb) is shown as "not used".
Repit: Haven't made my mind up, yet. As I only started testing it out of curiosity. I wasn't even a friend of the idea to break up with all the well-known tools and reinventing everything from scratch. But as it turns out now, it might not be the big step that it was expected to be. Meanwhile I would rather consider preserving a small internalsd, not to break with the partition layout.
Whether formatted or not shouldn't make a big difference. There have been so many people here, who used the odin-with-pitfile-method to change their partitions and who where not instructed properly to format the internalSD in the recovery right away, that I can recall the only thing happend was, the s2 booted up and if an App wanted to access that storage it resulted in "sdcard0 is corrupted"-errors.
But maybe I'll also manage to test the rest within the next week.

@ale5000
just take a look into your /system/addon.d folder and you'll find some scripts that are executed automatically upon ROM upgrades. They should serve as good examples already.

you should try it. when i migrated to unified storage many many years ago, it was a big hit. there is not big break up and nothing to invent, everything in android should work out of the box. take the plunge! and go for min sdcard. you dont need it! you have external SD if you need one. and in any case, if needed, using REPIT later to shrink data, and enlarge plus wipe sdcard should be almost instantaneous (no moving of any partition).

the only problem is TWRP:
-wipe data will wipe your sdcard too. but why would you wipe data? and you can adb shell to TWRP to manually wipe while keeping /data/media if you needed it.
-backup data will backup sdcard too. you'll need to backup to external storage of course.
-restore data will wipe and restore sdcard too.

i REALLY THINK an emulated official build for this device would be a great thing.

 

[Lanchon]

Does someone have any link that explain the "'addon' ROM flashing mechanism"?

PS: Does using emulated SD affect read/write performance?

addon is a channel between gapps (and whatever else) and custom roms. CM supports it. when you flash CM, it wipes system putting a new file system image there (in LP and later). but mysteriously if you had gapps before flashing, gapps remain. how's that?

well, CM runs hooks that might be present in /system/addon.d at several stages. i dont remember the details but at a minimum it's like this:
-run "before" hooks.
-install CM, completely wiping system.
-run "after" hooks.

so stuff you flash after flashing CM (eg: gapps) might add hooks in that folder. gapps in particular backs itself up from /system to somewhere in its "before" hook, then reinstalls itself from the backup in its "after" hook. where does it back itself up? i don't know, probably to /data, as REPIT gets /cache to be very small by default on many devices and i havent heard any complaints so far.

so when you flash stuff that uses addon, not only it flashes whatever it needs, but it also sets up hooks in the addon folder. and that's it.

for the life of me i can't understand why so many software doesn't use addon. examples that come to mind now: xposed and roots such as supersu. maybe rovo doesnt know about this mechanism? impossible, someone must have told him!

anyway, to get rid of existing addons, just format /system before flashing CM et al. pretty intuitive if you ask me.

 

[ale5000]

@Lanchon: Thanks.

Does it work on both flashing ROM update manually through recovery and normal OTA updates inside Android or not?

 

[Lanchon]

PS: Does using emulated SD affect read/write performance?

yes but only when apps access storage, not when the system accesses it. this is because access goes though FUSE.

google's FUSE use in android is an abomination. samsung and moto both knew this and fixed their OSes by moving the FUSE functionality to a kernel driver. google is backporting samsung's implementation (sdcardfs) to the standard android kernel, and my guess is that FUSE will be retired on the next android version.

FUSE can have a performance impact going from 15 to over 50%. large numbers correspond to small files or small file operations.


BUT... maybe FUSE is used for the non-emulated sdcard too, to synthesize permissions, so there might be no extra hit if you switch. this is at least very probable. i think FUSE is used for external microSDs these days, so the internal ones would be handled in the same way.

there is a native process in your phone called sdcard (that's the binary's name). if you find it, kill it. it shouldn't respawn i guess. after the kill, apps such as the gallery should loose access to the sdcard until you reboot. if this happens, it proves that FUSE is already being used and you wont get a performance hit from switching to emulated sd.

well... strike all that! i remember from the FPBug days, sdcard WAS in fact used, and when it died, access to the sdcard WAS indeed lost. this was already happening in KK and maybe android 4.3 too. so yes, you are already running FUSE.

 

[Lanchon]

@Lanchon: Thanks.

Does it work on both flashing ROM update manually through recovery and normal OTA updates inside Android or not?

it works on both methods. the flashable zip never knows if it's manual or OTA.

 

[the.gangster]

there is a native process in your phone called sdcard (that's the binary's name). if you find it, kill it. it shouldn't respawn i guess. after the kill, apps such as the gallery should loose access to the sdcard until you reboot. if this happens, it proves that FUSE is already being used and you wont get a performance hit from switching to emulated sd.

And I thought, simply looking at the mount command would already tell, wouldn't it?

i9100:/ # mount
rootfs on / type rootfs (ro,seclabel,relatime)
tmpfs on /dev type tmpfs (rw,seclabel,nosuid,relatime,mode=755)
devpts on /dev/pts type devpts (rw,seclabel,relatime,mode=600)
proc on /proc type proc (rw,relatime)
sysfs on /sys type sysfs (rw,seclabel,relatime)
selinuxfs on /sys/fs/selinux type selinuxfs (rw,relatime)
none on /acct type cgroup (rw,relatime,cpuacct)
tmpfs on /mnt type tmpfs (rw,seclabel,relatime,mode=755,gid=1000)
none on /dev/memcg type cgroup (rw,relatime,memory)
none on /dev/cpuctl type cgroup (rw,relatime,cpu)
/dev/block/mmcblk0p9 on /system type ext4 (ro,seclabel,noatime,us
/dev/block/mmcblk0p7 on /cache type ext4 (rw,seclabel,nosuid,node
/dev/block/mmcblk0p1 on /efs type ext4 (rw,seclabel,nosuid,nodev,
/dev/block/mmcblk0p10 on /data type ext4 (rw,seclabel,nosuid,node
/dev/block/mmcblk0p12 on /preload type ext4 (rw,seclabel,nosuid,n
tmpfs on /storage type tmpfs (rw,seclabel,relatime,mode=755,gid=1
/sys/kernel/debug on /sys/kernel/debug type debugfs (rw,seclabel,
/dev/fuse on /mnt/runtime/default/emulated type fuse (rw,nosuid,n
/dev/fuse on /storage/emulated type fuse (rw,nosuid,nodev,noexec,
/dev/fuse on /mnt/runtime/read/emulated type fuse (rw,nosuid,node
/dev/fuse on /mnt/runtime/write/emulated type fuse (rw,nosuid,nod
/dev/block/vold/public:179_11 on /mnt/media_rw/8C8D-EB5C type vfa
o)
/dev/block/vold/public:179_13 on /mnt/media_rw/E1C9-1514 type vfa
o)
/dev/fuse on /mnt/runtime/default/E1C9-1514 type fuse (rw,nosuid,
/dev/fuse on /storage/E1C9-1514 type fuse (rw,nosuid,nodev,noexec
/dev/fuse on /mnt/runtime/read/E1C9-1514 type fuse (rw,nosuid,nod
/dev/fuse on /mnt/runtime/write/E1C9-1514 type fuse (rw,nosuid,no
i9100:/ #

 

[Lanchon]

lol, i suppose, that too!

 

[Lanchon]

btw i found this lying on my harddrive. i dont know what it is, i dont remember. it is probably a attempt to hack official TWRP to work in emulated mode. i suppose it might not work because i dont remember success, but for sure it wont be anything malicious, so you guys can try it if you want.

 

[the.gangster]

btw i found this lying on my harddrive. i dont know what it is, i dont remember. it is probably a attempt to hack official TWRP to work in emulated mode. i suppose it might not work because i dont remember success, but for sure it wont be anything malicious, so you guys can try it if you want.

Thx. I would have taken a look at TWRP anyway so I'll have a look at that one as well.
If I only had more time....

 

[the.gangster]

.... and you can adb shell to TWRP ...

hm,
adb shelling to to both my i9100 devices doesn't work when in TWRP, as adb doesn't find my devices then.

error: no devices/emulators found

Tested that with four adb versions (1.0.31, 1.0.32, 1.0.35 and 1.0.36).
Also tested against TWRP 2.8.7.0.
MTP-mounting in TWRP works fine.
As I use it frequently it is needless to say that when booted into the ROMs (CM13+LineageOS14.1) both work fine with each of those adb versions.

Any idea? Special drivers needed for that?

 

[Lanchon]

hm,
adb shelling to to both my i9100 devices doesn't work when in TWRP, as adb doesn't find my devices then.

error: no devices/emulators found

Tested that with four adb versions (1.0.31, 1.0.32, 1.0.35 and 1.0.36).
Also tested against TWRP 2.8.7.0.
MTP-mounting in TWRP works fine.
As I use it frequently it is needless to say that when booted into the ROMs (CM13+LineageOS14.1) both work fine with each of those adb versions.

Any idea? Special drivers needed for that?

some adb operations require being root on the pc side. why? i don't know. it might be a strange safeguard in the adb client, i never understood that.

try:
sudo adb kill-server
sudo adb shell
if on linux.

on windows open an administrative cmd prompt and run those two same commands (without sudo of course).

 

[the.gangster]

Thank you but: No it's none of that. (Yes, I am still on Win7 x64 Pro)
Those were the first actions that I checked already even before trying with the other adb versions. And if you are switching from one version to another the old adb is also killed automatically.
It's not that simple.
But thanks anyway.