FORUMS
Remove All Ads from XDA

[MOD] Move all your internal and external data to encrypted, external SD

70 posts
Thanks Meter: 26
 
By Defier525, Member on 8th January 2015, 01:41 AM
Post Reply Email Thread
Announcement from Defier525: This tutorial shows how you can run all your internal and external data from removable, encrypted SD
Disclaimer:

THIS TUTORIAL IS PROVIDED ''AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES,
INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY
AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL I
BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
ARISING IN ANY WAY OUT OF THE USE OF THIS TUTORIAL, EVEN IF ADVISED OF
THE POSSIBILITY OF SUCH DAMAGE.


Motivation and idea

Starting with Android 3.x, the system theoretically supports encrypting the personal data on your device. But this has several caveats:

- it turns out that AOSP only supports encrypting /data, leaving other personal data (e.g. your pictures or storage from varios messengers) completely unencrypted on internal (emulated) or external SD storage.
- ever since, the partitioning of /data to a small, limited area leads to "no space left" issues (that's why many people go for app2sd etc.)
- in case your device suffers from physical damage, you usually loose all data you didn't backup previously.
- in case you want to sell or dispose your broken device you never know who will get access to your data anywhere in the future.
- securely wiping flash memory is complicated due to the fact that many controllers do not let you access their trim areas. Having built-in flash makes the situation even worse, especially because almost no manufacturer provides even a "manufacturer-declared-secure" wiping method - unlike some SSD drive vendors.

In this tutorial I will describe how I managed to move all personal data (probably excluding cached data, but this could be enhanced) to my external microSD card, overcoming these issues at the expense of a few drawbacks:

- (micro)SD cards are usually much slower than internal flash: boot time is about 2x slower and some apps feels a little slower, though not sluggish
- the mod requires root
- on most devices the mod also requires a modified boot image (due to change in initial RAM disk)
- it's pretty complicated to set it up (but following this guide can save you many efforts I already invested)
- mounting storage over MTP / USB is currently broken

Although this is more a proof-of-concept, I have been running this setup for a couple of days now as my daily driver without significant issues. I had some force-closes in Firefox, but I don't relate them to this mod and other browsers seem to work fine.


Enough said, let's get to work and move all personal data to your external SD card!

CURRENTLY, THIS GUIDE IS TARGETED AT ADVANCED USERS / DEVELOPERS / EXPERTS AND THUS ONLY COVERS THE ESSENTIAL BASICS, NOT THE DETAILS!

FOLLOWING THE PROCESS, YOU WILL LOOSE ALL YOUR PERSONAL DATA ON YOUR DEVICE! MAKE BACKUPS FIRST!!!


Compatibility

The tutorial is based on OmniROM 4.4 and TWRP. With not too much effort, it could be adopted to earlier Android Versions based on AOSP 4.0, probably even 3.0. (most likely, the location and internal layout of some configuration files will vary). Please note that I tried it only on one device, your milage may vary!


Step 1: Find out your partition layout

First off, you need to find out which devices are used for holding which data partitions. You can play around with "mount" or simply look in your fstab files which are located in /fstab.vendor in KK. While in some cases the devices are listed i.e. "by-name", the fstab files of custom recoveries often list the "real" device names. I.e. on my Xperia V they are as follows:


Code:
Internal Data partition: /dev/block/platform/msm_sdcc.1/by-name/Userdata
Internal, emulated SD card: /dev/block/mmcblk0p15
First partition of external SD card: /dev/block/mmcblk1p1
Step 2: Encrypt internal data

After backing up everything, run the usual Android encryption process so your data partition (in this case: /dev/block/platform/msm_sdcc.1/by-name/Userdata) will be encrypted by means of standard 128bit AES Android encryption.

Step 3: Extract the cryptofooter

Boot into TWRP recovery (don't unlock your /data if prompted!) and connect via ADB as root. Usually, the crypto footer is stored in the last 16384 bytes of your partition (but check your fstab to be sure). Now determine the size of your partition with:

Code:
fdisk -l /dev/block/platform/msm_sdcc.1/by-name/Userdata
In my case:

Code:
Disk Userdata: 2147 MB, 2147483648 bytes
255 heads, 63 sectors/track, 261 cylinders, total 4194304 sectors
Units = sectors of 1 * 512 = 512 bytes
...
In my case, the size is 2147483648 bytes. Subtracting 16384 bytes, you know the starting offset for the extraction. Now extract the footer using dd:

Code:
dd if=/dev/block/platform/msm_sdcc.1/by-name/Userdata of=/cryptofooter bs=1 seek=2147467264 count=16384
Please note, that due to a limitation in dd which supports only up to 2GB this may file - you will need a modified busybox dd command in this case - or just download the file to your UNIX machine and use the normal dd from there.


Step 4: Prepare the external SD card:

We de not want to partition the SDcard, thus we will not put any partition table on it. Nevertheless, we need to determine its exact size using fdisk:

Code:
fdisk -l /dev/block/mmcblk1p1

Disk mmcblk1: 31.9 GB, 31914983424 bytes
255 heads, 63 sectors/track, 3880 cylinders, total 62333952 sectors
Units = sectors of 1 * 512 = 512 bytes
...
Again, subtract the 16384 bytes from 31914983424 and use dd to write the cryptofooter on the end of SD:

Code:
dd if=/cryptofooter of=/dev/block/mmcblk1= bs=1 skip=31914967040 count=16384
Again, this might fail due to dd limitations.

For later testing, we copy over the existing (encrypted) data partition to the SD card:

Code:
dd if=/dev/block/platform/msm_sdcc.1/by-name/Userdata of=/dev/block/mmcblk1p1
Finally, we can wipe the internal data partition:

Code:
dd if=/dev/zero of=/dev/block/platform/msm_sdcc.1/by-name/Userdata

Step 5: Modifying fstab and init files:

Now we will need to modify the fstab files and init so they will reflect the new layout. On most devices, this requires compiling a new rootfs. This can be either done by uncompressing your boot.img, making the changes and recompressing it or recompiling it from sources. The modified boot.img has to be flashed back to the device, usually via fastboot.

In particular, we need to tell Android not to use the internal userdata partition at all, to mount the emulated sd card as usual and to use the external storage for /data using an encrypted ext4 filesystem. In my case, the modified fstab.qcom looks like this:

Code:
/dev/block/mmcblk1     /data             ext4        noatime,nosuid,nodev,barrier=1,data=ordered,noauto_da_alloc  wait,check,encryptable=footer,length=-16384
/devices/platform/msm_sdcc.1/mmc_host/mmc0          auto              ext4        defaults                                                     voldmanaged=sdcard0:15,nonremovable,noemulatedsd
#/devices/platform/msm_sdcc.3/mmc_host/mmc1          auto              auto        defaults                                                     voldmanaged=sdcard1:auto
/devices/platform/msm_hsusb_host                    auto              auto        defaults                                                     voldmanaged=usbdisk:auto
You will also need to modify the twrp.fstab file accordingly.

We also need to modify init (in my case through sony.init.rc) to avoid getting the external sd card being treated as sdcard during boot by uncommenting the relevant lines:

Code:
    chmod 0701 /mnt/media_rw
    mkdir /storage 0775 system system
    mkdir /mnt/media_rw/sdcard0 0700 media_rw media_rw
#    mkdir /mnt/media_rw/sdcard1 0700 media_rw media_rw
    mkdir /mnt/media_rw/usbdisk 0700 media_rw media_rw
    mkdir /storage/sdcard0 0775 system system
#    mkdir /storage/sdcard1 0775 system system
    mkdir /storage/usbdisk 0775 system system

    export EXTERNAL_STORAGE /storage/sdcard0
#    export SECONDARY_STORAGE /storage/sdcard1

    # for backwards compatibility
    symlink /storage/sdcard0 /sdcard
    symlink /storage/sdcard0 /mnt/sdcard
#    symlink /storage/sdcard1 /ext_card
#    symlink /storage/sdcard1 /mnt/ext_card
    symlink /storage/usbdisk /usbdisk
    symlink /storage/usbdisk /mnt/usbdisk
Step 6: First booting test

With all this set up, we can now boot again into our system. It should ask you for your encryption password (it's the same as you've set it in step 2) and you should notice that it runs a little slower - because SD cards are usually slower than internal flash. You will notice that the size of the internal space and external storage has not changed - we will fix this in the next step.


Step 7: Tweak the cryptofooter and reformat the SD card:

Now comes the fun part: Android determines the size of encrypted filesystems not by information from the filesystem, but by the size as it is stored in the cryptofooter. Therefore, we need to modify the information there. The cryptofooter is a complicated mess. Luckily, there is a great guide which describes its structure much better than the source code itself:

http://nelenkov.blogspot.de/2014/10/revisiting-android-disk-encryption.html

You will need to use a hex editor to edit the from bytes between Offset 0000:0018 and 0000:001B to match the size of your SD card. But please leave enough bytes at the end, at least for the cryptofooter!

Write back your modified cryptofooter as outlined in step 4 and boot into TWRP. Unlock your /data (which is now served from SD), connect via ADB and check the size of your encrypted device using fdisk:

Code:
fdisk -l /dev/block/dm-0
It should reflect the size as you modded it in your cryptofooter. Now, reformat the partition so fills the whole space:

Code:
make_ext4fs /dev/block/dm-0

Step 8: Adjust internal SD.

Now, you are ready to reboot into your system serving its freshly wiped, encrypted data from your external SD! Be amazed by the space you now have for your internal Userdata! Unfortunately, it will still serve "internal SD" from internal flash. First, create a directory on the sdcard to hold it using a root shell:

Code:
# mkdir -p /data/sdcard0
I tried various things to fix this already in the boot process, but somehow vold always got in the way or the device was mounted too early. Also, the media server indexes the internal flash instead of the stuff on the external SD. Therefore, I bind-mount the sdcard0 after every boot and rerun the media server by running the following commands in a root shell:

Code:
# busybox mount -bind /data/sdcard0 /mnt/media_rw/sdcard0
# am broadcast -a android.intent.action.MEDIA_MOUNTED -d file:///storage/sdcard0
Now, you can copy back all your data from the backup to the device and enjoy your new setup!


Final notes:

- Suggestions for correctly mounting the sdcard0 inside data at boot time without the bind-mounting is warmly welcome.
- Some (Stock) ROMs allow you to encrypt both SD "cards", but the implementations I have seen so far seem to be vendor-specific and closed source. Although I noticed that encrypting internal SD seems to work with AOSP out-of-the-box on some devices without external SD (e.g. Nexus S) I didn't find any Custom ROM for my Xperia V which worked with encrypted any of the SDs when putting the "encryptable=sdcard" flag in the fstab file: The storage seems to have been encrypted, but Android simply fails to mount it.

XDA:DevDB Information
All data 2 encrypted, removable SD, Tool/Utility for the Android General

Contributors
Defier525, Defier525

Version Information
Status: Alpha

Created 2015-01-08
Last Updated 2015-01-08
The Following 4 Users Say Thank You to Defier525 For This Useful Post: [ View ] Gift Defier525 Ad-Free
 
 
17th December 2017, 03:02 PM |#2  
Senior Member
Flag Toba Tek Singh
Thanks Meter: 241
 
More
It's old though, I was experimenting on old devices; a KK (Nokia X2) and a LL (QMobile Z8). In summary, what this guide tells is:
  1. Move /data partition (including internal sd card: /data/media) to external sd card
  2. Extract crypto-footer from encrypted internal /data partition
  3. Write crypto-footer to new /data partition on external sd card
I have tested this on QMobile with Stock Android 5. This device has a bricked eMMC and all my partitions reside on external sd card including /data. So, I tried encrypting using built-in "Encrypt Phone" Settings which fails with a simple reset of UI.
On my other phone with KK (CM-11), to overcome the shortage of internal memory, I Disabled Internal Memory (DIM) i.e. use device in "Physical Primary Only" mode instead of "Emulated primary, physical secondary". I tried same "Encrypt Phone" Settings which failed in same manner. So, I was wondering where this "encryptable=sdcard" thing works? I edited fstab in boot.img to add this flag but that makes no difference.
My second question, will this dumped crypto-footer hack work with both approaches? i.e. with DIM and with whole /data on external sd card?
Currently I'm using another working solution; an encrypted (EncFS) directory placed on external sd card. An initd script (replacing /system/bin/sdcard) mounts this directory using FUSE kernel module on the default sd card path i.e. /storag/sdcard0. Since "dm-crypt" (kernel module) is the default FDE used by AOSP, there must be a way out to use dm-crypt for SD card or any other (than /data) partition encryption like this.

EDIT:
One possible reason for in-encrypt-able /data partition on external SD card is the absence of footer space at the end of partition i.e. filesystem must be smaller than partition size by at least 16KB.
Reference:
Post Reply Subscribe to Thread

Guest Quick Reply (no urls or BBcode)
Message:
Previous Thread Next Thread
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes