[REF] LVM Partition Remapping

Search This thread

theradec

Senior Member
Jan 16, 2008
85
6
Dang !
The LVM File I have downloaded if for Find7, not find7a
(as what the red message said when I tried to flash setuplvm_find7_FULL_WIPE_09012014.zip on my 7a
Could someonekindly direct me towards the Find7a download link (search engine is down atm, so can't look for it in Omni Thread)
 
Last edited:

maxwen

Senior Member
Jun 10, 2012
8,068
10,288
Dang !
The LVM File I have downloaded if for Find7, not find7a
(as what the red message said when I tried to flash setuplvm_find7_FULL_WIPE_09012014.zip on my 7a
Could someonekindly direct me towards the Find7a download link (search engine is down atm, so can't look for it in Omni Thread)

there is no diff between 7a and 7s
 

theradec

Senior Member
Jan 16, 2008
85
6
When I try to flash setuplvm_find7_FULL_WIPE_09012014.zip on my Find7a, I have this error message
This package is for "FIND7" devices
This is a find7a
Error executing updater binary in zip '/external_sd/Roms/LVM/setuplvm_find7_FULL_WIPE_09012014.zip
Failed


TWRP is 2.8.0.1
 

Entropy512

Senior Recognized Developer
Aug 31, 2007
14,088
25,086
Owego, NY
Hello, a quick question based on information shared by aut0mat3d in the Oppo forum around basebands where he says ".........are the partitions i am talking about : modem: baseband and low level drivers, sbl1: Qualcomm SBL1 bootloader image and rpm: Resource and Power Manager. The android system have to communicate with this drivers which is done by propreitary binary modules - called "blobs".
If you flash or update ColorOs, these partitions would be updated in any way. Custom roms do not come with these updates, they resist on the driver-state the phone was when the custom rom was flashed. So, if you updated to a newer ColorOs you will have another base than the custom rom is made for. This can cause problems: If you have different low level drivers as the Rom depends on some functions will not wok correctly......" So if am on color os 1.2.6 i and shift to lvm based omni rom, do i have to flash the corresponding modem too? which in this case would be Omni Rom: Q_V1_P14 Modempackage_v1.2.0i_Find7a.zip. Some direction would help in understanding modems cos clearly "Baseband info shown in Device Info is no readout value, it is set by rom source."
This isn't really related to LVM, but in general:
Any ColorOS 1.2.x baseband should work fine with Omni.
JB 2.x basebands may work but aren't particularly tested
The basebands/bootloader of KK seem to have compatibility issues that need to be resolved

And I really need to update my baseband packages.
 
  • Like
Reactions: WhoIsJohnGalt1979

theradec

Senior Member
Jan 16, 2008
85
6
Is there anything I could do to fix my Error executing updater binary ?

* Storage was not unified (Was thinking about it, but Entropy warnings in OP kinda cool me off and showed me how LVM is safer).
* Did a 4way wipe, and tried just installing lvm-> no luck.

Do I need to reverse to the Entropy512's lvm aware TWRP or is 2.8.0.1 fine ?
Can I flash Entropy512 twrp with TWRP manager or is it better to push it ?
 

magostinelli

Senior Member
Mar 2, 2012
115
13
Prato
Whats is the disk space after LVM?

What size has each mount point?
(Posting the result of df command must be appreciated)
 

beton87

Senior Member
Jan 7, 2013
482
116
Wrocław
From my Find7a
Code:
shell@FIND7:/ $ df
Filesystem               Size     Used     Free   Blksize
/dev                   901.2M   128.0K   901.1M   4096
/sys/fs/cgroup         901.2M    12.0K   901.2M   4096
/mnt/asec              901.2M     0.0K   901.2M   4096
/mnt/obb               901.2M     0.0K   901.2M   4096
/mnt/fuse              901.2M     0.0K   901.2M   4096
/system                  1.3G   671.4M   629.7M   4096
/data                   12.2G     8.8G     3.4G   4096
/cache                 503.6M     8.1M   495.5M   4096
/persist                 4.9M     4.1M   788.0K   4096
/firmware               64.0M    54.7M     9.3M   16384
/data/qcks              20.0M     0.0K    20.0M   4096
/mnt/shell/emulated     12.2G     8.8G     3.4G   4096
/mnt/media_rw/sdcard1: Permission denied
/storage/sdcard1        59.5G    18.1G    41.3G   131072

@Entropy512
I see new version of TWRP is comming. Could you post a link to latest version of LVM setup zip in this thread download section or a link to it in OP? It would be usefull to have here quick guide/link for user to setup/revert LVM configuration.
 
Last edited:

Entropy512

Senior Recognized Developer
Aug 31, 2007
14,088
25,086
Owego, NY
From my Find7a

@Entropy512
I see new version of TWRP is comming. Could you post a link to latest version of LVM setup zip in this thread download section or a link to it in OP? It would be usefull to have here quick guide/link for user to setup/revert LVM configuration.

I'm going to try and figure out why 2.8.0.1 is problematic, although I have a guess:
There were some issues ayysir from PA had with the lvmsetup ZIP not properly extracting the LVM config file and binary from the ZIP. So it would only work if recovery had the LVM config file and ZIP.

Part of the "final" LVM support in TWRP 2.8.0.1 moved some binaries around, which may be why lvmsetup fails.

The ADB instructions in one of the commit messages linked from the OP will work. I'll try and clean things up this weekend to get rid of the lvmsetup incompatibility issues.
 

jaxenroth

Senior Member
Apr 28, 2011
814
152
Langhorne, PA
Little help please

Was running Omni w/ unified, twrp 2.8.0.1 Sold phone and went to go back to standard partitioning, did wipe and reloaded rom now lost internal storage. Any help would be great. Thx.
 

beton87

Senior Member
Jan 7, 2013
482
116
Wrocław
@Entropy512

Could write how to restore stoc partitions after LVM? I tried to flash ColorOS recovery and wipe data, but the it seems I lost internal sdcard partition. How to get back stock partitions?
 

beton87

Senior Member
Jan 7, 2013
482
116
Wrocław
thanks mate, totally forgot about that one.

So I went back and through with LVM setup process. TWRP 2.8.0.1 has issues, I used twrp_lvm_09012014.img i had downloaded earlier, so I have no link. This one works with setting up partitions properly
 
Last edited:

theradec

Senior Member
Jan 16, 2008
85
6
thanks mate, totally forgot about that one.

So I went back and through with LVM setup process. TWRP 2.8.0.1 has issues, I used twrp_lvm_09012014.img i had downloaded earlier, so I have no link. This one works with setting up partitions properly

What did you use to downgrade your TWRP from 2.8.0.1 to twrp_lvm ?
TWRP Manager ?
Flashify ?
other ?
 

Entropy512

Senior Recognized Developer
Aug 31, 2007
14,088
25,086
Owego, NY
@Entropy512

Could write how to restore stoc partitions after LVM? I tried to flash ColorOS recovery and wipe data, but the it seems I lost internal sdcard partition. How to get back stock partitions?

Hmm. That's interesting that a ColorOS factory reset doesn't format the internal SD. "stock" recoveries should wipe everything.

Another option is with any LVM-aware TWRP, esp. TWRP 2.8.0.1: If the recovery detected LVM, format the raw_sd and raw_data partitions. If the recovery did not detect LVM, just format the sdcard and data partitions.

Actually, even a non-LVM-aware TWRP can format data and sdcard. But "downgrading" back to a split partition when you're in an LVM config can be done by formatting raw_sd and raw_data and then rebooting recovery.

(raw_sd and raw_data only appear when TWRP boots and sees an LVM configuration)
 

beton87

Senior Member
Jan 7, 2013
482
116
Wrocław
But "downgrading" back to a split partition when you're in an LVM config can be done by formatting raw_sd and raw_data and then rebooting recovery.
(raw_sd and raw_data only appear when TWRP boots and sees an LVM configuration)

This was what I needed.

Oppo stock recovery wiped /data and lost /sdcard
It seems that it was the same in Samsung - /sdcard was always intact when resetting/wiping data.

non-LVM TWPR didn't mount partitions properly and didn't let me to format /sdcard
some CWM based recovery I found on XDA froze on "formatting /data" or "formatting /recovery"

I knew that simple format of both partitions should do it but it seems it would only work with LVM aware TWPR and the key was to format "raw" partition.
 

Entropy512

Senior Recognized Developer
Aug 31, 2007
14,088
25,086
Owego, NY
This was what I needed.

Oppo stock recovery wiped /data and lost /sdcard
It seems that it was the same in Samsung - /sdcard was always intact when resetting/wiping data.

non-LVM TWPR didn't mount partitions properly and didn't let me to format /sdcard
some CWM based recovery I found on XDA froze on "formatting /data" or "formatting /recovery"

I knew that simple format of both partitions should do it but it seems it would only work with LVM aware TWPR and the key was to format "raw" partition.
Oddly, the "raw" partition should only appear of an intact LVM config is present, so if you formatted /data, LVM-aware TWRP should still behave as if it is a non-LVM config. (e.g. you would have to format /sdcard - or I think it's called /internal_sd ? Don't have my non-LVM reference device with me)

OEM recoveries are SUPPOSED to fully wipe the device when doing a factory reset (there was a bit of controversy recently regarding the OnePlus not doing this). Although I think SOMETIMES on the Samsungs, whether /sdcard was formatted depended on if you did a wipe in recovery vs. doing it from the Settings menu. (Even if sdcard is hosed, you should be able to boot the device enough to wipe in Settings.)

I also got some more info this morning on why lvmsetup is failing on TWRP 2.8.0.1 - someone who had those issues finally pulled /tmp/recovery.log, and it seems that even if you manually unmount /sdcard, it somehow remains mounted, which causes lvmsetup to fail to create a PV on the sdcard partition. I'm going to look into why this happens later this week.
 

Entropy512

Senior Recognized Developer
Aug 31, 2007
14,088
25,086
Owego, NY
OK. There's a minor bug in recent TWRP releases where it seems to insist on constantly remounting the internal SD. It also double-mounts the internal SD, also mounting it to /and-sec

This is what causes the following failure in recovery.log:
Code:
  Can't open /dev/block/platform/msm_sdcc.1/by-name/sdcard exclusively.  Mounted filesystem?
run_program: child exited with status 5
when setuplvm attempts to pvcreate on the sdcard partition

Workaround: Explicitly unmount /and-sec in the recovery script.

Attached is:
Updated setuplvm which works with recent TWRP releases (should include 2.8.0.1) - it REQUIRES an LVM-aware recovery now, and no longer flashes its own LVM-aware recovery
A repost of removelvm which should be self-explanatory and should work with nearly any recovery other than stock

Later this week I'm going to work on splitting this into two threads: Reference for developers, and reference for users. This thread was intended to be the developer reference for people wanting to implement LVM into their projects. :)

EDIT: REMEMBER, AS HAS BEEN STATED BEFORE: YOU MUST FLASH THIS FROM THE EXTERNAL SD. FLASHING FROM THE INTERNAL SD OR ADB SIDELOAD WILL FAIL.
 

Attachments

  • setuplvm_find7_FULL_WIPE_10282014.zip
    862.5 KB · Views: 2,391
  • removelvm_find7_FULL_WIPE.zip
    178.2 KB · Views: 1,567
Last edited:

Top Liked Posts

  • There are no posts matching your filters.
  • 29
    OK, this thread is going to be a work in progress, intended to serve as a reference for the work I've been doing on LVM partition remapping.

    My work was done initially on a Find 7, but this should eventually be usable on many other devices (I have the Find 5 and N1 in mind for when I return from vacation). Also, this would not have been possible without the work Steven676 did years ago on the Nexus S, which has been used by all AOSP-derivative projects to support the Samsung Aries (Galaxy S) family for quite some time now.

    The current state of things is that the patches are solid and work very well for the system side of things, but there is still a bit of work needed on the recovery side of things. This is due to TWRP having an architectural limitation I need to work on - Whether a device uses emulated storage or not is set at compile time, which is a problem if your design requires automatic detection of configuration at run time.

    One of the key design goals here was to support both normal and LVM configurations automatically with a single build that detects which configuration is present on a device at run time.

    A second key design goal was that the underlying partition table of the device is not touched in any way. Touching the partition table of a mobile device in the field is a fundamentally dangerous operation, as many partitions contain data that is device-unique or will render a device unbootable if altered. Recovery methods that involve DDing partition images to nonstandard partitions is asking for trouble due to typos... There's no protection against a user typoing the name of a critical partition.

    Initially, I'm going to dump the contents of an email I wrote to someone giving them documentation on how to integrate LVM into their project. Over time I'll clean up and reorganize this post, including adding some more links. Also, since this email was written, I've added a LOT of comments to each patch explaining what is going on.

    For additional documentation, especially a more user-oriented view of things (such as how to set this up if you want to use it with Omni nightlies) - see the Omni nightlies thread on XDA.

    So here goes:

    How it's implemented - the complete patch set is at:
    https://gerrit.omnirom.org/#/q/topic:find7_lvm - Expect this to periodically change as work on this feature continues (Note: All patches required to support nightly builds of Omni have been merged. At this point, all remaining work that I expect is on polishing up TWRP.)

    With the rest of this post, I'll talk about each individual patch and what it does.

    https://gerrit.omnirom.org/#/c/9273/ - This is a patch against frameworks/base which adds an alternative to storage_list.xml called storage_list_lvm.xml - The frameworks will choose storage_list_lvm.xml instead of storage_list.xml if the property ro.lvm_storage is set to 1 - The device init scripts will set this property if they detect an LVM configuration.

    https://gerrit.omnirom.org/#/c/9207/ - This is an Omni-specific patch. Omni builds for both the Find 7 and OnePlus One (also known as find7op) and both share a common device tree. The LVM patches do not apply to the find7op, so we move init.recovery.rc out of the msm8974-common tree - You likely don't have to worry about this unless you also have a -common tree for find7 and find7op

    https://gerrit.omnirom.org/#/c/9276/ - Normal Android kernel ramdisks do not include busybox or any form of shell, making it impossible to run shell scripts without /system mounted. Since we need to run a shell script prior to mounting partitions, we need to add busybox to the ramdisk. This patch does that. For legal reasons you may wish to replace busybox with system/core/toolbox and system/core/sh - I have not tried doing so. If you choose to stay with busybox, you will have to provide the busybox source code in order to comply with the GPL.

    https://gerrit.omnirom.org/#/c/9205/ - This adds the LVM binary and LVM configuration file to the ramdisks of both normal boot and recovery. This patch does not actually begin doing anything with the binaries, I separated it out from the other patches as a way to keep things organized so I could start working with the binaries when I began this project. The original source code and documentation for the binary is at https://github.com/steven676/android-lvm-mod

    One change I made in lvm.conf that differs from the Samsung aries family (galaxysmtd, fascinatemtd, captivatemtd, etc.) is that I changed the filter line to only allow the userdata and sdcard partitions. This prevents LVM's vgscan from accidentally determining another partition is a physical volume, and also prevents users from accidentally running pvcreate on a critical partition.

    https://gerrit.omnirom.org/#/c/9206/ - This is where all of the "heavy lifting" is done. I'm going to work on adding more comments to the init scripts and shell scripts to document them tonight and tomorrow, but I'll try to explain things here.

    Android's init system is a bit limited in that it's very difficult to have conditional behavior defined in init.rc - which appears to be why Qualcomm loves to use shell scripts called from init. Similarly, much of the LVM magic happens in three shell scripts (which execute at three different phases within the boot sequence).

    In the early-init phase, the two "wait" blocks ensure that the underlying block devices are ready before vgscan/vgchange are called. This will probably slow down booting by a few fractions of a second unfortunately.

    vgscan will scan the volumes defined in lvm.conf (in this case, only the userdata and sdcard partitions) for LVM physical volumes. If LVM physical volumes are detected and form a proper volume group, vgscan will create appropriate device nodes. With the configuration I'm using, the device node will be /dev/lvpool/userdata - which consists of a single logical volume that merges the sdcard and physical userdata physical volumes (partitions). The configuration of lvm.conf prevents LVM commands (especially pvcreate) from altering partitions we don't want to alter. If someone accidentally tries to, for example, run pvcreate on the system partition, it will give an error indicating that the partition was not part of the filter.

    vgchange will activate the physical volumes detected by vgscan

    lvm_init.sh will check to see if /dev/lvpool/userdata exists, and copy fstab.qcom.lvm to fstab.qcom, init.fs.rc.lvm to init.fs.rc, and twrp.fstab.lvm to twrp.fstab if it does. If it does not, it selects fstab.qcom.std, etc.

    In the "on init" section, the init script exports all environment variables from init.fs.rc, and creates all storage-related directories and symlinks needed for both configurations (except for when they conflict). lvm_symlinks.sh will create directories/symlinks that must be configuration-specific. Just like lvm_init.sh - it decides what to do based on whether /dev/lvpool/userdata exists

    In the "on fs" section - we do an SELinux restorecon on /dev/mapper/lvpool-userdata (/dev/lvpool/userdata would probably work here too). If it doesn't exist, this will fail gracefully without causing any issues.

    In "on early-boot" - lvm_setprop.sh uses /system/bin/setprop to set ro.lvm_storage to 0 or 1 depending on the detected configuration. The property service is not available until early-boot - so this cannot be in lvm_init.sh or lvm_symlinks.sh This propery is used by the frameworks/base patch above to determine which storage_list to choose.

    At the end of the init.qcom.rc, the fuse daemon for emulated storage is added for all configurations. (I could not figure out a good way to make this conditional based on whether LVM was present or not). In a non-LVM configuration, it runs but is harmless - it maps /data/media (which is empty) to /mnt/shell/emulated (which nothing is looking at due to the environment variables and symlinks set in the "on init" section )

    You will probably notice that Omni's standard storage configuration is fairly different from ColorOS - this is due to the way KitKat storage works, but it allowed us to get away without using Oppo's ext4 permissions hacks in our kernel (by remapping permissions instead, in a manner similar to how the emulated storage system works) The way we handle our /sdcard partition does interoperate without issues with the ColorOS approach.

    https://gerrit.omnirom.org/#/c/9279/ is a patch specifically for TWRP. TWRP currently determines whether to use emulated storage (/sdcard on /data/media) at build time instead of at run time. Until I have time to fix this, the patch here operates as a workaround. It is similar to the behavior of the fuse sdcard daemon in the previous patch - it maps /data/media to /sdcard whether the configuration is actually emulated storage or not. If the device is not using emulated storage (LVM), mapping of /data/media to /sdcard is still mostly harmless. However it does result in undesirable changes to TWRP's user interface. DO NOT USE THIS APPROACH IN PRODUCTION RELEASES. It's a horrible hack. You'll need to figure out how to properly do /data/media handling depending on whether LVM is present or not based on how your own recovery architecture works.

    https://gerrit.omnirom.org/#/c/9281/ adds "raw" sdcard and userdata partition entries to the partition table for the LVM configuration. This allows a user to return their device to a standard configuration by formatting the underlying sdcard and userdata partitions directly, instead of the removelvm ZIP at the beginning of this email. - To be abandoned, this patch was squashed into 9206
    13
    FAQ

    Q: Coldbird already had repartitioning support. Why did you create this different approach?
    A: Even before he started work, I strongly recommended that he not touch the partition table of the device. It's a really bad idea and is fundamentally dangerous. It's pure luck that someone hasn't hardbricked yet. (A number of people have come close.) If you read through his thread and the ColorOS 2.0.2 thread, you'll see that the repartitioning approach fails frequently, and in multiple ways. (Missing partition contents, partition table ending early, etc. The latter is really scary, one person had the process fail at mmcblk0p19 - what if someone else's partition table write operation aborted even earlier?.) Also, nearly everyone that has implemented support for that approach has needed a separate build to support it. (Oppo is the first to manage autodetection.) I also provided him all of the reference information from Steven676's work.

    LVM is far safer. The underlying partition table is not touched in any way. Instead, LVM remaps sectors on the fly so that two partitions that are not adjacent to each other on the physical storage appear as a single contiguous partition to the filesystem drivers. Linux has supported LVM for on the order of a decade, if not more. I've been using LVM on my file server since 2006. (Yes, the system is 8 years old and still working other than needing a new power supply after a thunderstorm. Nothing to do with LVM. :) ) In addition, the lvm.conf configuration used here provides protection against accidental typos causing damage. Undoing the changes is as simple as doing a wipe of /data and /sdcard from any standard recovery and can be done in seconds, not of running a special batch file that runs a bunch of fastboot commands and takes 4-5 minutes. Similarly, the LVM setup process currently described in the Omni thread involves flashing a single ZIP from recovery that takes only 10-15 seconds, and most of that process is flashing an LVM-aware recovery. (The only limitation currently is that the ZIP must be on external storage - USB OTG or MicroSD)

    To put it simply, it Just Works. No need to back up a pile of partitions other than /data and /sdcard because those partitions are never touched or altered.

    Q: I have a device with a ridiculously oversized /system partition, can I get some of that back for /data?
    A: Yes, you can. Add the physical /system partition to the lvm.conf filters and add it to the lvpool when creating it, then create a smaller /system LV out of this big pool. (see updater.sh in device/samsung/aries-common/ of any AOSP-derivative for hints here.) Be careful though - leave enough spare space for growth (new Android versions, etc.) While it should be possible to use some of the LVM tools along with ext4 resize tools to reorganize the LVs without wiping, this is very difficult and you'll probably have to make users wipe /data if you want to alter /system.
    9
    OK. There's a minor bug in recent TWRP releases where it seems to insist on constantly remounting the internal SD. It also double-mounts the internal SD, also mounting it to /and-sec

    This is what causes the following failure in recovery.log:
    Code:
      Can't open /dev/block/platform/msm_sdcc.1/by-name/sdcard exclusively.  Mounted filesystem?
    run_program: child exited with status 5
    when setuplvm attempts to pvcreate on the sdcard partition

    Workaround: Explicitly unmount /and-sec in the recovery script.

    Attached is:
    Updated setuplvm which works with recent TWRP releases (should include 2.8.0.1) - it REQUIRES an LVM-aware recovery now, and no longer flashes its own LVM-aware recovery
    A repost of removelvm which should be self-explanatory and should work with nearly any recovery other than stock

    Later this week I'm going to work on splitting this into two threads: Reference for developers, and reference for users. This thread was intended to be the developer reference for people wanting to implement LVM into their projects. :)

    EDIT: REMEMBER, AS HAS BEEN STATED BEFORE: YOU MUST FLASH THIS FROM THE EXTERNAL SD. FLASHING FROM THE INTERNAL SD OR ADB SIDELOAD WILL FAIL.
    6
    Nice work, I hope all the patches can be widely used on some other devices and other roms.

    Yup. I know Andre from PA was working on it last week but I haven't heard from him lately.

    My priority when I return from vacation will be fixing up the TWRP side of things. It's working for now, but the user interface on non-LVM configs is a little funky thanks to RECOVERY_SDCARD_ON_DATA being compile time. This has never been a problem before since a single TWRP binary never had to support two different configurations before. I plan on either doing a property-based approach or fstab-based like CWM. (It should be possible for someone to make a CWM build that automatically detects configuration without any modifications to CWM, based on reading the code - but I haven't tried it myself.)

    Once TWRP is in better shape, I plan on doing the Find 5 and N1. These will have the challenge of not having a MicroSD slot, so I may have to change TWRP so that it use /tmp instead of /sdcard when doing "adb sideload", or at least gives the user that option.
    6
    Thanks for your answer.

    Seems I misunderstood the way it's implemented here. All space is allocated to /data? So there's no more internal sdcard right?
    But in that case an external sdcard is mandatory. How is it managed when there's no sdcard?

    Enjoy!

    Android has supported emulated storage (where /data/media is mapped to /sdcard with a special FUSE daemon that makes /sdcard have DOS-like permissions despite an underlying ext4 partition) since ICS. It's pretty much the standard in all new devices - the Find 7 is to my knowledge the only device launched in 2014 not to use emulated storage. Most devices in 2013 also did - Oppos were again the rare exception.

    As I understand it - for some reason Chinese users prefer the legacy pre-ICS partitioning scheme. My guess is due to UMS vs. MTP - MTP is required for access to emulated storage, UMS can't be used, but a lot of older desktop OSes have issues with MTP. So Oppo finds themselves in conflict between their home market (China) and expanding in the West. That said, the Find 7 was kind of a screwup in achieving this goal, since the internal sdcard partition was ext4 which meant UMS was a no-go for it.