Welcome to XDA

Search to go directly to your device's forum

Register an account

Unlock full posting privileges

Ask a question

No registration required
Post Reply

[INFO]What is EpicMTD, and what you need to know.

OP AproSamurai

24th December 2011, 09:14 PM   |  #51  
toadlife's Avatar
Recognized Developer
Flag Lemoore, CA
Thanks Meter: 1,015
 
1,202 posts
Join Date:Joined: Aug 2008
Donate to Me
More
Quote:
Originally Posted by codest3r

If you were to change the mapping (i.e. modify offset or size of a few partitions in samsung_epic.h) - compile and flash your kernel - what would happen? Assuming you didn't change ROM's - just kernels. My guess is nothing if your phone is already converted to MTD. The kernel should mount the partitions just fine without the map, right?

I just tested it out. I flashed my ROM using this kernel with this partition layout...

Code:
Filesystem                Size      Used Available Use% Mounted on
/dev/block/mtdblock4     25.0M      2.3M     22.7M   9% /cache
/dev/block/mtdblock3    695.0M    407.0M    288.0M  59% /data
/dev/block/mtdblock2    250.0M    206.2M     43.8M  82% /system
I then flashed a kernel with the following partition layout...

Code:
Filesystem                Size      Used Available Use% Mounted on
/dev/block/mtdblock4     50.0M      2.3M     47.7M   5% /cache
/dev/block/mtdblock2    250.0M    206.2M     43.8M  82% /system
/dev/block/mtdblock3    670.0M    407.0M    263.0M  61% /data
...and the partitions mounted just fine, with different sizes!

Unfortunately, after flashing Tortell's MTD kernel which has the "stock" partition layout (a different size for /system), I got this message when trying to mount system partition:

Code:
Failed to mount /dev/block/mtdblock2 on /system: Invalid argument
/data and /cache still mounted.

Flashing back the original kernel allowed my to mount to all partitions again.


Quote:

However, if your device wasn't converted to MTD yet the flashable ZIP for a ROM (with kernel) would contain the script instruction to check and if not converted -> execute conversion... at which point it would look for these values in the kernel. Right?

No, it just checks to see if the device is MTD or BML. If it is BML, it flashes the MTD kernel and reboots. I added a part to that script which checks the partition sizes and if they don't match up with what they should be, it does the same thing - flashes the kernel and reboots. I inserted the code into updater.sh. This could be put into a seperate shell script. Here is the code. It's adapted from the code that checks for BML/MTD.

Code:
# check to see if partition sizes are correct
/tmp/busybox mount -t yaffs2 /dev/block/mtdblock2 /system
/tmp/busybox mount -t yaffs2 /dev/block/mtdblock3 /data
if (! df /dev/block/mtdblock2 | grep "256000") || (! df /dev/block/mtdblock3 | grep "711680"); then
# Our partition sizes are not correct

# unmount, format and mount system
/tmp/busybox umount -l /system
/tmp/erase_image system

# unmount and format cache
/tmp/busybox umount -l /cache
/tmp/erase_image cache

# unmount and format datadata
/tmp/busybox umount -l /data
/tmp/erase_image userdata

# make sure sdcard is mounted
if ! /tmp/busybox grep -q /mnt/sdcard /proc/mounts ; then
/tmp/busybox mkdir -p /mnt/sdcard
/tmp/busybox umount -l /dev/block/mmcblk0p1
if ! /tmp/busybox mount -t vfat /dev/block/mmcblk0p1 /mnt/sdcard ; then
/tmp/busybox echo "Cannot mount sdcard."
exit 1
fi
fi

# remove old log
rm -rf /mnt/sdcard/cyanogenmod_bml.log

# everything is logged into /sdcard/cyanogenmod.log
exec >> /mnt/sdcard/cyanogenmod_bml.log 2>&1

# write the package path to sdcard cyanogenmod.cfg
if /tmp/busybox test -n "$UPDATE_PACKAGE" ; then
PACKAGE_LOCATION=${UPDATE_PACKAGE#/mnt}
/tmp/busybox echo "$PACKAGE_LOCATION" > /mnt/sdcard/cyanogenmod.cfg
fi

# Scorch any ROM Manager settings to require the user to reflash recovery
/tmp/busybox rm -f /mnt/sdcard/clockworkmod/.settings

# write new kernel to boot partition
/tmp/flash_image boot /tmp/boot.img
if [ "$?" != "0" ] ; then
exit 3
fi
/tmp/busybox sync

/sbin/reboot now
exit 0
fi
Quote:

Taking that a step further, each ROM maker could choose whether to check the sizes of MTD partitions and if in tolerance (some predefined range of acceptable) it would flash ROM package on existing partition sizes, OR if not acceptable it could wipe out the partition (and all data on it) and start over with preferred sizes. Right?

Exactly.

---------- Post added at 12:14 PM ---------- Previous post was at 12:04 PM ----------

Let me add one very important note. After flashing those different kernels and mounting partitions with them, when I finally booted my phone, it booted fine, but my home screen layout has reverted to stock layout, which suggests that some of my data was munched when I mounted /data using different sizes. My downloaded apps still show up in my launcher, so some of my data is still there, but obviously some of it was lost.

I think it's safe to say that if partition layouts are changed, partitions will need to be erased and recreated.
The Following 4 Users Say Thank You to toadlife For This Useful Post: [ View ]
25th December 2011, 07:25 AM   |  #52  
codest3r's Avatar
Senior Member
Flag Orlando, FL
Thanks Meter: 59
 
347 posts
Join Date:Joined: Oct 2010
More
Quote:
Originally Posted by toadlife

Let me add one very important note. After flashing those different kernels and mounting partitions with them, when I finally booted my phone, it booted fine, but my home screen layout has reverted to stock layout, which suggests that some of my data was munched when I mounted /data using different sizes. My downloaded apps still show up in my launcher, so some of my data is still there, but obviously some of it was lost.

I think it's safe to say that if partition layouts are changed, partitions will need to be erased and recreated.

First, thanks for taking the time to run this through the gamut and test it out! Also appreciate you sharing the results and code. Helps me understand more of whats happening. Thanks man.

Fascinating to me. Seems like a risky approach to map the partitions in the kernel and not have it fixed elsewhere, but I assume this is best solution for employing MTD. Wonder if any other approaches are available or were considered? Frankly, I don't want a kernel modifying the partition sizes, locations, etc. -- at least without me being fully aware of it before flashing. Hopefully all the kernel devs are aware of this!

And I agree with your conclusion. If you had modified the start, stop, or size parameters of the system partition, or even tried to swap the /data and /system locations, it likely might have prevented it from even booting up.

That said, it should be possible to build on your code that recognizes the different sizes (as you did) and if changes were warranted to partition layout it could first verify the contents of each partition would fit in the new partition layout and if so backup the system, data, and cache (easiest to just straight up copy contents to /sdcard instead of nandroid?), then execute the partition layout changes, then after changes are made to the partition sizes restore files from backup. I suppose... ;)
25th December 2011, 08:48 AM   |  #53  
Recognized Developer
Thanks Meter: 3,390
 
834 posts
Join Date:Joined: Jun 2010
More
Maybe you shouldn't poke around with partition map.
What wrong with the one we have now? Please don't mess around with our layout. Why do you need 50 MB of Cache space?

market fix: https://github.com/EpicCM/android_de...7f8a93e954d5ce
https://github.com/EpicCM/android_ve...it.d/06mountdl
Hope that helps!
Quote:
Originally Posted by toadlife

I just tested it out. I flashed my ROM using this kernel with this partition layout...

Code:
Filesystem                Size      Used Available Use% Mounted on
/dev/block/mtdblock4     25.0M      2.3M     22.7M   9% /cache
/dev/block/mtdblock3    695.0M    407.0M    288.0M  59% /data
/dev/block/mtdblock2    250.0M    206.2M     43.8M  82% /system
I then flashed a kernel with the following partition layout...

Code:
Filesystem                Size      Used Available Use% Mounted on
/dev/block/mtdblock4     50.0M      2.3M     47.7M   5% /cache
/dev/block/mtdblock2    250.0M    206.2M     43.8M  82% /system
/dev/block/mtdblock3    670.0M    407.0M    263.0M  61% /data
...and the partitions mounted just fine, with different sizes!

Unfortunately, after flashing Tortell's MTD kernel which has the "stock" partition layout (a different size for /system), I got this message when trying to mount system partition:

Code:
Failed to mount /dev/block/mtdblock2 on /system: Invalid argument
/data and /cache still mounted.

Flashing back the original kernel allowed my to mount to all partitions again.




No, it just checks to see if the device is MTD or BML. If it is BML, it flashes the MTD kernel and reboots. I added a part to that script which checks the partition sizes and if they don't match up with what they should be, it does the same thing - flashes the kernel and reboots. I inserted the code into updater.sh. This could be put into a seperate shell script. Here is the code. It's adapted from the code that checks for BML/MTD.

Code:
# check to see if partition sizes are correct
/tmp/busybox mount -t yaffs2 /dev/block/mtdblock2 /system
/tmp/busybox mount -t yaffs2 /dev/block/mtdblock3 /data
if (! df /dev/block/mtdblock2 | grep "256000") || (! df /dev/block/mtdblock3 | grep "711680"); then
# Our partition sizes are not correct

# unmount, format and mount system
/tmp/busybox umount -l /system
/tmp/erase_image system

# unmount and format cache
/tmp/busybox umount -l /cache
/tmp/erase_image cache

# unmount and format datadata
/tmp/busybox umount -l /data
/tmp/erase_image userdata

# make sure sdcard is mounted
if ! /tmp/busybox grep -q /mnt/sdcard /proc/mounts ; then
/tmp/busybox mkdir -p /mnt/sdcard
/tmp/busybox umount -l /dev/block/mmcblk0p1
if ! /tmp/busybox mount -t vfat /dev/block/mmcblk0p1 /mnt/sdcard ; then
/tmp/busybox echo "Cannot mount sdcard."
exit 1
fi
fi

# remove old log
rm -rf /mnt/sdcard/cyanogenmod_bml.log

# everything is logged into /sdcard/cyanogenmod.log
exec >> /mnt/sdcard/cyanogenmod_bml.log 2>&1

# write the package path to sdcard cyanogenmod.cfg
if /tmp/busybox test -n "$UPDATE_PACKAGE" ; then
PACKAGE_LOCATION=${UPDATE_PACKAGE#/mnt}
/tmp/busybox echo "$PACKAGE_LOCATION" > /mnt/sdcard/cyanogenmod.cfg
fi

# Scorch any ROM Manager settings to require the user to reflash recovery
/tmp/busybox rm -f /mnt/sdcard/clockworkmod/.settings

# write new kernel to boot partition
/tmp/flash_image boot /tmp/boot.img
if [ "$?" != "0" ] ; then
exit 3
fi
/tmp/busybox sync

/sbin/reboot now
exit 0
fi


Exactly.

---------- Post added at 12:14 PM ---------- Previous post was at 12:04 PM ----------

Let me add one very important note. After flashing those different kernels and mounting partitions with them, when I finally booted my phone, it booted fine, but my home screen layout has reverted to stock layout, which suggests that some of my data was munched when I mounted /data using different sizes. My downloaded apps still show up in my launcher, so some of my data is still there, but obviously some of it was lost.

I think it's safe to say that if partition layouts are changed, partitions will need to be erased and recreated.

25th December 2011, 10:49 AM   |  #54  
toadlife's Avatar
Recognized Developer
Flag Lemoore, CA
Thanks Meter: 1,015
 
1,202 posts
Join Date:Joined: Aug 2008
Donate to Me
More
Quote:
Originally Posted by noobnl

Maybe you shouldn't poke around with partition map.
What wrong with the one we have now? Please don't mess around with our layout. Why do you need 50 MB of Cache space?

market fix: https://github.com/EpicCM/android_de...7f8a93e954d5ce
https://github.com/EpicCM/android_ve...it.d/06mountdl
Hope that helps!

Yeah I know of that fix. More space for /data is actually what I might be interested in. Good luck with your partition size mandate. Honestly there is NO point to using MTD/Yaffs2 if we don't make use of the ability to change partition sizes.
25th December 2011, 11:22 AM   |  #55  
Recognized Developer
Thanks Meter: 3,390
 
834 posts
Join Date:Joined: Jun 2010
More
Quote:
Originally Posted by toadlife

Yeah I know of that fix. More space for /data is actually what I might be interested in. Good luck with your partition size mandate. Honestly there is NO point to using MTD/Yaffs2 if we don't make use of the ability to change partition sizes.

I think you should keep cache the same size but only change data and system because it is better to change only one offset instead of two offset.
25th December 2011, 09:38 PM   |  #56  
Member
Thanks Meter: 16
 
83 posts
Join Date:Joined: Dec 2010
There's plenty of reason to move to MTD without altering the current MTD partition layout, like getting rid of the crappy BML abstraction layer and replacing it with one actually built to reduce issues with flash filesystems, so we can use yaffs2 and have more advanced wear levelling. And reduce the reliance on a closed source filesystem and block layer.

Agree with noob here, you're really not creating anything but a huge problem for users.
26th December 2011, 01:39 AM   |  #57  
toadlife's Avatar
Recognized Developer
Flag Lemoore, CA
Thanks Meter: 1,015
 
1,202 posts
Join Date:Joined: Aug 2008
Donate to Me
More
Quote:
Originally Posted by hazridi

There's plenty of reason to move to MTD without altering the current MTD partition layout, like getting rid of the crappy BML abstraction layer and replacing it with one actually built to reduce issues with flash filesystems, so we can use yaffs2 and have more advanced wear levelling. And reduce the reliance on a closed source filesystem and block layer.

Agree with noob here, you're really not creating anything but a huge problem for users.

You are right that this will cause issues for users, especially those who love the kernel/ROM hop, but how will those issues be avoided when ICS/CM9 rolls out and more space is needed in system? Will another official partition layout be shouted down from the mountain top and those of us who might want to continue to build TW based ROMs be forced to use massive system partitions which will go only half-used?
26th December 2011, 04:17 AM   |  #58  
codest3r's Avatar
Senior Member
Flag Orlando, FL
Thanks Meter: 59
 
347 posts
Join Date:Joined: Oct 2010
More
Quote:
Originally Posted by noobnl

I think you should keep cache the same size but only change data and system because it is better to change only one offset instead of two offset.

Would it be possible to merge system and data on a common root partition? No semi-arbitrary boundaries. Anything not used by ROM/system is avail for data. Still leave something tiny for cache like you have now.

I know this can cause problems with problem/offending apps bugging out and hogging space beyond reason or debugging logs running unchecked.... But that's unlikely to be an issue for us right?

Sorry if it's a dumb question but thought it would eliminate any differences of opinion (or need) for different ROM's partition sizing allocations on MTD. Probably some dependencies I'm not aware of... ;)
26th December 2011, 08:50 AM   |  #59  
toadlife's Avatar
Recognized Developer
Flag Lemoore, CA
Thanks Meter: 1,015
 
1,202 posts
Join Date:Joined: Aug 2008
Donate to Me
More
Quote:
Originally Posted by codest3r

Would it be possible to merge system and data on a common root partition? No semi-arbitrary boundaries. Anything not used by ROM/system is avail for data. Still leave something tiny for cache like you have now.

I know this can cause problems with problem/offending apps bugging out and hogging space beyond reason or debugging logs running unchecked.... But that's unlikely to be an issue for us right?

Sorry if it's a dumb question but thought it would eliminate any differences of opinion (or need) for different ROM's partition sizing allocations on MTD. Probably some dependencies I'm not aware of... ;)

I don't see that as a viable option. It would require a radical change in the way partitions are arranged and it would mean /system has to be mounted read+write, whereas it's typically mounted read-only.

I think I'm going to stick with the CM7 defaults for now. With the way people like to hop from kernel to kernel, and the way people don't read or follow directions, it would cause too much havoc for users. It would be nice to have the partition layouts defined somewhere else, so the kernel wouldn't have to have them baked in.
27th December 2011, 06:11 PM   |  #60  
Recognized Developer
Thanks Meter: 830
 
470 posts
Join Date:Joined: Aug 2009
Quote:
Originally Posted by codest3r

Seems like a risky approach to map the partitions in the kernel and not have it fixed elsewhere, but I assume this is best solution for employing MTD. Wonder if any other approaches are available or were considered?

There is an on-flash partition table, bml2 (not mapped in mtd), and it would certainly make sense to support it under MTD kernels to allow for different partition sizes. It's not done simply becuase there's no MTD-pathway code for reading the on-flash partition table. Partition sizes are hardcoded in the Samsung MTD driver on the Nexus S, and, since that's how the driver was written, was kept for Galaxy S MTD kernels.

It's possible to add on-flash partition table support for MTD, but would require some amount of code. I'd guess it's a reasonably-sized project for someone sufficiently ambitious. The format is known for the relevant portion of the partition table, plus there's code in the tfsr driver for reading it also--which I wouldn't copy, but might help in making sense of it.

Quote:
Originally Posted by toadlife

Honestly there is NO point to using MTD/Yaffs2 if we don't make use of the ability to change partition sizes.

To be honest, the main benefit of MTD vs BML is maintenance of the CM port. Since all other supported SGS devices use MTD, for the Epic to use MTD reduces the amount of our device-specific configutation. While a BML-based CM7 port is certainly doable (an has been done), being the sole BML device means there's more configuration junk that we have to maintain.

The other major benefit of MTD would be the ability to forward port victory (Epic) support to a Nexus S-derived (S5PV210) ICS kernel, i.e., for CM9. With Samsung's proprietary storage stack, we're stuck on our present kernel version unless newer modules leak out somewhere. Given that Samsung appears uninterested in porting ICS to the SGS, those might be hard to come by, especially timely.

Personally I don't think MTD is particularly advantageous to TW kernels, aside from where basing on the EpicCM kernel makes such kernels easier to maintain.
Last edited by mkasick; 27th December 2011 at 06:13 PM.

The Following 3 Users Say Thank You to mkasick For This Useful Post: [ View ]
Post Reply Subscribe to Thread

Tags
cyanogenmod7, epic4g, future, mtd, opensource
Previous Thread Next Thread
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes