Samsung really needs to hire you, garwynn and Entropy.
Discussion thread for /data EMMC lockup/corruption bug
- Thread starter sfhub
- Start date
Hmm. CONFIG_MMC_DISCARD is enabled on I9100/I777 Gingerbread kernels, which is why I said that MMC_CAP_ERASE was enabled (MMC_CAP_ERASE is #ifdefed based on CONFIG_MMC_DISCARD) - I'll look elsewhere tomorrow to see if maybe the "chain of doom" was broken elsewhere.
It's a lot easier to find stuff when you know what you're looking for.
I believe that still, removing MMC_CAP_ERASE from mshci.c is guaranteed to render you safe - but even if it's enabled, there are other places in the kernel where the chain could be broken.
It's a lot easier to find stuff when you know what you're looking for.
I believe that still, removing MMC_CAP_ERASE from mshci.c is guaranteed to render you safe - but even if it's enabled, there are other places in the kernel where the chain could be broken.
I am not surprised I9100/I777 source tree differs from E4GT in some areas. I saw a bunch of #ifdef CONFIG_MMC_DISCARD_MERGE code which indicates to me they were (for our tree) in the process of merging the discard code so it probably could have gone either way depending on when GB for a particular platform was released.
It is easier to search when you know what to search forIt's a lot easier to find stuff when you know what you're looking for.
In this case I wasn't starting with MMC_CAP_ERASE though.I traced top-down from the wipe.c ioctl() from userspace to kernelspace and eventually down to mmc_erase() through the block interface indirection layers. Then I traced from mmc_erase() upwards to see who else might be making calls to it. It was all a bit messy because of the CONFIG_MMC_DISCARD_MERGE code and the resulting multiple definitions of functions.
I kept wondering why it seemed like the ioctl() function was seemingly left undefined by the ifdef's, then realized the function table wasn't populated thus the ioctl() functionality was essentially disabled for the mmc driver (on our platform) I tried to stress a few times above this was based on our platform and that I wasn't sure about others.
Most likely that is correct.
My goal in the previous post was not necessarily to determine where to do a workaround in the kernel, but to answer the question of why the "unsafe" update-binary included with our CM9/AOKP installs didn't trigger mmc_erase() (and expose themselves to the EMMC lockup/superbrick) when run on a GB kernel/recovery which I thought had mmc_erase() enabled. I found that for our platform, the most immediate reason was the ioctl() was disabled.
The difference in the CONFIG_MMC_DISCARD ifdef status in E4GT and I9100/I777 might end up resulting in slightly different bricking behavior when coupling an "unsafe" update-binary with a stock GB kernel/recovery.
I say "might" because I am not sure if the issue is with ERASE, TRIM, SECDISCARD, all three, or some combination of the three. It is possible only SECDISCARD is problematic but ERASE and TRIM are ok because the SECDISCARD code wasn't even in the GB source (for our platform) so it would be something new introduced in ICS (and also we can see it is being used in wipe.c). It is also possible they are all bad.
Last edited:
garwynn
Retired Forum Mod / Inactive Recognized Developer
Then I would have to leave my current job, which I can say is the only job in 17 years that I love. The biggest reason are the managers and coworkers. Without the people and mindset in that office I could easily see a polar opposite of working there.
It is honestly a miracle how I've had opportunities that I did... Much like getting a response from Samsung, Sprint and Google on inquiries with no previous working relationship.
Sent from my SPH-D710 using XDA
Last edited:
I posted this less technical writeup on agat63's cwm repacked thread but figured it would be useful to have here also. I am working with CM9/AOKP to have their install scripts replace format("/system") with delete_recursive("/system") After that I think if we still have volunteers, we are ready to do some more testing.
I'll provide more details when the pieces become available, but if you'd like to take one for the team and help test, please post. If our understanding of the problem is accurate, you should be safe this time around, but there is always the chance we are not understanding the problem completely.
====
I would like to stress that the information I gave agat was based on code tracing and NOT based on real-world testing. You should treat this as a *testing period* to confirm the analysis. You MAY BRICK if the analysis is incorrect.
The following is all assuming you are repacked with ICS kernel (ie we aren't talking about the GB kernel)...
Background
The nature of the problem is a call to the function make_ext4fs(). This function isn't provided by the kernel, rather it is provided as a library (libext4_utils.a) that is used when compiling Recovery and the update installer. It does end up eventually calling kernel mmc driver routines, which then trigger the EMMC firmware lockup/superbrick bug.
The make_ext4fs() function changed between GB and ICS. In GB the function didn't try to erase the partition before creating the EXT4 fs. In ICS it tries to erase it first. The erase is triggering an EMMC firmware bug that was always there via the kernel MMC driver. GB is also "doubly" safe in that not only do Recovery and update-binary never attempt to do the erase, even if they did, the request to erase is blocked in the GB-kernel and never run.
The EMMC firmware bug will lockup your phone and corrupt internal EMMC meta-data which cannot be accessed or repaired at this time. It isn't crashing your hard drive per-se, it is crashing your hard drive controller in a way that prevents the hard drive controller from accessing parts of your disk. We don't have any way of updating the EMMC hard drive controller at this time.
The EMMC firmware lockup/superbrick bug is likely contained in the wear-level firmware code which shifts mmc-internal memory block usage around to prevent any one area from overuse. The bug MIGHT NOT be triggered every time, so you can do the same operation with no issues then on your Nth attempt it bricks.
Details
So what does that mean for you? There are 2 executables we are concerned with, Custom Recovery and the update installer (update-binary)
Custom Recovery is responsible for 2 potential bricking points:
1) wipe data/factory reset
2) nandroid backup/restore
These are both handled by the Recovery itself so if your Recovery is "safe" then these operations should be safe. The nandroid backup is safe regardless. Our concern is only for wipe data/factory reset and nandroid restore. Both of these make the call to make_ext4fs(), so if they are using the GB-based version they are safe. If they are using the ICS version, they are not safe (when used with ICS kernel) Agat has made the effort to make sure the recovery he has provided is compiled against GB CM7 source.
You may ask what about Installing ROMs, you thought Recovery was responsible for that too?
This is only partially true. You use the menu option in Recovery to choose to install an update.zip. Recovery is responsible for providing the location of the update.zip and verifying the signature, but when it comes to actually "installing" the update.zip, Recovery uses a "helper app" called update-binary contained in the update.zip.
This update-binary helper app is responsible for running the Edify install script in the update.zip. It communicates with Recovery just to update the progress bar, output ui messages, and set up the updating of firmware. The rest of the script functions, it handles by itself directly, so Recovery isn't involved.
update-binary also calls make_ext4fs() so it can also do potentially "unsafe" operations, just like we discussed for Recovery above. If the update-binary, that was included in the update.zip, was compiled using GB-sources, then it is "safe". If it was compiled against ICS sources then there is one function in the Edify script that can potentially cause bricking, format().
To be clear, Recovery has no control over the update-binary that is included in the update.zip. Whomever built the ROM update.zip package made that decision. So this is why even with a "safe" Recovery, you can brick your phone installing ROMs (with an ICS kernel).
Even if the Recovery is "safe", if you ask it to use an "unsafe" update-binary to install a ROM AND that ROM install script chose to do a format(), then the EMMC lockup/superbrick bug can be triggered.
The reason why most stock-based ROMs don't brick in ICS is because
1) most of them probably include a GB-based update-binary
2) most of them are not performing a format() within their Edify updater-script
So a ROM builder has 2 ways to make a ROM update.zip install "safe" to install in ICS. Either package a known GB-based update-binary OR eliminate format(), if present, from the Edify install script.
So why does Calk's format_all seem to never brick even on ICS? Given the date on the update-binary and when he created the package, it is most likely using a GB-based update-binary
So why does CM9/AOKP seem to brick more often than stock-based ICS ROM installs? The Edify install script for CM9/AOKP uses new functions that were introduced in the ICS update-binary. This in turn is why they bundle the ICS-based update-binary. They could still potentially be safe, but in the install script a format("/system") is performed. If that format is run under an ICS kernel it will trigger the EMMC firmware lockup/superbrick bug. Under a GB-kernel, the request to erase "/system" is blocked by the GB-kernel.
What can CM9/AOKP do to make their installs "safe" to install in ICS? All they need to do is replace the format("system") with delete_recursive("/system"). They could also replace the ICS-based update-binary with a GB-based update-binary, but that would require more rewrites to the install script. Replacing the format() call is simpler/easier.
Why are some superbricks blue-light specials and others only make ODIN hang at data.img? This likely has to do with whether you got your brick from
1) the format() in the CM9/AOKP install
2) restoring nandroid backup
3) doing the wipe data/factory reset in Recovery
The first two tend to be blue-light specials as they affect /system and/or kernel. The last one tends to affect /data and/or /cache.
So how do you make sure you are totally safe?
1) make sure you are using a "safe" recovery repacked with the stock ICS kernel. This is a Recovery that was compiled against GB-based libext4_utils.a (ie GB source) This will assure you that wipe data/factory reset and nandroid restores are safe
2) whenever you install a ROM for the first time, verify EITHER
a) the ROM install script is NOT performing any format() calls
b) the ROM install has bundled a GB-based update-binary
If neither 2a NOR 2b are true (ie ICS-based update-binary and install performs format) then you DO NOT want to flash that ROM in Recovery while on an ICS kernel. Flash that ROM on a GB-based kernel/recovery.
Hope that clears things up, and once again, remember, this analysis is only based on tracing code. I may have made a mistake in the analysis or our understanding of the problem could be wrong. We will not be sure if all these statements hold UNTIL WE DO REAL-WORLD TESTING.
I'll provide more details when the pieces become available, but if you'd like to take one for the team and help test, please post. If our understanding of the problem is accurate, you should be safe this time around, but there is always the chance we are not understanding the problem completely.
====
I would like to stress that the information I gave agat was based on code tracing and NOT based on real-world testing. You should treat this as a *testing period* to confirm the analysis. You MAY BRICK if the analysis is incorrect.
The following is all assuming you are repacked with ICS kernel (ie we aren't talking about the GB kernel)...
Background
The nature of the problem is a call to the function make_ext4fs(). This function isn't provided by the kernel, rather it is provided as a library (libext4_utils.a) that is used when compiling Recovery and the update installer. It does end up eventually calling kernel mmc driver routines, which then trigger the EMMC firmware lockup/superbrick bug.
The make_ext4fs() function changed between GB and ICS. In GB the function didn't try to erase the partition before creating the EXT4 fs. In ICS it tries to erase it first. The erase is triggering an EMMC firmware bug that was always there via the kernel MMC driver. GB is also "doubly" safe in that not only do Recovery and update-binary never attempt to do the erase, even if they did, the request to erase is blocked in the GB-kernel and never run.
The EMMC firmware bug will lockup your phone and corrupt internal EMMC meta-data which cannot be accessed or repaired at this time. It isn't crashing your hard drive per-se, it is crashing your hard drive controller in a way that prevents the hard drive controller from accessing parts of your disk. We don't have any way of updating the EMMC hard drive controller at this time.
The EMMC firmware lockup/superbrick bug is likely contained in the wear-level firmware code which shifts mmc-internal memory block usage around to prevent any one area from overuse. The bug MIGHT NOT be triggered every time, so you can do the same operation with no issues then on your Nth attempt it bricks.
Details
So what does that mean for you? There are 2 executables we are concerned with, Custom Recovery and the update installer (update-binary)
Custom Recovery is responsible for 2 potential bricking points:
1) wipe data/factory reset
2) nandroid backup/restore
These are both handled by the Recovery itself so if your Recovery is "safe" then these operations should be safe. The nandroid backup is safe regardless. Our concern is only for wipe data/factory reset and nandroid restore. Both of these make the call to make_ext4fs(), so if they are using the GB-based version they are safe. If they are using the ICS version, they are not safe (when used with ICS kernel) Agat has made the effort to make sure the recovery he has provided is compiled against GB CM7 source.
You may ask what about Installing ROMs, you thought Recovery was responsible for that too?
This is only partially true. You use the menu option in Recovery to choose to install an update.zip. Recovery is responsible for providing the location of the update.zip and verifying the signature, but when it comes to actually "installing" the update.zip, Recovery uses a "helper app" called update-binary contained in the update.zip.
This update-binary helper app is responsible for running the Edify install script in the update.zip. It communicates with Recovery just to update the progress bar, output ui messages, and set up the updating of firmware. The rest of the script functions, it handles by itself directly, so Recovery isn't involved.
update-binary also calls make_ext4fs() so it can also do potentially "unsafe" operations, just like we discussed for Recovery above. If the update-binary, that was included in the update.zip, was compiled using GB-sources, then it is "safe". If it was compiled against ICS sources then there is one function in the Edify script that can potentially cause bricking, format().
To be clear, Recovery has no control over the update-binary that is included in the update.zip. Whomever built the ROM update.zip package made that decision. So this is why even with a "safe" Recovery, you can brick your phone installing ROMs (with an ICS kernel).
Even if the Recovery is "safe", if you ask it to use an "unsafe" update-binary to install a ROM AND that ROM install script chose to do a format(), then the EMMC lockup/superbrick bug can be triggered.
The reason why most stock-based ROMs don't brick in ICS is because
1) most of them probably include a GB-based update-binary
2) most of them are not performing a format() within their Edify updater-script
So a ROM builder has 2 ways to make a ROM update.zip install "safe" to install in ICS. Either package a known GB-based update-binary OR eliminate format(), if present, from the Edify install script.
So why does Calk's format_all seem to never brick even on ICS? Given the date on the update-binary and when he created the package, it is most likely using a GB-based update-binary
So why does CM9/AOKP seem to brick more often than stock-based ICS ROM installs? The Edify install script for CM9/AOKP uses new functions that were introduced in the ICS update-binary. This in turn is why they bundle the ICS-based update-binary. They could still potentially be safe, but in the install script a format("/system") is performed. If that format is run under an ICS kernel it will trigger the EMMC firmware lockup/superbrick bug. Under a GB-kernel, the request to erase "/system" is blocked by the GB-kernel.
What can CM9/AOKP do to make their installs "safe" to install in ICS? All they need to do is replace the format("system") with delete_recursive("/system"). They could also replace the ICS-based update-binary with a GB-based update-binary, but that would require more rewrites to the install script. Replacing the format() call is simpler/easier.
Why are some superbricks blue-light specials and others only make ODIN hang at data.img? This likely has to do with whether you got your brick from
1) the format() in the CM9/AOKP install
2) restoring nandroid backup
3) doing the wipe data/factory reset in Recovery
The first two tend to be blue-light specials as they affect /system and/or kernel. The last one tends to affect /data and/or /cache.
So how do you make sure you are totally safe?
1) make sure you are using a "safe" recovery repacked with the stock ICS kernel. This is a Recovery that was compiled against GB-based libext4_utils.a (ie GB source) This will assure you that wipe data/factory reset and nandroid restores are safe
2) whenever you install a ROM for the first time, verify EITHER
a) the ROM install script is NOT performing any format() calls
b) the ROM install has bundled a GB-based update-binary
If neither 2a NOR 2b are true (ie ICS-based update-binary and install performs format) then you DO NOT want to flash that ROM in Recovery while on an ICS kernel. Flash that ROM on a GB-based kernel/recovery.
Hope that clears things up, and once again, remember, this analysis is only based on tracing code. I may have made a mistake in the analysis or our understanding of the problem could be wrong. We will not be sure if all these statements hold UNTIL WE DO REAL-WORLD TESTING.
Last edited:
I don't think I ever stated that and I haven't found code which can support that statement with the bug as we understand it.
There may be a separate bug that supports the statement you are saying, but it isn't the bug we are dealing with here, though I really think we would have heard more complaints if such a bug existed.
Perhaps you are confusing with the 32KB zero bug. I don't know if we are afflicted with that bug. fwrev 0x25 definitely is and possibly we are too (which is basically what Mr. Sumrall assumed), but that one would likely end up corrupting data in a recoverable manner (as in you might lose data, but you can reformat/reflash and things should be ok), rather than a superbrick.
Once again, SF, you do a remarkable job of clarifying a nettlesome and thorny issue. Thanks my friend.
Sent from my SPH-D710 using XDA
Sent from my SPH-D710 using XDA
It is not flashing them "over" ICS kernel per-se that triggers the bug.
It is running an ICS-based update-binary while in an ICS kernel and having that update-binary process a "format()" command in the Edify install script.
I think that is what you meant, but "flashing over" could have different meanings and I wanted to clarify that point.
Once the CM9 and AOKP get rid of the format() command in their Edify install scripts, then the above scenario should be safe.
Yes, that is correct. Even with the ICS-based update-binary and the format() currently in the Edify install script, if update-binary is running under a GB-based kernel, the problematic "erase" code initiated by make_ext4fs() will be blocked by the GB kernel, so no issue arises.
Thanks for trying that. The behavior you describe is as expected. When you flashed CM9 on GB rooted, the GB kernel blocked the CM9 update-binary from making the problematic call when the install script asked to format("/system") so that is why there are no issues.
Then when you "came back" using a nandroid backup, that is being handled by Recovery. Assuming the Recovery was compiled against GB source, which I believe the Recoveries you get from here have, then it will Recovery will not make the problematic "erase" call when it restores, even though the restore process is calling make_ext4fs(). The GB source for make_ext4fs() just doesn't try to erase at all.
BTW may I ask which Kernel/Recovery did you use to do your Nandroid restore?
The only "Recovery" I believe you need to worry about is CWM Touch fake-flashed onto an ICS kernel, as I believe (but have not confirmed because I can't find source code) that Recovery has been compiled with an ICS version of make_ext4fs().
i tried CM9 long ago, and i never knew it would contribute anything..
before trying CM9, i messaged XplodWild if nandroid restore is safe and he boldly said "Yes". i trusted him and i tried CM9 for a day..
the recovery that i used is the one (if im not mistaken) that is included in the CM9 safe kernel. flashing their CM9.zip includes the kernel as stated by Entropy on cm9 thread
before trying CM9, i messaged XplodWild if nandroid restore is safe and he boldly said "Yes". i trusted him and i tried CM9 for a day..
the recovery that i used is the one (if im not mistaken) that is included in the CM9 safe kernel. flashing their CM9.zip includes the kernel as stated by Entropy on cm9 thread
Last edited:
I didn't even realize you were talking about Note until you mentioned some unfamiliar names and I looked at your sig.
My answer was based on E4GT.
To answer based on Note and your newly provided info, if it was a "safe" ICS kernel with a Recovery compiled against ICS sources, then your Recovery would have called the problematic make_ext4fs() when you did the Nandroid restore, but the erase command would have been blocked by your "safe" ICS kernel, thus your nandroid restore worked as expected.
Same result, but the in-between happenings were different.
My answer was based on E4GT.
To answer based on Note and your newly provided info, if it was a "safe" ICS kernel with a Recovery compiled against ICS sources, then your Recovery would have called the problematic make_ext4fs() when you did the Nandroid restore, but the erase command would have been blocked by your "safe" ICS kernel, thus your nandroid restore worked as expected.
Same result, but the in-between happenings were different.
That's bringing back something I suspected long long ago back when we knew far less about this bug, but I'm still not sure... Obviously a userspace-triggered erase operation will fire ERASE commands at the card - but what about TRIM (which is a variant of ERASE) - is it blocked by another mechanism? (I suspect so but I'm not entirely sure.)
As to musashiro's questions - the CM9 kernel for N7000 has MMC_CAP_ERASE removed, which prevents attempts to erase from ever reaching the eMMC chip.
Last edited:
What about using a different file system?
Apologies if this was already tried/suggested, but since we're dealing with a Gnu/Linux kernel, and if setting up the ext4 file system triggers the bug in the eMMC firmware, what about using a different one? (See en.wikipedia.org/wiki/Flash_file_system)
Apologies if this was already tried/suggested, but since we're dealing with a Gnu/Linux kernel, and if setting up the ext4 file system triggers the bug in the eMMC firmware, what about using a different one? (See en.wikipedia.org/wiki/Flash_file_system)
It has nothing directly to do with ext4 itself - it's just that recovery was modified such that it would issue ERASE commands prior to formatting the partition for privacy purposes.
The way I answered that question for E4GT is to observe that ERASE, TRIM, DISCARD basically essentially all do the same mmc operation but pass different arguments to the mmc command. I believe in the ICS source tree they all get filtered down to mmc_erase() so if you should do something to short-circuit mmc_erase() (like not setting MMC_CAP_ERASE in the host capabilities) TRIM and DISCARD will get blocked also. They won't be blocked explicitly, but since mmc_erase() checks the host's ability to do MMC_CAP_ERASE first, it will return with EOPNOTSUPP because presumably if your host doesn't support ERASE, it wouldn't support TRIM and DISCARD since ERASE is the older base functionality.
In the case of the E4GT GB source tree the only references I saw were having it called from userspace through the ioctl(). Since the ioctl() is disabled in our GB build, there is no way to eventually reach mmc_erase(), thus TRIM and DISCARD should never be reached.
Since our build has CONFIG_MMC_DISCARD disabled, the discard code isn't even compiled. I believe the TRIM and ERASE code is still there, but since the ioctl to switch to kernelspace to run mmc_erase() isn't present it won't ever get run.
I still am not sure if ERASE, TRIM, and DISCARD versions of erase are all borked or some smaller subset are the issues.
So to answer your question, if you have blocked mmc_erase() as you described earlier, by not enabling MMC_CAP_ERASE in the host capabilities, based on the generic ICS tree, that should block TRIM and DISCARD as well. I haven't looked in the Samsung specific source tree though.
Last edited:
I'm pretty sure the wipe code that was added between GB and ICS only gets called when creating ext4 filesystems. However I didn't specifically check if the code paths for the other filesystems would reach some variation of the wipe code.
It is probable the other filesystems aren't doing any wipes so you could theoretically change the FS and the bug wouldn't be triggered. You'd have to change all the config on your ROMs/kernels/recovery yourself to use a different FS though and it would be more intrusive than just making sure your recovery, update-binary, updater-script, and in some cases recompiled kernel are safe.
To be clear, it isn't a problem with the ext4fs filesystem itself. It is that the userspace "utility" function make_ext4fs() (this is not part of the kernel filesystem code) eventually tries to erase the partition and this leads to a bad reaction with EMMC firmware that has the lockup/superbrick bug. make_ext4fs() could be renamed joes_erase_ext4_partition() for all we care. It is just some arbitrary utility function being used by Recovery and update-binary to create an ext4fs() filesystem. Don't let the ext4fs() lead you into thinking it represents some problem with generic ext4fs code in the kernel.
Entropy, do you recall how it came about that deleting large files was a possible suspect?
Was it just a theory or was there specific code people were looking at that led to that belief?
For completeness purposes I would like to research if this possibility exists.
Was it just a theory or was there specific code people were looking at that led to that belief?
For completeness purposes I would like to research if this possibility exists.
im curious if its a possibility also. i regulary delete large files off my sd card and ive been using a ICS kernel for a couple of months now.
Without looking at source code, my guess is you would hear of more bricks when people cleared out files if this was a likely scenario.
I don't like to completely rule something out until/unless I look over the source code to verify though.
Clearly the wipe data and format from ICS-based update-binary are the most likely scenarios where people see superbricks as the boards are littered with anecdotes for those two.
I don't like to completely rule something out until/unless I look over the source code to verify though.
Clearly the wipe data and format from ICS-based update-binary are the most likely scenarios where people see superbricks as the boards are littered with anecdotes for those two.
Similar threads
Top Liked Posts
-
There are no posts matching your filters.
-
70This post will evolve over time as more info is found:
Latest Updates
6/24 Custom PIT repartition workaround posted by Jaymoon.
You lose 8GB (some of which might be further recoverable with extra work) and restores using the original PIT will lockup your phone again (a scenario that could happen if you brought your phone back to Sprint for some unrelated problem) so if you have the opportunity to get your phone replaced with little to no cost, IMO that should be your primary option.
http://xdaforums.com/showthread.php?p=27852689#post27852689
E4GT specific PIT file here (theoretically instead of losing 8GB, you'll only lose 2GB):
http://xdaforums.com/showpost.php?p=28070569&postcount=654
6/8 Update for other platforms waiting for fix
Codeworkx's contact with Samsung got following response [discussion]
6/7 Update
Test plan posted - see bottom of post for results so far (esoteric68, krazy_smokezalot report success)
BIG THANKS to Esoteric68 (and robertm2011 before her) who took the plunge to benefit everyone else. She has completed the test plan and more. 6 flashes of CM9, 3 flashes of AOKP, 3 wipe data/factory resets, and 3 nandroid restores, 1 stock FF02 flash, all successful. We are ready to have more testers try out the test ROM installs. We are getting more confident the code analysis was correct.
6/2 Update
Less technical summary and preparation for new round of testing
5/31 Lots of discussion on the code path detailing how the problem occurs and where to put the workaround, select posts below
Call trace for CWM Recovery - wipe data/factory reset
Call trace for CWM Recovery - restore
Section of update-binary afflicted by same issue as wipe data/factory reset
Recap of where workarounds can be placed
MD5s of various update-binary executables
Pros/Cons of placing workaround in kernel vs libext4_utils.a
Are ICS nandroid backup/restores safe?
Are ICS recoveries safe?
Why do CM9/AOKP installs often brick in ICS but not in GB?
5/24 Update pretty much ties up all the loose ends - Thanks Mr. Sumrall, Garwynn, Entropy, and everyone else who pitched in!
http://xdaforums.com/showthread.php?p=26521643#post26521643
Potentially very GOOD NEWS
It appears Sprint/Samsung tested the EMMC brick issue, confirmed the problem, and tested a fix that appears to resolve the problem:
http://xdaforums.com/showthread.php?p=26465085#post26465085
We now have an update on why this bug is happening and which PRV/fwrevs are affected. PRV/fwrev 0x19 are susceptible to the EMMC /data corruption issue (which should now be referred to as EMMC lockup issue). PRV/fwrev 0x25 has the fix for the lockup issue but has a separate 32KB of zeros data corruption issue, which is being patched in the kernel (our kernels don't have that patch). All these problems are in the EMMC firmware. It can potentially be updated, but nothing is publicly available. EMMC lockup issue is triggered on erasing the EMMC. The only piece we have not been able to explain is why GB-based kernels seem immune to the EMMC lockup problem whereas ICS seems more susceptible to the problem. Presumably both are doing ERASE commands, but possibly in slightly different ways. See these posts for more details [#1 / #2]
To get your PRV/fwrev, you can use this (if you have busybox installed):
If you don't have busybox installed just visually parse the line, match the serial # (0xd3f24fe6 - example only - yours will be different) with the cid, and look at the 2 numbers before the serial #.
It appears after looking at the code more closely and examining the results of the card info dumps, we do not have this fix in our kernel. It isn't clear whether the fix would resolve our /data EMMC brick issues, but the point is moot right now because we don't have the fix.
Possible BRICK here. Please do NOT do any more testing until further notice. Please do NOT use Wipe Data/Factory Reset. It is the main difference between first and 2nd round of testing and is the current suspect
FE10 repacks added to Resource section
Esoteric68, azyouthinkeyeiz, and robertm2011 are testing flashing different ROMs with FE07/FE10 repacked with unlocked recovery. We all owe them our thanks for risking their phones to help the community (taking one for the team) No bricks so far.
Separately we are still discussing whether the fix Samsung checked in will get applied to our phone. No firm conclusions yet. Even if it doesn't apply, the hope is the data we get from testing will help us produce more flexible "safe" flashing practices.
Please do NOT test CWM Touch for now. We want to isolate just the FE07 kernel and unlocked stock recovery before introducing new variables.
Executive Summary
Garwynn has found a recent checkin from Samsung in the kernel code handling EMMC memory that fixes a data corruption problem. It is possible this might fix the /data EMMC corruption we have been seeing, but we aren't sure if it is fixing the same problem. The first release to include that checked in code is FE07. There has been some communication with the developers in charge of that area to gather further info.
This thread's purpose is to foster discussion on the issue and to determine if the potential fix actually does fix our issue. Even if the fix doesn't address the issue, it is hoped in the process we are able to gather more info into specific "safe" and "unsafe" scenarios.
Please do NOT jump ahead and think it is fixed. It is TOO EARLY to make that claim.
Background
As many of you are aware since ICS has come out, there has been a nagging issue where in some situations flashing ROMs with an ICS-based kernel and custom recovery has left the phone with EMMC corruption. This EMMC corruption is so far non-recoverable, even with JTAG bit blasting, which should bypass all but hardware issues.
This problem is NOT limited to the Epic 4G Touch. Other GS2 models as well as Galaxy Note are experiencing the same thing as can be seen by this Public Service Announcement in the Galaxy Note section.
The problem first cropped up when people used ROM Manager to temporary "fake" flash CWM Touch onto an ICS-based kernel to do their flashing needs. In particular wipe data/factory reset seemed to often trigger the /data EMMC corruption. However later we found it wasn't limited to just CWM Touch and temporary flashing as CWM repacks with the ICS-based kernel also exhibited that behavior, albeit not as often.
Even more frustrating is that this bug is not always deterministic, in that you could do some operation 3 times and have it work fine, then on the 4th, trigger the /data EMMC corruption.
Complicating the testing/debugging is the issue that once the problem is triggered, your phone is basically not recoverable. You can try and ODIN a stock ROM on top which will basically work for all the components except the /data partition. Once it reaches the /data partition, ODIN will hang. Similarly if you try and wipe data/factory reset, it will hang or timeout after a while. Attempts to repartition and reformat using ODIN have not changed this behavior. Attempts to edit the partition info manually have not been successful. JTAG bit blasting has not been successful.
You can read about the past experiences in the Stuck at "Data.img" thru odin thread. By the time you get to ODIN, the damage to /data EMMC is already done. ODIN is NOT causing the damage. ODIN is hanging on data.img because the hardware won't let it write successfully to that area of EMMC.
This has led to many custom ROMs giving special procedures to go back to a GB-based kernel repacked with CWM recovery to do all your flashing (EL26+CWM). It is also the motivation for the How Not To Brick Your E4GT thread.
Details
The code checkin that has piqued our interest is in regards to data corruption caused by problem in the wear-level firmware code of the emmc. This is low-level code that runs on a processor in the emmc module. It basically tries to spread out the data writes so you get an even distribution of writes so as any one section of emmc memory does not get worn out prematurely. This code apparently can corrupt data by writing 32KB of incorrect data under some situations.
https://bitbucket.org/franciscofranco/android-tuna-omap/changeset/cea631bdac53
The code appears to restrict the firmware fix to only certain "affected" emmc modules. Also it is not able to persistently/permanently patch the firmware so this code must run at each startup. The following modules were identified in the code:
Name: VYL00M
HwRev: 0x0
FwRev: 0x25
Name: KYL00M
HwRev: 0x0
FwRev: 0x25
Name: MAG4FA
HwRev: 0x0
FwRev: 0x25
Unfortunately during ad-hoc polling we have found a case of an EMMC /data bricked phone with fwrev 0x0, so either we are not understanding what Samsung's fix is doing or they may not have addressed the full scope of the problem. Do NOT assume if your fwrev is 0x0 you are safe.
At this point, this does NOT mean the fix is not applicable. We might be looking at the wrong data. The kernel might not be exporting the data to us. The fix might need to be expanded to more modules. The fix could be for something else entirely but we might be able to avoid the bug anyway using stock recovery.
To determine what version you have (keep in mind we are at the preliminary stage, so this info might not be the right info to collect or could be meaningless for the /data EMMC corruption issue)
The comments for the code checkin give the following info:
The critical piece of code appears to be the following:
Code:/* set value 0x000000FF : It's hidden data * When in vendor command mode, the erase command is used to * patch the firmware in the internal sram. */ err = mmc_movi_erase_cmd(card, 0x0004DD9C, 0x000000FF); if (err) { pr_err("Fail to Set WL value1\n"); goto err_set_wl; } /* set value 0xD20228FF : It's hidden data */ err = mmc_movi_erase_cmd(card, 0x000379A4, 0xD20228FF); if (err) { pr_err("Fail to Set WL value2\n"); goto err_set_wl; }
Action items
At this point we would like to
1) gather more info on which emmc modules folks have and see if we can detect any patterns, so if you could post your EMMC info and optionally include whether you have the ability to do testing (presumably because you have a way to replace your phone if it is damaged)
2) solicit one volunteer to try different flashing scenarios using the unlocked stock recovery and FE07 kernel repack (bigpeng indicated earlier he would be willing to do this for the community, but that was before the fwrev info, so he might have had a false sense of security, so no pressure on him if he changed his mind)
If we find that the volunteer does not see any corruption despite trying to do so, then we can expand testing to a few more people and also work on getting CWM repacks.
If the volunteer hits the bug, then we will know the issue is still there even with stock recovery and FE07 kernel.
Keep in mind, at some point someone will need to take one for the team or we will be forever in fear of bricking our phones using ICS-based kernels.
Resources
1) FE07-based repacks
Unlocked Recovery Only [update.zip / tar]
Plus (unlocked recovery, init.d, adb-root) [update.zip / tar]
2) FE10-based repacks
Unlocked Recovery Only [update.zip / tar]
Plus (unlocked recovery, init.d, adb-root) [update.zip / tar]
3) JEDEC eMMC documentation
Related threads
Galaxy Note CID investigation thread30I posted this less technical writeup on agat63's cwm repacked thread but figured it would be useful to have here also. I am working with CM9/AOKP to have their install scripts replace format("/system") with delete_recursive("/system") After that I think if we still have volunteers, we are ready to do some more testing.
I'll provide more details when the pieces become available, but if you'd like to take one for the team and help test, please post. If our understanding of the problem is accurate, you should be safe this time around, but there is always the chance we are not understanding the problem completely.
====
I would like to stress that the information I gave agat was based on code tracing and NOT based on real-world testing. You should treat this as a *testing period* to confirm the analysis. You MAY BRICK if the analysis is incorrect.
The following is all assuming you are repacked with ICS kernel (ie we aren't talking about the GB kernel)...
Background
The nature of the problem is a call to the function make_ext4fs(). This function isn't provided by the kernel, rather it is provided as a library (libext4_utils.a) that is used when compiling Recovery and the update installer. It does end up eventually calling kernel mmc driver routines, which then trigger the EMMC firmware lockup/superbrick bug.
The make_ext4fs() function changed between GB and ICS. In GB the function didn't try to erase the partition before creating the EXT4 fs. In ICS it tries to erase it first. The erase is triggering an EMMC firmware bug that was always there via the kernel MMC driver. GB is also "doubly" safe in that not only do Recovery and update-binary never attempt to do the erase, even if they did, the request to erase is blocked in the GB-kernel and never run.
The EMMC firmware bug will lockup your phone and corrupt internal EMMC meta-data which cannot be accessed or repaired at this time. It isn't crashing your hard drive per-se, it is crashing your hard drive controller in a way that prevents the hard drive controller from accessing parts of your disk. We don't have any way of updating the EMMC hard drive controller at this time.
The EMMC firmware lockup/superbrick bug is likely contained in the wear-level firmware code which shifts mmc-internal memory block usage around to prevent any one area from overuse. The bug MIGHT NOT be triggered every time, so you can do the same operation with no issues then on your Nth attempt it bricks.
Details
So what does that mean for you? There are 2 executables we are concerned with, Custom Recovery and the update installer (update-binary)
Custom Recovery is responsible for 2 potential bricking points:
1) wipe data/factory reset
2) nandroid backup/restore
These are both handled by the Recovery itself so if your Recovery is "safe" then these operations should be safe. The nandroid backup is safe regardless. Our concern is only for wipe data/factory reset and nandroid restore. Both of these make the call to make_ext4fs(), so if they are using the GB-based version they are safe. If they are using the ICS version, they are not safe (when used with ICS kernel) Agat has made the effort to make sure the recovery he has provided is compiled against GB CM7 source.
You may ask what about Installing ROMs, you thought Recovery was responsible for that too?
This is only partially true. You use the menu option in Recovery to choose to install an update.zip. Recovery is responsible for providing the location of the update.zip and verifying the signature, but when it comes to actually "installing" the update.zip, Recovery uses a "helper app" called update-binary contained in the update.zip.
This update-binary helper app is responsible for running the Edify install script in the update.zip. It communicates with Recovery just to update the progress bar, output ui messages, and set up the updating of firmware. The rest of the script functions, it handles by itself directly, so Recovery isn't involved.
update-binary also calls make_ext4fs() so it can also do potentially "unsafe" operations, just like we discussed for Recovery above. If the update-binary, that was included in the update.zip, was compiled using GB-sources, then it is "safe". If it was compiled against ICS sources then there is one function in the Edify script that can potentially cause bricking, format().
To be clear, Recovery has no control over the update-binary that is included in the update.zip. Whomever built the ROM update.zip package made that decision. So this is why even with a "safe" Recovery, you can brick your phone installing ROMs (with an ICS kernel).
Even if the Recovery is "safe", if you ask it to use an "unsafe" update-binary to install a ROM AND that ROM install script chose to do a format(), then the EMMC lockup/superbrick bug can be triggered.
The reason why most stock-based ROMs don't brick in ICS is because
1) most of them probably include a GB-based update-binary
2) most of them are not performing a format() within their Edify updater-script
So a ROM builder has 2 ways to make a ROM update.zip install "safe" to install in ICS. Either package a known GB-based update-binary OR eliminate format(), if present, from the Edify install script.
So why does Calk's format_all seem to never brick even on ICS? Given the date on the update-binary and when he created the package, it is most likely using a GB-based update-binary
So why does CM9/AOKP seem to brick more often than stock-based ICS ROM installs? The Edify install script for CM9/AOKP uses new functions that were introduced in the ICS update-binary. This in turn is why they bundle the ICS-based update-binary. They could still potentially be safe, but in the install script a format("/system") is performed. If that format is run under an ICS kernel it will trigger the EMMC firmware lockup/superbrick bug. Under a GB-kernel, the request to erase "/system" is blocked by the GB-kernel.
What can CM9/AOKP do to make their installs "safe" to install in ICS? All they need to do is replace the format("system") with delete_recursive("/system"). They could also replace the ICS-based update-binary with a GB-based update-binary, but that would require more rewrites to the install script. Replacing the format() call is simpler/easier.
Why are some superbricks blue-light specials and others only make ODIN hang at data.img? This likely has to do with whether you got your brick from
1) the format() in the CM9/AOKP install
2) restoring nandroid backup
3) doing the wipe data/factory reset in Recovery
The first two tend to be blue-light specials as they affect /system and/or kernel. The last one tends to affect /data and/or /cache.
So how do you make sure you are totally safe?
1) make sure you are using a "safe" recovery repacked with the stock ICS kernel. This is a Recovery that was compiled against GB-based libext4_utils.a (ie GB source) This will assure you that wipe data/factory reset and nandroid restores are safe
2) whenever you install a ROM for the first time, verify EITHER
a) the ROM install script is NOT performing any format() calls
b) the ROM install has bundled a GB-based update-binary
If neither 2a NOR 2b are true (ie ICS-based update-binary and install performs format) then you DO NOT want to flash that ROM in Recovery while on an ICS kernel. Flash that ROM on a GB-based kernel/recovery.
Hope that clears things up, and once again, remember, this analysis is only based on tracing code. I may have made a mistake in the analysis or our understanding of the problem could be wrong. We will not be sure if all these statements hold UNTIL WE DO REAL-WORLD TESTING.19Morning Update
Well, it's been some time but thankfully Mr. Sumrall from Android did get back to us on our questions. I think the community will find that this was worth the wait.
Issue: fwrev not set properly.
As we suspected the bugfix is not in our build. (The patch applies this unconditionally.)
Ken Sumrall said:
Question: Revision didn't match the fix
(Emphasis mine in red as it discusses the superbrick issue.)
Ken Sumrall said:
Question: Why the /data partition?
Ken Sumrall (Android SE) said:
Question: Why JTAG won't work?
Ken Sumrall said:
Question: Can a corrupted file system be repaired (on the eMMC)?
Ken Sumrall said:
So, while the fix doesn't apply to us at the moment, we've been given a great insight into the superbrick issue as well as information that a fix is already developed (hopefully we'll see it released!). The bug likely applies to us and assuming the fix for the 0x19 firmware is given then it would apply to our devices.
On a lighter note, I wanted to include his close:
Ken Sumrall said:You are getting a glimpse into the exciting life of an Android kernel developer. Turns out the job is mostly fighting with buggy hardware. At least, it seems that way sometimes.
185/24 Update
Fresh off the press...
Q: Do you know the commands to reset the eMMC controller in question?
Ken Sumrall (Android) said:
Q: Is there any documentation available on this issue? If so and it is private is it possible to have it released?
Ken Sumrall (Android) said:
Q: Alternate erase method?
Ken Sumrall (Android) said:
Q: Difference between GB/ICS Wipe
Ken Sumrall (Android) said:
Regarding Entropy512's summary of observations:
Ken Sumrall (Android) said:
Additional info not directly related to questions:
Ken Sumrall (Android) said:16I was able to confirm with chris41g and T.C.P that the update-binary, included with both the CM9 and AOKP ROM update.zip files, was compiled against ICS source code.
This means that this update-binary can potentially trigger the EMMC firmware lockup/superbrick bug if "format()" is used in the Edify install script. Both CM9 and AOKP updater-script format("/system") as part of their installs.
This explains why they brick so often with the bluelight-only style brick, it is because the format("/system") elicits the lockup when make_ext4fs() is called on the /system partition and eventually results in an mmc_erase(). With a boarked /system, you are left with the blue-light special and ODIN will hang on factoryfs.img.
The other style of brick is the one that hangs on the logo. That one is caused by Wipe Data/Factory Reset from an "unsafe" cWM Recovery. Just as above, the "unsafe" CWM Recovery would have been compiled against the ICS libext4_utils.a and make_ext4fs() would be called on /data and /cache. Eventually that would lead to mmc_erase() being called. With a boarked /data, you are left with a phone that hangs on the log and ODIN will hang on data.img
As an aside, I now believe the main "unsafe" Recovery we were dealing with was CWM Touch fake-flashed onto the ICS kernel. I don't have the source code for CWM Touch, but I believe it was compiled against the ICS libext4_utils.a. I am not sure if Rogue CWM repacked with an ICS kernel was compiled against ICS sources, but the version that Steady checked into github was not.
Now previously I recall discussions where it was determined that GB source/kernels also had mmc_erase() and it was an open question why, if both GB and ICS had mmc_erase(), Recovery Wipe Data/Factory Reset would brick one but not the other. Well we finally got the answer to that, the GB version of make_ext4fs() from libext4_utils.a did not perform the ioctl() calls which would eventually lead to mmc_erase() being called, while the ICS version of libext4_utils.a had the ioctl() calls.
Now if you've been following closely so far, you may be wondering why if the update-binary that is bundled with both CM9 and AOKP ROM update.zip files is using the ICS version of libext4_utils.a (that makes the ioctl() calls which lead to mmc_erase()) it doesn't brick when run under a GB-kernel/CWM. Both ICS and GB have mmc_erase() functionality so it should brick both right?
Well I wondered the same thing, went digging a little, and believe have found the answer.
It turns out that even though the GB kernel MMC driver has mmc_erase() the ioctl() calls that would eventually lead to mmc_erase() being called were ifdef'd out. So if you had an "unsafe" update-binary which made the ioctl() calls, they would NOT result in mmc_erase() being called when the ROMs update.zip was run under a GB kernel/recovery.
Previously the discussion centered around MMC_CAP_ERASE as the explanation, which I believe is a red-herring. This is neither a pre-processor directive nor a direct control mechanism. It is a bit field used to specify whether an MMC device supports the capability of doing erases. It was not ifdef'd out of GB so the functionality was still there.
The actual conditional compile was on the ioctl() function for mmc cards, which was basically disabled under GB kernels. This is how I came to that conclusion:
Notice in the kernel config file Samsung provided, both CONFIG_MMC_DISCARD and CONFIG_TARGET_LOCALE_NTT are disabled.
.config
Code:# CONFIG_MMC_DEBUG is not set [B][U]# CONFIG_MMC_DISCARD is not set[/U][/B] CONFIG_MMC_UNSAFE_RESUME=y # CONFIG_MMC_EMBEDDED_SDIO is not set # CONFIG_MMC_PARANOID_SD_INIT is not setCode:# CONFIG_MACH_C1_NA_SPR_EPIC2_REV00 is not set # CONFIG_TARGET_LOCALE_EUR is not set # CONFIG_TARGET_LOCALE_KOR is not set [B][U]# CONFIG_TARGET_LOCALE_NTT is not set[/U][/B] CONFIG_TARGET_LOCALE_NA=y
Now if you look at the mmc block device driver, due to the kernel config above, the ioctl() function call in the function table is left unpopulated. This basically means ioctl() calls are not supported for the mmc device (based on E4GT GB source, don't know about other platforms).
drivers/mmc/card/block.c
Code:static const struct block_device_operations mmc_bdops = { .open = mmc_blk_open, .release = mmc_blk_release, [B][U]#ifdef CONFIG_MMC_DISCARD[/U][/B] .ioctl = mmc_blk_ioctl, [B][U]#endif[/U][/B] [B][U]#if defined(CONFIG_TARGET_LOCALE_NTT)[/U][/B] #if 0 //def CONFIG_MMC_CPRM .ioctl = mmc_blk_ioctl, //int (*ioctl) (struct block_device *, fmode_t, unsigned, unsigned long); in blkdev.h #endif [B][U]#endif[/U][/B]
Now when the "unsafe" update-binary from CM9/AOKP is paired with a GB kernel the wipe.c from libext4_utils.a will try and make these calls:
system/extras/ext4_utils/wipe.c
Code:int wipe_block_device(int fd, s64 len) { u64 range[2]; int ret; range[0] = 0; range[1] = len; ret = [B][U]ioctl(fd, BLKSECDISCARD, &range);[/U][/B] if (ret < 0) { range[0] = 0; range[1] = len; ret = [B][U]ioctl(fd, BLKDISCARD, &range);[/U][/B] } return 0; }
I believe that both those calls have basically been disabled because ioctl() calls are not supported in the mmc block device driver under the GB kernel (for E4GT EL29 GB kernel source).
So what does that all mean?
I believe GB systems are doubly "safe". The kernel will not execute mmc_erase() even though the functionality is there because the ioctl() call entry point from user space to kernel space has been disabled and also GB recovery and GB update-binary never even attempt to make the ioctl() calls.
That explains why, if you pair an "unsafe" update-binary from the CM9/AOKP ROM update.zip with a GB kernel, it is still safe.
