FORUMS

Analysis & Opinion

Top Forum Discussions

eMMC sudden death research

52 posts
Thanks Meter: 259
 
By Oranav, Member on 12th January 2013, 05:26 PM
Post Reply Subscribe to Thread Email Thread
Update from Feb 17th:
Samsung has started to upgrade eMMC firmwares on the field - only for GT-I9100 for now.
See post #79 for additional details.

Update from Feb 13th:
If you want to dump the eMMC's RAM yourself, go ahead to post #72.
I'm looking for a dump of firmware revision 0xf7 if you've got one.
-----------------------


Since it's very likely that the recent eMMC firmware patch by Samsung is their patch for the "sudden death" issue, it would be very nice to understand what is really going on there.

According to a leaked moviNAND datasheet, it seems that MMC CMD62 is vendor-specific command that moviNAND implements.
If you issue CMD62(0xEFAC62EC), then CMD62(0xCCEE) - you can read a "Smart report". To exit this mode, issue CMD62(0xEFAC62EC), then CMD62(0xDECCEE).


So what are they doing in their patch?

1. Whenever an MMC is attached:
a. If it is "VTU00M", revision 0xf1, they read a Smart report.
b. The DWORD at Smart[324:328] represents a date (little-endian); if it is not 0x20120413, they don't patch the firmware. (Maybe only chips from 2012/04/13 are buggy?)
2. If the chip is buggy, whenever an MMC is attached or the device is resumed:
a. Issue CMD62(0xEFAC62EC) CMD62(0x10210000) to enter RAM write mode. Now you can write to RAM by issuing MMC_ERASE_GROUP_START(Address to write) MMC_ERASE_GROUP_END(Value to be written) MMC_ERASE(0).
b. *(0x40300) = 10 B5 03 4A 90 47 00 28 00 D1 FE E7 10 BD 00 00 73 9D 05 00
c. *(0x5C7EA) = E3 F7 89 FD
d. Exit RAM write mode by issuing CMD62(0xEFAC62EC) CMD62(0xDECCEE).
10 B5 looks like a common Thumb push (in ARM architecture). Disassembling the bytes that they write to 0x40300 yields the following code:
Code:
ROM:00040300                 PUSH    {R4,LR}
ROM:00040302                 LDR     R2, =0x59D73
ROM:00040304                 BLX     R2
ROM:00040306                 CMP     R0, #0
ROM:00040308                 BNE     locret_4030C
ROM:0004030A
ROM:0004030A loc_4030A                               ; CODE XREF: ROM:loc_4030Aj
ROM:0004030A                 B       loc_4030A
ROM:0004030C ; ---------------------------------------------------------------------------
ROM:0004030C
ROM:0004030C locret_4030C                            ; CODE XREF: ROM:00040308j
ROM:0004030C                 POP     {R4,PC}
ROM:0004030C ; ---------------------------------------------------------------------
Disassembling what they write to 0x5C7EA yields this:
Code:
ROM:0005C7EA                 BL      0x40300
Looks like it is indeed Thumb code.
If we could dump the eMMC RAM, we would understand what has been changed.


By inspecting some code, it seems that we know how to dump the eMMC RAM:
Look at the function mmc_set_wearlevel_page in line 206. It patches the RAM (using the method mentioned before), then it validates what it has written (in lines 255-290). Seems that the procedure to read the RAM is as following:
1. CMD62(0xEFAC62EC) CMD62(0x10210002) to enter RAM reading mode
2. MMC_ERASE_GROUP_START(Address to read) MMC_ERASE_GROUP_END(Length to read) MMC_ERASE(0)
3. MMC_READ_SINGLE_BLOCK to read the data
4. CMD62(0xEFAC62EC) CMD62(0xDECCEE) to exit RAM reading mode


I don't want to run this on my device, because I'm afraid - messing with the eMMC doesn't sound like a very good idea on my device (I don't have a spare one).
Does someone have a development device which he doesn't mind to risk, and want to dump the eMMC firmware from it?
Last edited by Oranav; 17th February 2013 at 05:50 PM.
The Following 51 Users Say Thank You to Oranav For This Useful Post: [ View ]
 
 
14th January 2013, 02:48 AM |#2  
sponix2ipfw's Avatar
Member
Thanks Meter: 27
 
More
Arrow
Quote:
Originally Posted by Oranav

Since it's very likely that the recent eMMC firmware patch by Samsung is their patch for the "sudden death" issue, it would be very nice to understand what is really going on there.

According to a leaked moviNAND datasheet, it seems that MMC CMD62 is vendor-specific command that moviNAND implements.
If you issue CMD62(0xEFAC62EC), then CMD62(0xCCEE) - you can read a "Smart report". To exit this mode, issue CMD62(0xEFAC62EC), then CMD62(0xDECCEE).


So what are they doing in their patch?

1. Whenever an MMC is attached:

a. If it is "VTU00M", revision 0xf1, they read a Smart report.
b. The DWORD at Smart[324:328] represents a date (little-endian); if it is not 0x20120413, they don't patch the firmware. (Maybe only chips from 2012/04/13 are buggy?)
2. If the chip is buggy, whenever an MMC is attached or the device is resumed:
a. Issue CMD62(0xEFAC62EC) CMD62(0x10210000) to enter RAM write mode. Now you can write to RAM by issuing MMC_ERASE_GROUP_START(Address to write) MMC_ERASE_GROUP_END(Value to be written) MMC_ERASE(0).
b. *(0x40300) = 10 B5 03 4A 90 47 00 28 00 D1 FE E7 10 BD 00 00 73 9D 05 00
c. *(0x5C7EA) = E3 F7 89 FD
d. Exit RAM write mode by issuing CMD62(0xEFAC62EC) CMD62(0xDECCEE).
10 B5 looks like a common Thumb push (in ARM architecture). Disassembling the bytes that they write to 0x40300 yields the following code:
Code:
ROM:00040300                 PUSH    {R4,LR}
ROM:00040302                 LDR     R2, =0x59D73
ROM:00040304                 BLX     R2
ROM:00040306                 CMP     R0, #0
ROM:00040308                 BNE     locret_4030C
ROM:0004030A
ROM:0004030A loc_4030A                               ; CODE XREF: ROM:loc_4030Aj
ROM:0004030A                 B       loc_4030A
ROM:0004030C ; ---------------------------------------------------------------------------
ROM:0004030C
ROM:0004030C locret_4030C                            ; CODE XREF: ROM:00040308j
ROM:0004030C                 POP     {R4,PC}
ROM:0004030C ; ---------------------------------------------------------------------
Disassembling what they write to 0x5C7EA yields this:
Code:
ROM:0005C7EA                 BL      0x40300
Looks like it is indeed Thumb code.
If we could dump the eMMC RAM, we would understand what has been changed.


By inspecting some code, it seems that we know how to dump the eMMC RAM:
Look at the function mmc_set_wearlevel_page in line 206. It patches the RAM (using the method mentioned before), then it validates what it has written (in lines 255-290). Seems that the procedure to read the RAM is as following:
1. CMD62(0xEFAC62EC) CMD62(0x10210002) to enter RAM reading mode
2. MMC_ERASE_GROUP_START(Address to read) MMC_ERASE_GROUP_END(Length to read) MMC_ERASE(0)
3. MMC_READ_SINGLE_BLOCK to read the data
4. CMD62(0xEFAC62EC) CMD62(0xDECCEE) to exit RAM reading mode


I don't want to run this on my device, because I'm afraid - messing with the eMMC doesn't sound like a very good idea on my device (I don't have a spare one).
Does someone have a development device which he doesn't mind to risk, and want to dump the eMMC firmware from it?

--> **Ultimate GS3 sudden death thread**
Just wanted to link to a prior thread with some information/testing that as been done. Completely understand if you nuke it because it doesn't meet the proper criteria or is way to noobish to be posted here. Anyway, just though it _might_ help, so giving it a shot..
The Following 3 Users Say Thank You to sponix2ipfw For This Useful Post: [ View ]
15th January 2013, 12:42 AM |#3  
OP Member
Thanks Meter: 259
 
More
So I decided to do a small RAM dump after all.

Before the patch, 0x5C7EA reads FD F7 C2 FA, which is "BL 0x59D72".
As I thought, they replace a function call to the new one.

I will dump function 0x59D72 later this week.
The Following 16 Users Say Thank You to Oranav For This Useful Post: [ View ]
15th January 2013, 01:32 PM |#4  
Junior Member
Thanks Meter: 2
 
More
Quote:
Originally Posted by Oranav

So I decided to do a small RAM dump after all.

Before the patch, 0x5C7EA reads FD F7 C2 FA, which is "BL 0x59D72".
As I thought, they replace a function call to the new one.

I will dump function 0x59D72 later this week.

So it looks like the new function calls the old function, and then if it returns ZERO then in goes into an INFINITE loop?!?

Seems like an odd fix, maybe self presevation?

Oli
The Following 2 Users Say Thank You to odewdney For This Useful Post: [ View ]
15th January 2013, 05:02 PM |#5  
Senior Member
Thanks Meter: 305
 
More
Quote:
Originally Posted by odewdney

So it looks like the new function calls the old function, and then if it returns ZERO then in goes into an INFINITE loop?!?
Seems like an odd fix, maybe self presevation?
Oli

WELL... after I changed to XXELLA stock firmware and stock kernel in 01/13 my 06/12 SGS3 had the _first freeze ever_ on XXELLA. Maybe its completely unrelated and was only a random thing.

But, could it be, that this fix temporary (until reboot) locks the eMMC in a bad situation to avoid damaging internal data structures?
But then in this cases you get a phone freeze, cause the eMMC is temporary unaviable so the phone crashes until you reboot it. But it avoided eMMC data structure damage.

Sounds not very logical, but when you have to fix a problem and only have a few bytes to patch it (cause it must be run on every emmc-start), and the problem only occurs on a hand full devices (out of millions) then it is maybe acceptable to have a freeze instead of a dead eMMC in that rare cases that it occurs.

But this is only a idea... don't now if it is like that.

BR
Rob

PS: Oranav, thank you very much for your effort.
Last edited by Rob2222; 15th January 2013 at 05:07 PM.
The Following 4 Users Say Thank You to Rob2222 For This Useful Post: [ View ]
15th January 2013, 08:47 PM |#6  
OP Member
Thanks Meter: 259
 
More
Quote:
Originally Posted by odewdney

So it looks like the new function calls the old function, and then if it returns ZERO then in goes into an INFINITE loop?!?

Seems like an odd fix, maybe self presevation?

Oli

Right, haven't spotted this. Thanks for the observation.
Self preservation sounds possible.

Quote:
Originally Posted by Rob2222

WELL... after I changed to XXELLA stock firmware and stock kernel in 01/13 my 06/12 SGS3 had the _first freeze ever_ on XXELLA. Maybe its completely unrelated and was only a random thing.

But, could it be, that this fix temporary (until reboot) locks the eMMC in a bad situation to avoid damaging internal data structures?
But then in this cases you get a phone freeze, cause the eMMC is temporary unaviable so the phone crashes until you reboot it. But it avoided eMMC data structure damage.

Sounds not very logical, but when you have to fix a problem and only have a few bytes to patch it (cause it must be run on every emmc-start), and the problem only occurs on a hand full devices (out of millions) then it is maybe acceptable to have a freeze instead of a dead eMMC in that rare cases that it occurs.

But this is only a idea... don't now if it is like that.

BR
Rob

PS: Oranav, thank you very much for your effort.

This could be possible - this patch looks like a quick and dirty fix, so maybe they didn't have the time to properly fix this. Instead, they just avoid the bug absolutely (with the cost of data corruption).
But I don't think this would cause lockups - I believe the chip has a watchdog...


All in all, I think the best thing we can do right now is to dump the whole firmware out of it. I will do it soon.
Last edited by Oranav; 15th January 2013 at 09:26 PM.
The Following 10 Users Say Thank You to Oranav For This Useful Post: [ View ]
16th January 2013, 12:44 PM |#7  
Bäcker's Avatar
Senior Member
Thanks Meter: 198
 
More
there is also a chance that this is just a temporary workaround to prevent further bricking - until there is a final fix.

As of now we asume that this is the fix as it directly adresses the eMMC in concern, but all this is just based on asumptions.
17th January 2013, 02:10 PM |#8  
Member
Thanks Meter: 6
 
More
Quote:
Originally Posted by Rob2222

WELL... after I changed to XXELLA stock firmware and stock kernel in 01/13 my 06/12 SGS3 had the _first freeze ever_ on XXELLA. Maybe its completely unrelated and was only a random thing.

But, could it be, that this fix temporary (until reboot) locks the eMMC in a bad situation to avoid damaging internal data structures?
But then in this cases you get a phone freeze, cause the eMMC is temporary unaviable so the phone crashes until you reboot it. But it avoided eMMC data structure damage.

Sounds not very logical, but when you have to fix a problem and only have a few bytes to patch it (cause it must be run on every emmc-start), and the problem only occurs on a hand full devices (out of millions) then it is maybe acceptable to have a freeze instead of a dead eMMC in that rare cases that it occurs.

But this is only a idea... don't now if it is like that.

BR
Rob

PS: Oranav, thank you very much for your effort.

I think we can prove that the fix is actualy locking into loop ,but you must risk your phone :/ . If you are in ->
1) Flash back to older version without the fix
2) Wait and pray
a) If your phone dies --> the guys were right about the fix and the loop
b) If it stays alive ,then ...

We wont know for sure ,but your phone is maybe in the "perfect" condition for the test :/ .
(Sorry if this makes no sence)
17th January 2013, 02:46 PM |#9  
Senior Member
Thanks Meter: 305
 
More
@ivan:
Sorry, can't do that. Cause of high air humidity my humidity indicator is already a little soaked. Cause of the warranty-repair reports in out local forums I am not sure if I would get warranty. I think theres a fair chance, that they would deny my warranty. Cause of that I don't want to take any extra risk. I am on unrooted stock at the moment, cause of that.

@Oranav:
In our local forum we get some reports about a rising count of locks and restarts on S3's in the last time. Some like my freeze.
It also seems that after a while this problems gets better and even disappear completely.
Cause of that I am thinking, if it could be, that the fix maybe locks the eMMC if it finds a bad data structure, then this locks maybe could bring a phone-freeze (already stated that), and in the same time it repairs the data structure in this block with the bad data structure.
At least this would explain some rising count of freezes with the fix and the point, that the freezes become less and less over time.

I have no idea if it's that way, I just wanted to post it as theory to think about.

BTW, do you think when the watchdog restarts the eMMC that it goes that fast that the phone isn't affected?

BR
Robert
Last edited by Rob2222; 17th January 2013 at 02:48 PM.
17th January 2013, 08:07 PM |#10  
OP Member
Thanks Meter: 259
 
More
Quote:
Originally Posted by Rob2222

I have no idea if it's that way, I just wanted to post it as theory to think about.

The problem is that there are too many theories imaginable, but I can't think of no way to prove them but to reverse engineer the MoviNAND firmware.

Quote:
Originally Posted by Rob2222

BTW, do you think when the watchdog restarts the eMMC that it goes that fast that the phone isn't affected?

Certainly not. Watchdogs are slow, drivers running on a Cortex-A9 are blazing fast.
But I do think Linux's MMC driver can handle device restarts during an MMC operation.
The Following 2 Users Say Thank You to Oranav For This Useful Post: [ View ]
17th January 2013, 08:48 PM |#11  
Member
Thanks Meter: 21
 
More
Quote:
Originally Posted by Rob2222

@ivan:
Sorry, can't do that. Cause of high air humidity my humidity indicator is already a little soaked. Cause of the warranty-repair reports in out local forums I am not sure if I would get warranty. I think theres a fair chance, that they would deny my warranty. Cause of that I don't want to take any extra risk. I am on unrooted stock at the moment, cause of that.

@Oranav:
In our local forum we get some reports about a rising count of locks and restarts on S3's in the last time. Some like my freeze.
It also seems that after a while this problems gets better and even disappear completely.
Cause of that I am thinking, if it could be, that the fix maybe locks the eMMC if it finds a bad data structure, then this locks maybe could bring a phone-freeze (already stated that), and in the same time it repairs the data structure in this block with the bad data structure.
At least this would explain some rising count of freezes with the fix and the point, that the freezes become less and less over time.

I have no idea if it's that way, I just wanted to post it as theory to think about.

BTW, do you think when the watchdog restarts the eMMC that it goes that fast that the phone isn't affected?

BR
Robert

if those freezes are really caused by the firmware fix, I don't see how the would disappear over time...

I mean if it really is the case that the fix trades data corruption for eMMC survival, it would make sense to see freezes... but depending on what data is affected, they should only be treatable by reinstalling the affected app or deleting its data/cache.

updated theory:
for all we know the error condition where the eMMC dies is quite rare, since most devices have been used for month before they passed away. So under the assumption that the error condition appears randomly and that there is a chance of data corruption every time the condition appears with fixed kernel, we could expect to see freezes and other problems some time after the fix was applied. So that would explain the raising number of freezes reported. Furthermore I'd assume that people getting freezes would try to do something about it, like reinstalling/deleting apps wiping caches and/or data... or even reflashing, thus repairing the corrupted data. So freezes would disappear.

Wait, doesn't most evidence point to the fact that the error condition does NOT appear in a random fashion, since there were no cases in the beginning, and then a lot all of a sudden? Well, it might be that this is just the way we perceive the issue. Maybe there were cases before, but they weren't reported... phones died, people sent them in, got new ones and went on with their lives. But after some time the issue got known... bloggers wrote about it... and so on... people realized their phones died because of a wider problem... voila, steep raise in reported cases. Also the number of dying S3s would simply rise by a rising number of overall S3s, I mean Samsung kept selling phones, right?

But even under the assumption the bug is related to wear-levelling and not random, here is another idea: I have no clue how the algorithms work, but maybe it uses some sort of pseudo-random data to do whatever, with the same seed on all eMMCs... and thus all of them go through the same series of numbers. And now imagine the error condition is only triggered by a specific number or number set (say someone screwed up a boundary condition). Under this theory the error condition wouldn't appear randomly, but after a certain amount of write ops (or something).

Another question I asked myself is: shouldn't there be cases were data corruption does damage beyond all repair except for reflashing?
Well, it might be, but it seems reasonable to assume that it is a lot less likely than user-data corruption, since most critical files on the phone shouldn't be opened writeable (or are on a read-only mounted partition in the first place), hence shouldn't be affected by ****-ups during writes.

Like the previous poster I want to add that this is most likely all bull****... but it is what came to my mind looking for a theory that supports the data we got.

Read More
Post Reply Subscribe to Thread
Previous Thread Next Thread
Thread Tools
Display Modes