LOST.DIR & sdcard data corruption

Search This thread

rsage

Senior Member
Jul 28, 2010
795
170
I think what we're establishing here, and correct me if I'm wrong, but data corruption appears to be correlated with mount events on the Epic. The more times you mount, the higher probability of encountering data corruption.

Furthermore, there's instances in which the corrupt data was observed to be OK on a PC prior to the remount. And it's independent of how much the card is actually used on the Epic between mounts.

If that's all the case, then there's definitely something suspicious about the mounting process. It's certainly not spurious corruption. It could still be "bad cards," in the sense that the mounting process is correct but tickles something in the controller. But I'm inclined to believe it may be a software-level issue afterall.

mkasick, my perception of the problem lines up with your statements. Without very deliberate & detailed testing I'm hesitant to say anything beyond that because I'm not examining the hundreds/thousands of files on the card every time I mount it on a computer, and I'm not checking it 100% of the time when I remount it on the phone. Actually, lately, I have been checking the file count and size of LOST.DIR frequently, so I do have a fairly good idea of when files are being moved to that directory, but I wouldn't take my casual testing as definitive at this point.

I'm still curious if the problem persists across multiple SD cards in the same device.

Me too, installing a new highly-rated card tonight, should know more soon
 

rsage

Senior Member
Jul 28, 2010
795
170
The problem is back. Check out my thread for more info.

That stinks - so you're seeing corruption from just rebooting the phone. I saw a little of that but not as much as mounting via USB. Couple questions - are you on custom ROM/kernel (which one), and do you have journaling on or off?

I'm currently up to 372 files in LOST.DIR, about 1.33gb. Largest files are around 150mb.

Edit: can we get these two threads merged?
 

zman519

Senior Member
Oct 27, 2010
2,240
255
the Land of Beer & Cheese
That stinks - so you're seeing corruption from just rebooting the phone. I saw a little of that but not as much as mounting via USB. Couple questions - are you on custom ROM/kernel (which one), and do you have journaling on or off?

I'm currently up to 372 files in LOST.DIR, about 1.33gb. Largest files are around 150mb.

Edit: can we get these two threads merged?

Merged yep ;-) sounds good to me.

Rom = midnight rom 5.2 rfs journaling On kernel that comes with the rom bonsai 4.0.1

Sent from the Drivers Seat of my Suby txting and Driving doing 100MPH+ in a school zone! Ha.
 

parasense

Member
Mar 25, 2011
19
1
Arlington, TX
To recover files on a damaged sdcard, use zar recovery. Short for zero action recovery, or something like that.

The files get corrupt because of volitility. The sdcard controler could be cheap which is why a better card solves, sandisk is rhe best and most expensive.

Thw android media scan is to blame in many case. Shutting phine down durring a process writing ro card, even one you not aware of, like background service.

Edit: also, format the partition and format the sdcard two or three times. I sugest you zero all the bits with a full format.

This type of issue is mostly fs corruption if the fat file system. If your using a journaling system, stop. It is way too much overhead for sdcard. These are volitil memory.

Deliver by Epic 4G xda app
 
Last edited:

rsage

Senior Member
Jul 28, 2010
795
170
To recover files on a damaged sdcard, use zar recovery. Short for zero action recovery, or something like that.

Found it, will look into it, thanks

Edit: also, format the partition and format the sdcard two or three times. I sugest you zero all the bits with a full format.

Have tried that, it didn't help

This type of issue is mostly fs corruption if the fat file system. If your using a journaling system, stop. It is way too much overhead for sdcard. These are volitil memory.

Hmmm, that goes against everything I've read and understand about file systems and journaling. Can you provide a link with more information?
 

rsage

Senior Member
Jul 28, 2010
795
170
Experiences with new sdcard

Ok, I've been rocking a new sdcard for about 24 hours. I've flashed a few ROMs, kernels, themes, shot photos and videos, copied tons of files to and from the card, & rebooted, unmounted, and mounted the card more times than I can count. I would normally have a couple dozen new files in LOST.DIR by now. Currently, LOST.DIR is clean, zero files, and I've had no data issues.

Something else interesting, I (along with many others) have never been able to consistently shoot HD video on my Epic (always got "recording failed" after a few seconds). Never was able to solve it, but that problem is gone now too. Probably due to the higher class of this card (6), or the quality of the card, or both.

I'm going to keep an eye on it, but so far so good.
 

zman519

Senior Member
Oct 27, 2010
2,240
255
the Land of Beer & Cheese
Ok, I've been rocking a new sdcard for about 24 hours. I've flashed a few ROMs, kernels, themes, shot photos and videos, copied tons of files to and from the card, & rebooted, unmounted, and mounted the card more times than I can count. I would normally have a couple dozen new files in LOST.DIR by now. Currently, LOST.DIR is clean, zero files, and I've had no data issues.

Something else interesting, I (along with many others) have never been able to consistently shoot HD video on my Epic (always got "recording failed" after a few seconds). Never was able to solve it, but that problem is gone now too. Probably due to the higher class of this card (6), or the quality of the card, or both.

I'm going to keep an eye on it, but so far so good.

Cool that's good to hear. What sd card did you buy?

I'm looking for a 32GB high class, but if its a lot of $ I may end up buying a 16GB.

Sent from the Drivers Seat of my Suby txting and Driving doing 100MPH+ in a school zone! Ha.
 

rsage

Senior Member
Jul 28, 2010
795
170
Cool that's good to hear. What sd card did you buy?

I'm looking for a 32GB high class, but if its a lot of $ I may end up buying a 16GB.

I got the same card I'm using in my nook color - 8gb class 6 transcend. I think it's going to be too small for my needs in the long run, but I wanted to get a card that I knew was high-quality to test the data corruption issue with.
 

parasense

Member
Mar 25, 2011
19
1
Arlington, TX
I used to work on filesystem design years ago, but I know a thing or two about filesystems, volatile storage, and how things go wrong. But don't believe me, I'm just somebody on the internet. Here is what I have to say: The FAT32 filesystem is the most common problem people have, though not sure you were using FAT? Anyhow, most of the time the data blocks (metacdata) that points to (referrences) the actualy data (your pictures, etc) gets corrupted. That is to say the reference gets trashed up in the FAT tables where an index is kept. Other times the FAT table get corrupted in various interesting ways, and when new data is sent to save on the SDcard the regionof the FAT table is writen to again, but perhaps not completely. The basic idea is a block of storage used for the FAT table is made relevant so it's now scanned when the filesystem is traversed, but parts of that data block are still having old pointers, and things get weird.

I'm just trying to stimulate your mind here, and I'll stop going into all the scenarios.

So anyhow the same goes for journaling, SDCARDS should never have journaling enabled in my humble opinion. They might help recover files, at the cost of destroying your sdcard. All those meta data and data sync's to the sdcard are expensive, and each bit on an sdcard has a 1 billion bit-flip cycle, which means the areas of the sdcard that are written to a billion times are forever gone. A journaling filesystem will used a dedicated area of the disk for the journal, and that area is constantly being written to.

That said, my advise before about the full format of the sdcard would touch every bits at least once, and shorten the lifespan. Pretty much anything, so it's not that big a deal, but something to keep in mind.
 

dito33

Senior Member
Jul 13, 2010
881
38
In a search of the most stable settings for my phone experiencing no boot with ext4 when no journaling also even with journaling on I have no fate in ON or OFF went to stock RFS thinking this is it for me. But guess what it happened the worst ever I always was afraid of not recovering SD card with all the tools on my pc I have lost pics and videos forever. When sd card is inserted in pc calls for format but then can't be done, and phone wouldn't recognize it. All this happened when I was backing settings of AWD-EX when phone froze on a generic 16gb card that I got from my insurance replacement with the phone. I'm puzzled what's next I know of backing up but sick of it, wanting to go dumb phones maybe. So I think writing to sd as in earlier post was mentioned causes the most corruption vs mount unmount.

Sent from my SPH-D700 using XDA App
 

zman519

Senior Member
Oct 27, 2010
2,240
255
the Land of Beer & Cheese
I used to work on filesystem design years ago, but I know a thing or two about filesystems, volatile storage, and how things go wrong. But don't believe me, I'm just somebody on the internet. Here is what I have to say: The FAT32 filesystem is the most common problem people have, though not sure you were using FAT? Anyhow, most of the time the data blocks (metacdata) that points to (referrences) the actualy data (your pictures, etc) gets corrupted. That is to say the reference gets trashed up in the FAT tables where an index is kept. Other times the FAT table get corrupted in various interesting ways, and when new data is sent to save on the SDcard the regionof the FAT table is writen to again, but perhaps not completely. The basic idea is a block of storage used for the FAT table is made relevant so it's now scanned when the filesystem is traversed, but parts of that data block are still having old pointers, and things get weird.

I'm just trying to stimulate your mind here, and I'll stop going into all the scenarios.

So anyhow the same goes for journaling, SDCARDS should never have journaling enabled in my humble opinion. They might help recover files, at the cost of destroying your sdcard. All those meta data and data sync's to the sdcard are expensive, and each bit on an sdcard has a 1 billion bit-flip cycle, which means the areas of the sdcard that are written to a billion times are forever gone. A journaling filesystem will used a dedicated area of the disk for the journal, and that area is constantly being written to.

That said, my advise before about the full format of the sdcard would touch every bits at least once, and shorten the lifespan. Pretty much anything, so it's not that big a deal, but something to keep in mind.

So if your sd card is FAT32 and you have jounaling ON in a rom. It dose not mean that your sd card has jjounaling turnd ON.... correct?

One more Q for you. What size do you recommend the file allocation size be for a FAT32 partiton on a micro sdcard?

I just orderd a 32GB lexar class 10 sdcard. And a few of the reviews on amazon talked about 64KB.

Sent from the Drivers Seat of my Suby txting and Driving doing 100MPH+ in a school zone! Ha.
 

DroidApprentice

Senior Member
Nov 7, 2010
778
79
Reading this has me a bit concerned. Even though I only run rooted stock with RFS and jounaling, I believe. Is the Samsung 16GB card OK or not? Sammy makes it themselves, and so far nothing bad has happened with mine. Just looking for opinions.
 

rsage

Senior Member
Jul 28, 2010
795
170
Reading this has me a bit concerned. Even though I only run rooted stock with RFS and jounaling, I believe. Is the Samsung 16GB card OK or not? Sammy makes it themselves, and so far nothing bad has happened with mine. Just looking for opinions.

Unless this is happening to everyone with Sammy cards, probably just isolated bad batches, or even more likely just the usual failure rate with this kind of media. Judging from the sdcard media reviews on newegg and other retailer sites, even the best make/model cards have some that are DOA or fail shortly after purchase. Just my best guess...

Edit: though these cards don't seem to be experiencing outright failure, so I don't know... hopefully someone more knowledgeable can chime in...
 
Last edited:

bobo9900

Senior Member
Dec 27, 2009
86
8
Hi rsage,

I've been having this same issue on my defy. stock rom eclair with 2gb sandisk.
whenever I copy file to sdcard, after mount/unmount sdcard or restart phone, I see some of my files gone, and lost.dir folder filling up.

Tried to format several times, format with fat or fat32, just the same.
I notice when I run chkdsk from windows while phone connected to computer as mass storage, the drive sometimes forcely dismount and remount back.

Can you please confirm, does a higher quality sdcard like class 6 fix this corruption issue?
 
Last edited:

rsage

Senior Member
Jul 28, 2010
795
170
Can you please confirm, does a higher quality sdcard like class 6 fix this corruption issue?

That's a tough one to answer definitively. A new, higher quality, higher class card seems to be alleviating these issues for me. But it could just be because my old OEM card was defective in some way. Any new high-quality card may have worked (regardless of class).
 

Top Liked Posts

  • There are no posts matching your filters.
  • 6
    I've been experiencing some maddening issues with corruption of sdcard data. After much googling and finding some other reported occurrences of this, but not finding much info that's been helpful in solving the problem, I thought I'd compile and post some data in the hopes that others could add to it - hopefully leading to some kind of conclusion and maybe even a solution. I apologize because this is going to be a long post - hopefully it will be worth it.

    Issue description:
    There's a directory called LOST.DIR on sdcards in android phones. From what I've gathered, it contains lost/corrupted/orphaned files from the sdcard. When these files are moved to LOST.DIR, they're renamed with seemingly random strings of numbers and any file suffixes are stripped. Other OS's (Mac, Windows, etc.) have similar directories that serve similar purposes. The problem that I and others are experiencing is an alarming rate of sdcard data corruption, causing the LOST.DIR directory to fill up rather quickly, sometimes reaching multiple GB's in size. I’ve seen up to a dozen new files per day appearing in this directory. It's gotten to the point where I no longer trust that any data I put on my sdcard will remain intact.

    I've read a few theories on how/why this happens. I've read that this is a Mac-related issue and only happens when mounting the sdcard on a Mac. This is not the case, as I've had this issue occur regardless if I'm mounting the card on a Mac, on a Windows PC (XP or 7), or even when mounting the card in a card reader and not from the phone directly (though I don't think the data corruption is occurring during mounting the card, but rather when the card is unmounted from the phone either when shutting the phone down or when unmounting the card from the phone during a request to remount the card via USB). I've even had a few (rare) occurrences of new files appearing in LOST.DIR without mounting the card on any device except the phone itself. I've also read that it's ok to just delete the files in this directory to reclaim space on the sdcard, but I've found that the files in LOST.DIR are often files I need, or files that various applications need to function. For the most part, when files are found in LOST.DIR, they've actually been removed from their original location. Deleting the files in LOST.DIR means you'll never be able to recover those files.

    Once files end up in LOST.DIR, it's painstaking to recover them. Since they no longer have file types/suffixes, you may have to try opening the files in a utility to be able to tell what they are. Or by trial and error try appending different suffixes until you can open the files in an appropriate application.

    Examples of files from my sdcard that have ended up in LOST.DIR:

    • whole mp3's
    • partial mp3's (more on this below)
    • photos & videos taken with the Epic's camera
    • xml config files
    • cached browser files
    • sql lite db's stored by various apps on the sdcard (causing one or two apps to FC since the sql lite files are no longer where the app expects them to be)
    • flashable zip files
    • etc.

    Basically, any file I have on my sdcard can become corrupted or orphaned and end up in LOST.DIR in a near-unidentifiable state. To add to the pain, sometimes the files in LOST.DIR are not complete files, and sometimes they're not moved there intact. I was playing an album stored on the sdcard, and one track cut off in the middle. I found the other half of the mp3 file in LOST.DIR. I've lost entire photos and videos to LOST.DIR, but for the most part the files that are moved there are munged together with other files. If you view some files in LOST.DIR with a utility that can open any file as text, you'll see that a single file in LOST.DIR can really be multiple files or parts of files munged together into one. I've seen parts of text files mixed in with jpg's from browser cache mixed in with html mixed in with xml mixed in with other unidentifiable data. Obviously if this happens the data is not recoverable.

    Some theories which I've ruled out or partially ruled out:
    • Mounting/unmounting method: I always shut down, mount, unmount, etc. the phone and the sdcard properly. If I have the card mounted on a Mac or PC, I always use the appropriate OS-specific method of unmounting the card before turning off USB mounting on the Epic. I ran tests where I kept track of every step I took in booting, unmounting, remounting, etc. the sdcard to confirm this. Other users have also ruled this out as a cause (though I'm sure improperly unmounting a card can cause data problems)
    • Journaling: I confirmed this happens regardless of whether journaling is turned off or on
    • Ext4 file system: don't think this has anything to do with it, as I've seen this happen on Ext4 and RFS
    • ROM's: I confirmed that this seems to be happening on multiple ROMs: midNIGHT/Bonsai, Baked Snack, stock deodexed, etc.
    • Kernels: Seems to be happening on multiple kernels: Bonsai kernel, Baked Snack kernel, maybe one or two others I've tried
    • Card file system corruption: I've reformatted my sdcard more times that I can count in trying to get rid of this problem, doesn't seem to have any effect
    • Epic4g itself: this seems to be happening on many different android phones, so I'm ruling out a design flaw with the Epic4g

    Theories I haven't ruled out:
    • android (froyo?) bug related to data integrity when mounting/unmounting the sdcard (let's hope not)
    • damaged or defective sdcard

    I've read that some people have had success in banishing this problem by moving to a higher-class sdcard (doesn't really make sense to me, but whatever). I'm currently using the OEM 16gb (I'm assuming class 2 or 4) card that came with the Epic. In researching class 6 cards, it seems that the 8gb cards are generally more reliable than the 16gb cards, so I ordered a highly-rated 8gb class 6 card (still waiting for its arrival). I'll gladly give up storage space if I can count on data integrity. If it's a low-level android bug, then who knows if/when it will ever get dealt with.

    Here are two links that have more info and user experiences with this issue:
    - Google Code issue:
    http://code.google.com/p/android/issues/detail?id=2500

    - ZDNet article:
    http://www.zdnet.com/blog/sybase/could-your-sd-card-cause-your-android-smartphone-to-crash-or-lose-data/1128

    To be honest, I don't know how long I've been experiencing this problem. I don't know if it happened on Eclair, but it definitely happens on Froyo.

    I apologize that this is so long, but hopefully it will help someone else and possibly even lead to some helpful information in dealing with this problem. I'll post my findings once I receive the new class 6 sdcard.

    Peace.
    2
    If I had to guess, it sounds like silent write corruption on the SD card, whereby writes to the card are reported successful, but the written blocks are (ocasionally) corrupted, which don't show up until they're later read. In which case I'd suspect the controller on the SD card itself is to blame.

    If it wasn't silent corruption, i.e., writes were returning errors, the SD card should remount read-only automatically and there would be some OS-level notification. Although I suppose any such notification could be ignored.

    The mounting issue is possibly a red herring, for two reasons. First, data blocks written to the SD card are also cached in RAM. Thus, once mounted, most of the read operations on critical file system metadata will come out of the file system buffer cache and not actually from the SD card. So even if the write corruption is an on-going thing, it could easily go masked until a device reboot or SD card unmount/remount, at which point the cached blocks are purged and (the corrupted versions) read from the card.

    Second, fsck_msdos, which is responsible for populating LOST.DIR, runs at the time the SD card is mounted which is why mount events often corresponds to lost files showing up there.

    Here are two links that have more info and user experiences with this issue:
    - Google Code issue:
    - ZDNet article:
    Not to outright dismiss the existence of a kernel-level software bug, but the fact that relatively few folks for a given device are frequently reporting the issue leads me to think it's often hardware (likely SD card) related, or a result of a very unusual set of circumstances. At least, for the many bugs folks report having on the Epic, rampant SD card data corruption is not one I'm aware of.

    See if the new SD card helps, if it does, I'd say you just have a bum card and wouldn't even blame it on a particular brand or class.

    If you continue to have issues even with the new card, it would be nice to rule out a hardware defect with your particular Epic, but short of you getting a new phone or other people reporting the same issue, that would be difficult to do.

    Otherwise the two best things to do are to: (i) figure out a set of files/operations/whatever that reliably repeat the corruption issue, if possible, and (ii) narrow the sources of corruption down by avoiding mounting the SD card on other machines. You may have to occasoinally "Unmount SD card" and remount it from the Settings -> Sd card and phone storage for the corruption to show up though.
    2
    *lol* You'll get no argument from me.

    I'm not one to segregate information based on phone. This seems to be an issue regardless of the phone, so in actuality, if you want to be picky, this whole thread belongs in the general android section. But honestly, I really don't care much about the model, I care about resolution of the common issue :) I figure that my particular situation is informative b/c it is something that hasn't been experienced before (or @ least posted), so I posted it.
    1
    Very good theory, though I and others have noticed one thing that goes against it - some of the files were living and being used on the sdcard for quite a while before they got corrupted and moved to LOST.DIR.
    The Wikipedia page on the FAT file system explains how it works a bit, although I imagine there's more illustrative examples out there. But basically, file data blocks are stored in clusters, and there's a table that contains entires for each cluster in the file system, where each table entry contains the index (location) of the next cluster, if that cluster is part of a file.

    So it's quite possible that file data blocks for a given file are written out fine, but when the FAT is modified while writing/modifying a file that happens to be adjacent to it, the cluster indexes for the existing file are corrupted.

    1) I had listened to the affected mp3 several times before one day it suddenly cut off near the 1/2 way mark, after which I discovered the other half of the file was in LOST.DIR
    This is likely the result of a single corrupt cluster index. The corrupt index points to a cluster in the middle of the mp3, and if it were accidentally changed to "0" then (roughly) half the file is lopped off. Later, when fsck_msdos runs and scans the FAT and directory entries, it sees a chain of valid clusters that's not attached to any file. So it gives it a new file name and stuffs it in LOST.DIR.

    When the latter happened, a zero byte file with the same name existed in the original file location, but the real file had been renamed as something like "567801" and moved to LOST.DIR.
    If the entire file was in LOST.DIR, it sounds like a corrupt directory entry. That's a bit more suspicious, particularly if other part of the directory entry (name, time, etc.) were OK.

    Most of the corrupt/orphaned files I'm seeing in LOST.DIR wouldn't necessarily cause any overt symptom that anyone would notice - files like browser cache files, etc.
    One way orphaned files happen is when a file is deleted, and it's corresponding directory entry is removed, but the updated FAT hasn't been written out yet. The phone shouldn't be doing this too often, as long as the SD card is always properly unmounted (e.g., phone is properly shutdown). I could see computers doing this rather frequently, particular if the USB device isn't "safely removed".

    Afterall, orphaned files in that instance aren't a safety/corruption issue. There's just less free space in the file system until a fsck or other consistency check tool is run.

    In any event, I don't find the existence of these alone to be terribly surprising, especially in absence of missing files or other more significant corruption.

    One of the most common things I see on these boards is sudden and unexplained FC's of various apps, sometimes remedied by clearing data for the app, sometimes by reinstalling the app, and often users simply wipe all data and start over - never sure what the cause of the problem was.
    In almost all instances I expect those are due to data corruption on the /data file system, which, with ext4 can often happen on a phone crash or battery pull. Even with a journal, ext4 uses a delayed allocation scheme whereby newly allocated data may not be committed up to a few minutes after being "written", a window during which if the phone crashes, those files will experience data loss.

    Every phone that's mentioned in various threads on this issue, or other android phones I've personally checked, have had LOST.DIR and LOST.DIR has never been empty.
    As an anecdote: I usually end up with 2-3 orphaned files in LOST.DIR everytime I do a kernel upgrade, which I issue from within Android using redbend_ua without a clean shutdown. But I've never seen more obvious corruption. And I never see files in LOST.DIR after preforming a clean shutdown and clean unmounts.

    But I think it's at least a possibility that the issue is more widespread and users may not be noticing it due to the nature of the affected files or maybe they're thinking it's some other problem.
    You may be right about that, particularly if the affected files are thumbnails or related to the media cache.

    Still, I suspect if the problem were somewhat-spread among Epic owners here, we'd see a lot more posts about SD card corruption.

    do you think it's possible that the characteristics of certain ROM's, kernels, CW versions, or simply the act of flashing things all the time could have anything to do with this?
    Since most of those activities don't result in SD card mutations, I suspect not. Also, it doesn't look like "the problem" is confined to users to root and custom ROM.

    A few more notes:

    If it always happens after a phone crash, random reboot, etc, it's just file system corruption and there's little that can be done. Amusingly, RFS is essentially a "journaled FAT" and would solve that problem, but it's not
    used on SD cards on Samsung devices, which otherwise seems like a perfectly appropriate use for it.

    I'm suspicious of it being a bug in the file system driver. If it was, corruption would be discovered at random times instead of specifically after a reboot or remount event, which is when folks seem to report it happening.

    It could be a bug in fsck_msdos. I'd be a bit surprised if it was, but I don't think fsck_msdos is used particularly often outside of Android or other embedded devices, so the code may not be as well vetted as, say, fsck.ext3. One way to tell would be to disable fsck_msdos entirely and see if data still goes missing.
    1
    The problem still exists in my class 10 SanDisk micro sd card used with Samsung galaxy s4

    Any ideas for quick recovery
    I say we fund and make a software to end this problem and copyright it :) anyone up for business

    The software/app just has to recognise what extensions the files are and where they are supposed to go out make new files and fill them with the same extensions

    Sent from my GT-I9500 using XDA Premium 4 mobile app