Filesystem Corruption with SD cards
Disk corruption is a common issue in embedded development and considerations must be made for a robust system. The Sandisk SD cards we test with and provide can support on average 8TB of writes without any corruption.
Counterfeit cards, or unsupported media
It is estimated that a third of all Sandisk branded flash cards are fake. We recommend Sandisk SD cards as that is what we use for our testing, but make sure you use a reputable source for acquiring any flash media.
If you use any other brand or supplier we strongly recommend testing the cards thoroughly with our board before going into production. We recommend avoiding ATP flash media as well as "Industrial SD cards". We have experienced corruption with Industrial Cards, though they seem to work if multiwrite is disabled but this makes the card extremely slow to write.
You can use sdctl to evaluate the card which will write and verify data until the card shows any corruption. This test can take roughly 4 weeks when using a Sandisk card, but obvious incompatibilities will show up almost immediately or within a few days.
# This will launch sdctl in the background and
sdctl --dblstor-stresstest &
To see the progress of the stresstest you can view the log with:
sdctl --dmesg-follow
Once per second dmesg will output:
<1> <2> <3> <4> /<5> <6> <7> <8>/<9> <10> Jul 2 15:53:06 dblstor 0: reqs: 864 writes: 3.53MB/583.48MB reads: 0KB seeks: 0 errs: 0/0 status: OK
- Date / Time
- SD Socket # (LUN)
- Write Requests
- MB Written since last log
- Write Total MB
- Read Total MB
- Seeks since last log
- Errors
- Unrecoverable Errors
- Current card status
Once sdctl detects a failure it will print messages with more details on the failures. For example:
May 7 17:22:20 0: 515 more contiguous CRCFAILS May 7 17:22:20 0: FALLBACK_BADBLOCK: sector 0x1ad800, len 520 May 7 17:22:20 0: CRCFAIL: sector 0x1ada08/900.99MB, expected crc 0x5af0ed37, got crc:0x5af0ed37/sector:0x361208 fallback=1 May 7 17:22:20 0: CRCFAIL: sector 0x1ada09/900.99MB, expected crc 0x46aaeade, got crc:0x46aaeade/sector:0x361209 fallback=1 May 7 17:22:20 0: CRCFAIL: sector 0x1ada0a/900.99MB, expected crc 0x55115f66, got crc:0x55115f66/sector:0x36120a fallback=1 May 7 17:22:20 0: CRCFAIL: sector 0x1ada0b/900.99MB, expected crc 0xd7136c18, got crc:0xd7136c18/sector:0x36120b fallback=1 May 7 17:22:20 0: CRCFAIL: sector 0x1ada0c/900.99MB, expected crc 0xd6d75d2d, got crc:0xd6d75d2d/sector:0x36120c fallback=1 May 7 17:22:20 0: FALLBACK_BADBLOCK: sector 0x1ada08, len 1528 May 7 17:22:21 0: 1523 more contiguous CRCFAILS May 7 17:22:21 0: FALLBACK_RECOVERED: sector 0x1ad800, len 2048
CRCFAIL errors are the most common when a card begins failing which indicates that it read a block of data where the fallback and primary do not match. Contact us for more information on any other errors you receive.
Interrupting a Write
The most common issue resulting in data corruption is when powering off SD cards in the middle of a write. SD cards usually use MLC NAND flash for storage coupled with a manufacturer and model specific firmware. NAND flash has a limitation where in order to perform a single byte write it must first read about 128KB to 256KB (or more) containing that byte into memory and erase that sector on the NAND chip. This is the erase block size and can vary based on the card. It takes the block in memory, changes the single byte in that copy, erases the intended location on the flash and then commits it back to the disk. If power is lost in the middle of this write procedure it will lose whatever data was being written.
Most SD card firmwares also contain a wear leveling mechanism where they maintain a logical to physical mapping. This means that writing a contiguous file may actually end up in different areas in the NAND chip. If you interrupt a write cycle where it has erased a block, but not yet committed changes to the disk it will have lost data seemingly randomly across the card.
There are several strategies you can adopt to avoid or limit your chance of corruption.
Using a Completely Read Only FS
The most safe method is possible if you do not need to perform any writes that need to persist across reboots. If you are designing a data logger this is certainly not a good option, but if you're only responding to outside I/O this is your best choice. Once you get your application developed and ready for deployment you can run it from the initramfs. This is already a read only filesystem which will never write to the disk, but it mounts the Debian filesystem as read only so all of Debian is still available under /mnt/root. The largest difference is that the Debian init system is not used, so any daemons or services needed will need to be manually launched. See the
In the initramfs powering off in the middle of a read is still safe assuming no additional filesystems have been mounted read/write. You can still write data to /tmp/ which will go to memory, but it will be lost on a poweroff. If you require the full Debian filesystem you could use the linuxrc-sdroot-readonly startup script. This will mount the Debian filesystem on the SD card as read only, but will commit any attempted writes to a ramdisk using unionFS. The downside to this configuration while booting to Debian is that you have to manage all writes to the system or risk filling up your memory and causing a crash.
The next best method is to use a battery backup. Most UPS backup solutions, or one you build yourself should contain a method to see the battery level. Once you reach lower levels you can simply run a "shutdown -h now". When power is available the board will boot back up, but you may want to check in the initrd while it is read only if the battery is back to an acceptable level before making the SD card read/write.
The last option is to limit your writes to reduce the chance of corruption. Make all of your writes to a ramdisk like /tmp/ and copy them to the physical media periodically or when you know power will be safe. This is the option many consumer electronics choose. You can make writes more predictable by mounting with the options "sync" which will stop linux from buffering writes in RAM, and "noatime" which will prevent linux from writing access times when reading a file.
If your application is losing power from users disconnecting it or powering it off you may want to consider using an LED to indicate when it is not safe to turn off. Most people have seen cameras or video games that say "Do not remove power while X is displaying" which is usually a graphic of a disk or an LED like on floppy drives to indicate that there is a write that should not be interrupted. You can implement this very simply with a system call:
system("ts7500ctl --redledon");
usleep(350000);
int ret = write(...);
system("ts7500ctl --redledoff");
The usleep allows the user time to react to a new write. Most people should be able to react to an LED in about 215ms, but to be safe I would use a higher number.
You may also want to consider using the XNAND which from our testing has proven to be much more reliable with sudden power loss.
Other Resources