Dblstorctl

Overview

The 'dblstorctl' utility is designed to interface with and test our DoubleStore format from POSIX compliant systems. This has been tested to run on our boards, OSX, and common Linux distributions.

This can be downloaded for various systems as a binary here.

Usage

Usage: dblstorctl [OPTION] ...
Technologic Systems DoubleStore image manipulation.

General options:
  -p, --primary=FILE      Use FILE as primary store
  -F, --fallback=FILE     Use FILE as fallback store
  -e, --resilver          Force a resilvering of the dataset
  -Z, --size=SZ           Set dataset size to SZ (2.0G, 50M, etc)
  -I, --init              Initialize and zero the dataset
  -s, --stresstest        Start a stress test
  -S, --savebadblocks     Save bad blocks to files in the CWD
  -n, --random=SEED       Do random seeks for tests
  -d, --nbdserver=NBDSPEC Run NBD userspace block driver server
  -R, --read=N            Read N blocks to stdout
  -W, --write=N           Write N blocks
  -i, --writeimg=FILE     Use FILE as file to write
  -t, --writetest         Run write speed test
  -r, --readtest          Run read speed test
  -a, --read-ahead        Turn on 128kbyte read-ahead
  -c, --write-combine     Enables combining small writes together
  -P, --printmbr          Print MBR and partition table
  -M, --setmbr            Write MBR from environment variables
  -f, --foreground        Run NBD server in foreground
  -z, --blocksize=SZ      Use SZ bytes each read/write call
  -k, --seek=SECTOR       Seek to 512b sector number SECTOR
  -I, --bind=IPADDR       Bind NBD server to IPADDR
  -m, --dmesg             Print latest debug messages
  -T, --dmesg-follow      Follow/tail debug messages
  -Q, --stats             Print stats and status

When running a NBD server, NBDSPEC is a comma separated list of
devices and partitions for the NBD servers starting at port 7550.
e.g. "lun0:part1,lun0:disc" corresponds to 2 NBD servers, one at port
7550 serving the first partition of the dataset, and the other at TCP
port 7551 serving the whole disc device of the dataset.

Getting Started

This utility has 3 main functions.

• Stress testing cards
• Diagnosing Conflicts and Failures between stores
• Mounting and accessing doublestore partitions from other POSIX operating systems (Linux, OSX, BSD, etc)

Installing dblstorctl

We have a binary that can be used on x86 Linux, and boards supporting doublestore will already contain this utility, or equivalent functions provided by the sdctl driver.

# Copy this to somewhere in your $PATH.  This example works on most common Linux systems
cp dblstorctl /usr/local/bin/

Initializing and Mounting DoubleStore Media

To set a card up for DoubleStore you will need to init the card. You can do this with sdctl or dblstorctl.

A single card configuration where it splits the usable space of a single card in half for the redudant sectors:

dblstorctl --primary=/dev/device --init

For a two card configuration:

dblstorctl --primary=/dev/device1 --fallback=/dev/device2 --init

Once a two card system is initialized, there is no differentiation in terms of primary and fallback, they both have the exact same image and can be interchanged

To access a DoubleStore formatted SD card(s):

dblstorctl --primary=/dev/device(1) --fallback=/dev/device(2) --nbdserver=lun0:disc
nbd-client 127.0.0.1 7550 /dev/nbd0

Note: for this to work, Linux needs to have NBD drivers available, and the nbd-client application installed. TS products that support DoubleStore will have this available already, please check the documentation for other distributions for installing the needed tools. (1) and (2) are optional, and should be different in a two card system, and the same in a single card system.

From this point, /dev/nbd0 represents the whole disk, with sub partitions of /dev/nbd0pX being populated from the partition table. These can then be mounted and used like a normal linux disk. A background resilver will also be run once dblstorctl is first started to check for data consistancy.

When removing, unmount the NBD mounts first to flush the caches, kill nbd-client, and then kill dblstorctl.

Stress Testing Cards

The 4GB SDHC Sandisk cards we ship with have been thoroughly tested with DoubleStore. If you order your own SD cards from another distributor we recommend expending several cards from each batch to verify stability of the media. This way you can estimate a predictable lifetime of your product.

To stress test a card insert the SD card into your PC and run these commands. After you insert the card type 'dmesg' which on most Linux systems will print the block device such as /dev/sd? or /dev/mmcblk?. In my example this is /dev/sdf.

dblstorctl --primary=/dev/sdf --stresstest --foreground

Once you have a good idea of how long your cards last, you can test future batches to verify lifetime. For example if you found your cards make it consistently to 6TB you can test future cards to that specification. For example:

export STRESSTEST_LENGTH="6T"
dblstorctl --primary=/dev/yourSDdevice --stresstest

You can run the stresstest for a number of bytes (b), kilobytes (k), megabytes (m), gigabytes (g), or terabytes (t). You can also test by hours (h), days (d), or weeks (w).

Failure Indications

You can check the failure status of any DoubleStore card using dblstorctl:

dblstorctl --primary=yourSDdevice

This will return statistics on the disk and then quit:

 autodetect_ok=1
 size=0x1a8000
 humanized_size=889.19MB
 fb_offset=1736704
 ll_minsize=0x3ba000
 humanized_ll_minsize=2.00GB
 primary_tainted=0
 primary_failed=0
 fallback_tainted=0
 fallback_failed=0

The last 4 lines will indicate if it was tainted or failed.

Tainted Media

DoubleStore includes many checks throughout its operation to ensure the card is sane. As soon as we see something that should not be possible the card will become tainted. This can indicate corruption, but corruption that DoubleStore was able to prevent. Once you see a tainted card this is a warning that the card may be failing. If only one card is tainted then you should be able to initialize a doublestore card the same size as the working card and swap it out with the bad card. If you are on a single card configuration then you will want to move to a new card immediately. Since SD cards usually implement their own wear leveling mechanisms it is not predictable whether corruption or other strange behavior will stay specific to a primary or fallback.

Failed Media

After you see this message there has been proven data loss on both cards. There was a CRC failure on both the primary and the secondary for the same sector. You will need to restore from your backups to get this image to a usable state again.

Causes of a Tainted or Failed SD card

There are typically 3 items that will commonly cause failed or tainted media.

• Old age. After the card has been used for a long time you will run down on the lifetime of the card. Our testing has seen that the Sandisk SD cards we provide typically last from 8-12TB of writes. Reads have almost no visible impact on the lifetime of the media. The best solution to prepare for this is to watch how much data your application writes in a day using 'dblstorctl --stats' or 'sdctl --stats' and estimate accordingly to how long until you will hit the typical SD card write limit.

• Improper shutdowns. If you interrupt an SD card during a write cycle you will see unpredictable behavior. We can only speculate what happens on the closed firmwares included in each various SD cards, but we have seen large chunks of data set to 0 and corruption spread throughout the disk even beyond partition boundaries while interrupting write cycles. The safest method is to verify that the SD card never powers off while a filesystem residing on a physical media is mounted read/write. We provide startup scripts for all of our doublestore capable boards that can use a read only filesystem. This will allow writes to a ramdisk, but no permanent media. If you need to be able to read/write where power is not guaranteed you could implement a battery backup and use a DIO on the board to signal the ARM system to do a proper shutdown or remount read only.

• Counterfit or heavily used cards. We have heard multiple reports of customers buying their SD cards from the cheapest sources they can find, only to result in cards that only last for a few GB of writes. If you do not use the SD cards we provide then make sure you buy and test your cards from another reputable source. We test cards from every batch we receive with "dblstorctl --stresstest" to verify their expected lifetime has not gone down.

We also recommend using the kernel version that we provide with the filesystem we use by default. On each board we test each our driver under our provided kernel version with the provided filesystem.

Replace a single card in a double card configuration

If 1 card fails you can rewrite it with a second card formatted with all zeros.

Assuming your card is /dev/mmcblk0

dd if=/dev/zero of=/dev/mmcblk0 bs=32k && sync && sync

DoubleStore Debug Messages

Using dblstorctl or sdctl you can use --dmesg to print out the shared memory to see the previous logs of the application. This can show you what happened to a card and specifically how a failure mode presented itself. These are especially good to check after seeing a failure indication.

FALLBACK_CONFLICT

This happens when the primary does not match the fallback, but the CRC for both the primary and fallback are both good. In this case the fallback had the older data written. These messages happen commonly when there is an improper shutdown, or if dblstorctl is otherwise killed during a write operation. These do not indicate a failure, but this is not a message you should see regularly.

For example:

 4fc655e9 Resilver 55% done... 
 4fc655ea FALLBACK_CONFLICT: sect 0x39fa00, len 64, 1338397326 vs 1338395098
 4fc655ea FALLBACK_CONFLICT: sect 0x39fb00, len 64, 1338397326 vs 1338395098
 4fc655ea FALLBACK_CONFLICT: sect 0x39fc00, len 64, 1338397326 vs 1338395098
 4fc655ea FALLBACK_CONFLICT: sect 0x39fd00, len 64, 1338397326 vs 1338395098
 4fc655ea FALLBACK_CONFLICT: sect 0x39fe00, len 64, 1338397326 vs 1338395098
 4fc655ea FALLBACK_CONFLICT: sect 0x39ff00, len 64, 1338397326 vs 1338395098

PRIMARY_CONFLICT

This happens when the primary does not match the fallback, but the CRC for both the primary and fallback are both good. This specific message happens when the fallback has newer data than the primary which should not be possible. Normally DoubleStore will write to the primary, and then write to the fallback. This is possible if the write to the primary does not happen, but the write to the fallback succeeds.

BADBLOCK_RECOVERED

This happens when the CRC is invalid for a sector on the primary, but the fallback retained good data.

FALLBACK_BADBLOCK

This happens when the CRC is invalid for a sector on the fallback. These are only found during the resilvering process. This is recovered unless followed by a #BADBLOCK_UNRECOVERABLE message.

CRCFAIL

Doublestore will keep a CRC of the data and metadata for each sector. This indicates that the data is not valid. The metadata includes the sector number so if valid data and metadata are present, but the sector number doesn't match this will also be considered a CRC failure.

These messages do not indicate failure themselves, but will always be paired with a more specific error that will indicate if this can be ignored, will taint the card, or indicate a complete failure.