Image replicator generic: Difference between revisions

From embeddedTS Manuals
m (Links auto-updated for 2022 re-branding ( https://github.com/embeddedarm/buildroot-ts →‎ https://github.com/embeddedTS/buildroot-ts))
(Still WIP)
Line 1: Line 1:
The blast image and scripts require a minimum of 50 MB; this plus any disk images or tarballs used dictate the minimum disk size required. The USB drive must have at least 1 partition, with the first partition being formatted ext2/3 or fat32/vfat.
=== Image Capture ===
{{Note|The ext4 filesystem can be used instead of ext3, but it may require additional options. U-Boot does not support the 64bit addressing added as the default behavior in recent revisions of mkfs.ext4. If using e2fsprogs 1.43 or newer, the options "-O ^64bit,^metadata_csum" must be used with ext4 for proper compatibility. Older versions of e2fsprogs do not need these options passed nor are they needed for ext3.}}


<source lang=bash>
If no [[#Image_Write|valid images]] exist on the booted USB drive, the image capture process starts automatically. Startup scripts will check if the single partition of the USB drive can be expanded and do so if it is possible. If this process fails, image capture will be aborted and the red LED will blink to indicate a failure.
# This assumes USB drive is /dev/sdc:
sudo mkfs.ext3 /dev/sdc1
sudo mkdir /mnt/usb/
sudo mount /dev/sdc1 /mnt/usb/
sudo tar --numeric-owner -xf /path/to/tsimx6ul_usb_blaster-latest.tar.bz2 -C /mnt/usb/
</source>


At this point, disk images or tarballs would be copied to the <source inline>/mnt/usb/</source> folder and named as noted below. The latest disk images we provide can be downloaded from our FTP site, see the [[#Backup_.2F_Restore|backup and restore]] section for links to these files. Note that the script expects images and tarballs to have specific names.  When using an ext* filesystem, symlinks can be used.
While in progress, the red LED will remain solid while the green LED flashes once per second. Upon completion, the red LED turns off and the green LED will slowly blink to indicate success, while a blinking red LED indicates a failure. Note that while in progress, the USB disk is mounted read-write. It is not recommended to power off the unit while the image capture is in progress.


The formatted USB drive boots into a small Buildroot initramfs environment with filesystem and partitioning tools installed. This can be used to format SD, eMMC, or other disks. The Buildroot environment starts up and calls <source inline>/blast.sh</source> on the USB device. By default this script is set up to look for a number of of specific files on the USB disk and write these to media on the host device. Upon completion of the script the green or red LEDs will blink to visually indicate a pass or fail of the script. This script can be used without modification to write images from USB with these filenames:
To help diagnose failures, files in <source inline>/tmp/logs/</source> contain output from each capture process.
 
For each media present on the unit (SD/eMMC/SATA/etc.), the image capture process will do the following:
 
# Copy the whole disk image to an appropriately named file on the USB drive, e.g. <source inline>sdimage.dd</source>, no data is written to the source media and it is never mounted. The source disk can follow our stock partition layout, or implement a customized one.
# Perform an fsck on the Linux rootfs in the image file. Note that, if deviating from our standard partition layout, it may be necessary to modify the scripts handling the capture process.
# Perform a loopback mount of the Linux rootfs in the image file and sanitize the filesystem. The sanitization process removes temporary files (e.g. <source inline>/log/</source> files), unique files (e.g. ssh private key files, machine ID files), adds a file to indicate that it is a custom image with the date as its contents, etc. The full list of operations can be found in [https://github.com/embeddedTS/buildroot-ts/blob/main/technologic/board/usbprod-common/scripts/sanitize_linux_rootfs.sh this script]. It may be necessary to modify this file for unique situations.
# If the partition layout uses only a single partition, the filesystem is packed in to a tarball which is appropriately named and compressed, e.g. <source inline>sdimage.tar.xz</source>. The image file is then unmounted and removed.
# If the partition layout uses multiple partitions, then the image file is then unmounted, an md5sum of the image file taken, compressed to an appropriately named file, e.g. <source inline>emmcimage.dd.xz</source>, and then an md5sum of the compressed image is taken.
 
Note that when using this process, the USB drive used must be sized large enough to handle multiple compressed images as well as the uncompressed copy of the media image actively being worked with. If the image capture process runs out of space, the process will indicate a failure.
 
The images files captured are saved to the root folder of the USB disk. Upon completion, it is safe to remove power or unplug the USB drive.
 
For more details on the image capture process, see [https://github.com/embeddedTS/buildroot-ts/blob/main/technologic/board/usbprod-common/scripts/blast_funcs.sh this script].
 
 
=== Image Write ===
 
This process is used to write existing images to media on a target unit. If appropriately named disk images or tarballs (see table below) are present in the root folder of the USB drive when booted, then the startup scripts will start the image writing process. The latest disk images we provide can be downloaded from our FTP site, see the [[#Backup_.2F_Restore|backup and restore]] section for links to these files.
 
While in progress, the red LED will remain solid while the green LED flashes once per second. Upon completion, the red LED turns off and the green LED will slowly blink to indicate success,;hile a blinking red LED indicates a failure. Note that the USB drive remains read-only through the entire process, but target devices may be mounted or actively written. It is not advised to remove power or the USB drive until the whole process has completed.
 
To help diagnose failures, files in <source inline>/tmp/logs/</source> contain output from each writing process.
 
Note that the script expects images and tarballs to have specific names. When using an ext2/3/4 filesystem on the USB drive, symlinks can be used. The following table is the list of valid file names:


{| class=wikitable
{| class=wikitable
Line 25: Line 43:
* sdimage.tar.gz
* sdimage.tar.gz
* sdimage.tar
* sdimage.tar
| Tar of the filesystem.  This will repartition the SD card to a single partition and extract this tarball to the filesystem.  If present, a file named <source inline>/md5sums.txt</source> in the tarball will have its contents checked against after the tarball is extracted. This <source inline>md5sums.txt</source> file is optional and can be omitted, but it must not be blank if present. This file is present in our official images and is created during image capture with the image replicator tool.
| Tar of the filesystem.  This will repartition the SD card to a single partition and extract this tarball to the filesystem.  If present, a file named <source inline>/md5sums.txt</source> in the tarball will have its contents checked against the whole filesystem after the tarball is extracted. This <source inline>md5sums.txt</source> file is optional and can be omitted, but it must not be blank if present. This file is present in our official images and is created during image capture with the Image Replicator tool.
|-
|-
|
|
Line 32: Line 50:
* sdimage.dd.gz
* sdimage.dd.gz
* sdimage.dd
* sdimage.dd
| Disk image of the card. This will be written to the SD card block device directly. If present on the USB drive, a file named <source inline>sdimage.dd.md5</source> will be used to verify the data written to the SD card against this checksum. This file is provided with our official images and is created during image capture with the image replicator tool.
| Disk image of the card. This will be written to the SD card block device directly. If present on the USB drive, a file named <source inline>sdimage.dd.md5</source> will be used to verify the data written to the SD card against this checksum. This file is provided with our official images and is created during image capture with the Image Replicator tool.
|-
|-
! rowspan=2 | eMMC
! rowspan=2 | eMMC
Line 40: Line 58:
* emmcimage.tar.gz
* emmcimage.tar.gz
* emmcimage.tar
* emmcimage.tar
| Tar of the filesystem.  This will repartition the eMMC to a single partition and extract this tarball to the filesystem. If present, a file named <source inline>/md5sums.txt</source> in the tarball will have its contents checked against after the tarball is extracted. This <source inline>md5sums.txt</source> file is optional and can be omitted, but it must not be blank if present. This file is present in our official images and is created during image capture with the image replicator tool.
| Tar of the filesystem.  This will repartition the eMMC to a single partition and extract this tarball to the filesystem. If present, a file named <source inline>/md5sums.txt</source> in the tarball will have its contents checked against the whole filesystem after the tarball is extracted. This <source inline>md5sums.txt</source> file is optional and can be omitted, but it must not be blank if present. This file is present in our official images and is created during image capture with the Image Replicator tool.
|-
|-
|
|
Line 47: Line 65:
* emmcimage.dd.gz
* emmcimage.dd.gz
* emmcimage.dd
* emmcimage.dd
| Disk image of the card. This will be written to the eMMC block device directly. If present on the USB drive, a file named <source inline>emmcimage.dd.md5</source> will be used to verify the data written to the SD card against this checksum. This file is provided with our official images and is created during image capture with the image replicator tool.
| Disk image of the card. This will be written to the eMMC block device directly. If present on the USB drive, a file named <source inline>emmcimage.dd.md5</source> will be used to verify the data written to the SD card against this checksum. This file is provided with our official images and is created during image capture with the Image Replicator tool.
|-
|-
! rowspan=2 | SATA
! rowspan=2 | SATA
Line 55: Line 73:
* sataimage.tar.gz
* sataimage.tar.gz
* sataimage.tar
* sataimage.tar
| Tar of the filesystem.  This will repartition the first SATA drive with a single partition and extract this tarball to the filesystem. If present, a file named <source inline>/md5sums.txt</source> in the tarball will have its contents checked against after the tarball is extracted. This <source inline>md5sums.txt</source> file is optional and can be omitted, but it must not be blank if present. This file is present in our official images and is created during image capture with the image replicator tool.
| Tar of the filesystem.  This will repartition the first SATA drive with a single partition and extract this tarball to the filesystem. If present, a file named <source inline>/md5sums.txt</source> in the tarball will have its contents checked against the whole filesystem after the tarball is extracted. This <source inline>md5sums.txt</source> file is optional and can be omitted, but it must not be blank if present. This file is present in our official images and is created during image capture with the Image Replicator tool.
|-
|-
|
|
Line 62: Line 80:
* sataimage.dd.gz
* sataimage.dd.gz
* sataimage.dd
* sataimage.dd
| Disk image of the card. This will be written to the first SATA block device directly. If present on the USB drive, a file named <source inline>sataimage.dd.md5</source> will be used to verify the data written to the SD card against this checksum. This file is provided with our official images and is created during image capture with the image replicator tool.
| Disk image of the card. This will be written to the first SATA block device directly. If present on the USB drive, a file named <source inline>sataimage.dd.md5</source> will be used to verify the data written to the SD card against this checksum. This file is provided with our official images and is created during image capture with the Image Replicator tool.
|}
|}


Most users should be able to use the above script without modification.  Our buildroot sources are available from [https://github.com/embeddedTS/buildroot-ts our github repo].  To build the whole setup and create a USB drive, the following commands can be used. See the repository README for information on the project and how to make modifications.
Upon completion, it is safe to remove power or unplug the USB drive.
<source lang=bash>
make tsimx6ul_usbprod_defconfig && make
</source>


The resulting output file <source inline>tsimx6ul-usb-production-rootfs.tar.bz2</source> can be unpacked to the first partition of the USB drive as outlined above.
For more details on the image write process, see [https://github.com/embeddedTS/buildroot-ts/blob/main/technologic/board/usbprod-common/scripts/blast_funcs.sh this script].

Revision as of 19:13, 8 February 2022

Image Capture

If no valid images exist on the booted USB drive, the image capture process starts automatically. Startup scripts will check if the single partition of the USB drive can be expanded and do so if it is possible. If this process fails, image capture will be aborted and the red LED will blink to indicate a failure.

While in progress, the red LED will remain solid while the green LED flashes once per second. Upon completion, the red LED turns off and the green LED will slowly blink to indicate success, while a blinking red LED indicates a failure. Note that while in progress, the USB disk is mounted read-write. It is not recommended to power off the unit while the image capture is in progress.

To help diagnose failures, files in /tmp/logs/ contain output from each capture process.

For each media present on the unit (SD/eMMC/SATA/etc.), the image capture process will do the following:

  1. Copy the whole disk image to an appropriately named file on the USB drive, e.g. sdimage.dd, no data is written to the source media and it is never mounted. The source disk can follow our stock partition layout, or implement a customized one.
  2. Perform an fsck on the Linux rootfs in the image file. Note that, if deviating from our standard partition layout, it may be necessary to modify the scripts handling the capture process.
  3. Perform a loopback mount of the Linux rootfs in the image file and sanitize the filesystem. The sanitization process removes temporary files (e.g. /log/ files), unique files (e.g. ssh private key files, machine ID files), adds a file to indicate that it is a custom image with the date as its contents, etc. The full list of operations can be found in this script. It may be necessary to modify this file for unique situations.
  4. If the partition layout uses only a single partition, the filesystem is packed in to a tarball which is appropriately named and compressed, e.g. sdimage.tar.xz. The image file is then unmounted and removed.
  5. If the partition layout uses multiple partitions, then the image file is then unmounted, an md5sum of the image file taken, compressed to an appropriately named file, e.g. emmcimage.dd.xz, and then an md5sum of the compressed image is taken.

Note that when using this process, the USB drive used must be sized large enough to handle multiple compressed images as well as the uncompressed copy of the media image actively being worked with. If the image capture process runs out of space, the process will indicate a failure.

The images files captured are saved to the root folder of the USB disk. Upon completion, it is safe to remove power or unplug the USB drive.

For more details on the image capture process, see this script.


Image Write

This process is used to write existing images to media on a target unit. If appropriately named disk images or tarballs (see table below) are present in the root folder of the USB drive when booted, then the startup scripts will start the image writing process. The latest disk images we provide can be downloaded from our FTP site, see the backup and restore section for links to these files.

While in progress, the red LED will remain solid while the green LED flashes once per second. Upon completion, the red LED turns off and the green LED will slowly blink to indicate success,;hile a blinking red LED indicates a failure. Note that the USB drive remains read-only through the entire process, but target devices may be mounted or actively written. It is not advised to remove power or the USB drive until the whole process has completed.

To help diagnose failures, files in /tmp/logs/ contain output from each writing process.

Note that the script expects images and tarballs to have specific names. When using an ext2/3/4 filesystem on the USB drive, symlinks can be used. The following table is the list of valid file names:

Target device Accepted filenames Description
SD Card
  • sdimage.tar.xz
  • sdimage.tar.bz2
  • sdimage.tar.gz
  • sdimage.tar
 Tar of the filesystem. This will repartition the SD card to a single partition and extract this tarball to the filesystem. If present, a file named /md5sums.txt in the tarball will have its contents checked against the whole filesystem after the tarball is extracted. This md5sums.txt file is optional and can be omitted, but it must not be blank if present. This file is present in our official images and is created during image capture with the Image Replicator tool.
  • sdimage.dd.xz
  • sdimage.dd.bz2
  • sdimage.dd.gz
  • sdimage.dd
 Disk image of the card. This will be written to the SD card block device directly. If present on the USB drive, a file named sdimage.dd.md5 will be used to verify the data written to the SD card against this checksum. This file is provided with our official images and is created during image capture with the Image Replicator tool.
eMMC
  • emmcimage.tar.xz
  • emmcimage.tar.bz2
  • emmcimage.tar.gz
  • emmcimage.tar
 Tar of the filesystem. This will repartition the eMMC to a single partition and extract this tarball to the filesystem. If present, a file named /md5sums.txt in the tarball will have its contents checked against the whole filesystem after the tarball is extracted. This md5sums.txt file is optional and can be omitted, but it must not be blank if present. This file is present in our official images and is created during image capture with the Image Replicator tool.
  • emmcimage.dd.xz
  • emmcimage.dd.bz2
  • emmcimage.dd.gz
  • emmcimage.dd
 Disk image of the card. This will be written to the eMMC block device directly. If present on the USB drive, a file named emmcimage.dd.md5 will be used to verify the data written to the SD card against this checksum. This file is provided with our official images and is created during image capture with the Image Replicator tool.
SATA
  • sataimage.tar.xz
  • sataimage.tar.bz2
  • sataimage.tar.gz
  • sataimage.tar
 Tar of the filesystem. This will repartition the first SATA drive with a single partition and extract this tarball to the filesystem. If present, a file named /md5sums.txt in the tarball will have its contents checked against the whole filesystem after the tarball is extracted. This md5sums.txt file is optional and can be omitted, but it must not be blank if present. This file is present in our official images and is created during image capture with the Image Replicator tool.
  • sataimage.dd.xz
  • sataimage.dd.bz2
  • sataimage.dd.gz
  • sataimage.dd
 Disk image of the card. This will be written to the first SATA block device directly. If present on the USB drive, a file named sataimage.dd.md5 will be used to verify the data written to the SD card against this checksum. This file is provided with our official images and is created during image capture with the Image Replicator tool.

Upon completion, it is safe to remove power or unplug the USB drive.

For more details on the image write process, see this script.

Retrieved from "http://docs.embeddedTS.com/index.php?title=Image_replicator_generic&oldid=15233"