Image replicator intro: Difference between revisions

From embeddedTS Manuals
(Initial creation, still WIP)
 
(Still WIP)
Line 1: Line 1:
This platform supports our Image Replicator tool. The Image Replicator tool is intended for use by our end customers as a means to format and load bootable images or filesystems on to a device's media as part of their production process. In addition to writing media, the Image Replicator tool is capable of capturing images from a unit. This golden image can then be used to write other units.
This platform supports our Image Replicator tool. The Image Replicator tool is intended for use by our end customers as a means to format and load bootable images or filesystems on to a device's media (SD/eMMC/SATA/etc.) as part of their production process. In addition to writing media, the Image Replicator tool is capable of capturing images from a unit. This golden image can then be used to write other units.


The Image Replicator tool is a bootable USB drive based on Busybox with a set of scripts to handle the capture and write processes. This process, however, is flexible and can be used as-is or adapted in to larger production processes to format and load data on to devices. The single USB drive can be used to capture images from a unit, and then can be inserted in to other units to write those same images back to other devices.
The Image Replicator tool is a bootable USB drive based on Busybox with a set of scripts to handle the capture and write processes. This process, however, is flexible and can be used as-is or adapted in to larger production processes to format and load data on to devices. The single USB drive can be used to capture images from a unit, and then can be inserted in to other units to write those same images back to other devices.
Line 6: Line 6:
''' Image Capture Process '''
''' Image Capture Process '''


1. Once booted, the scripts check to see if any valid images files exist on the drive. If none do, then it starts the capture process.
The image capture process does the following steps. For more detailed information, see the NEED SECTION NAME below.
2. For each media present (eMMC, SD, SATA, etc.), the tool will repeat the following steps 3 through 7
 
3. Copy the whole disk image to an appropriately named file on the USB drive, e.g. <source inline>sdimage.dd</source>, no data is written to the source media and it is never mounted. The source disk can follow our stock partition layout, or implement a customized one.
# If no valid images exist on the disk, image capture starts.
4. Perform an fsck on the Linux rootfs in the image file. Note that, if deviating from our standard partition layout, it may be necessary to modify the scripts handling the capture process.
# For each valid media present on the unit, a bit for bit copy of the source is made.
5. Perform a loopback mount of the Linux rootfs in the image file and sanitize the filesystem. The sanitization process removes temporary files (e.g. <source inline>/log/</source> files), unique files (e.g. ssh private key files, machine ID files), adds a file to indicate that it is a custom image with the date as its contents, etc. The full list of operations can be found in [https://github.com/embeddedTS/buildroot-ts/blob/main/technologic/board/usbprod-common/scripts/sanitize_linux_rootfs.sh this script]. It may be necessary to modify this file.
# This image is mounted, sanitized (to remove unneeded files and safely allow copying to other units), and saved as either a disk image or a tarball depending on the partition layout of the source disk
6. If the partition layout uses only a single partition, the filesystem is packed in to a tarball which is appropriately named and compressed, e.g. <source inline>sdimage.tar.xz</source>. The image file is then unmounted and removed.
# All images and tarballs are compressed, with both the output files having their MD5 hash saved as well as all of the files contained in the root partition having their MD5 hashes saved to a file for later verification.
7. If the partition layout uses multiple partitions, then <source inline>/dev/zero</source> is written to a single file, flushed to disk, and removed. This is to create large areas of null bytes on disk for better compression. The image file is then unmounted, an md5sum of the image file taken, compressed to an appropriately named file, e.g. <source inline>emmcimage.dd.xz</source>, and then an md5sum of the compressed image is taken.
 
8. After all images are captured, indicate that the process is done.
The images and tarballs are named such that the disk can be immediately used to boot another unit and write the images to its media.


Note that when using this process, the USB drive used must be sized large enough to handle multiple compressed images as well as the uncompressed copy of the media image actively being worked with. If the image capture process runs out of space, the process will indicate a failure.
Note that when using this process, the USB drive used must be sized large enough to handle multiple compressed images as well as the uncompressed copy of the media image actively being worked with. If the image capture process runs out of space, the process will indicate a failure.
''' Image Write Process '''
The image write process takes the following steps. For more details information see NEED SECTION NAME below.
# For each valid media present on the unit, find the first valid source image file for it; either disk image or tarball.
# If a source image exists for a media that is not present on the unit, then the process indicates a failure.
# If the source image is a tarball, format the target disk with an appropriate filesystem, and unpack it to that disk, verifying all files against the MD5 hash file list after they are written.
# If the source image is a disk image, write that to the target disk. If an MD5 file for the disk image exists, read back the written disk and compare it to the hash.

Revision as of 19:14, 8 February 2022

This platform supports our Image Replicator tool. The Image Replicator tool is intended for use by our end customers as a means to format and load bootable images or filesystems on to a device's media (SD/eMMC/SATA/etc.) as part of their production process. In addition to writing media, the Image Replicator tool is capable of capturing images from a unit. This golden image can then be used to write other units.

The Image Replicator tool is a bootable USB drive based on Busybox with a set of scripts to handle the capture and write processes. This process, however, is flexible and can be used as-is or adapted in to larger production processes to format and load data on to devices. The single USB drive can be used to capture images from a unit, and then can be inserted in to other units to write those same images back to other devices.


Image Capture Process

The image capture process does the following steps. For more detailed information, see the NEED SECTION NAME below.

  1. If no valid images exist on the disk, image capture starts.
  2. For each valid media present on the unit, a bit for bit copy of the source is made.
  3. This image is mounted, sanitized (to remove unneeded files and safely allow copying to other units), and saved as either a disk image or a tarball depending on the partition layout of the source disk
  4. All images and tarballs are compressed, with both the output files having their MD5 hash saved as well as all of the files contained in the root partition having their MD5 hashes saved to a file for later verification.

The images and tarballs are named such that the disk can be immediately used to boot another unit and write the images to its media.

Note that when using this process, the USB drive used must be sized large enough to handle multiple compressed images as well as the uncompressed copy of the media image actively being worked with. If the image capture process runs out of space, the process will indicate a failure.


Image Write Process

The image write process takes the following steps. For more details information see NEED SECTION NAME below.

  1. For each valid media present on the unit, find the first valid source image file for it; either disk image or tarball.
  2. If a source image exists for a media that is not present on the unit, then the process indicates a failure.
  3. If the source image is a tarball, format the target disk with an appropriate filesystem, and unpack it to that disk, verifying all files against the MD5 hash file list after they are written.
  4. If the source image is a disk image, write that to the target disk. If an MD5 file for the disk image exists, read back the written disk and compare it to the hash.
Retrieved from "http://docs.embeddedTS.com/index.php?title=Image_replicator_intro&oldid=15234"