Image replicator intro: Difference between revisions

From embeddedTS Manuals
(Initial creation, still WIP)
 
No edit summary
 
(5 intermediate revisions by one other user not shown)
Line 1: Line 1:
This platform supports our Image Replicator tool. The Image Replicator tool is intended for use by our end customers as a means to format and load bootable images or filesystems on to a device's media as part of their production process. In addition to writing media, the Image Replicator tool is capable of capturing images from a unit. This golden image can then be used to write other units.
This platform supports our Image Replicator tool. The Image Replicator tool is intended for use by developers as a means to write bootable images or filesystems on to a device's media (SD / eMMC / SATA / etc.) as part of their production or preparation process. In addition to writing media, the Image Replicator tool is capable of capturing images from a device's media and preparing them to be written to other devices.


The Image Replicator tool is a bootable USB drive based on Busybox with a set of scripts to handle the capture and write processes. This process, however, is flexible and can be used as-is or adapted in to larger production processes to format and load data on to devices. The single USB drive can be used to capture images from a unit, and then can be inserted in to other units to write those same images back to other devices.
The Image Replicator tool is a USB disk image that can be booted on a target device to capture or write its media directly without the need for a host workstation. The USB disk image is based on Buildroot and contains a set of scripts which handle the capture and write process. The process and its scripts are flexible and can be used as-is or adapted in to larger production processes to format and load data on to devices. The single USB drive can be used to capture images from a device, and then can be inserted in to other devices to write those same images on to other devices. The capture process is not necessary if it is not needed. Images for the target device can be copied to the USB drive, booted on compatible units, and have the target images written to that unit's media.




''' Image Capture Process '''
''' Image Capture Process '''


1. Once booted, the scripts check to see if any valid images files exist on the drive. If none do, then it starts the capture process.
The image capture process performs the following steps. For more detailed information, see the [[#Image_Capture|Image Capture section]] below.
2. For each media present (eMMC, SD, SATA, etc.), the tool will repeat the following steps 3 through 7
3. Copy the whole disk image to an appropriately named file on the USB drive, e.g. <source inline>sdimage.dd</source>, no data is written to the source media and it is never mounted. The source disk can follow our stock partition layout, or implement a customized one.
4. Perform an fsck on the Linux rootfs in the image file. Note that, if deviating from our standard partition layout, it may be necessary to modify the scripts handling the capture process.
5. Perform a loopback mount of the Linux rootfs in the image file and sanitize the filesystem. The sanitization process removes temporary files (e.g. <source inline>/log/</source> files), unique files (e.g. ssh private key files, machine ID files), adds a file to indicate that it is a custom image with the date as its contents, etc. The full list of operations can be found in [https://github.com/embeddedTS/buildroot-ts/blob/main/technologic/board/usbprod-common/scripts/sanitize_linux_rootfs.sh this script]. It may be necessary to modify this file.
6. If the partition layout uses only a single partition, the filesystem is packed in to a tarball which is appropriately named and compressed, e.g. <source inline>sdimage.tar.xz</source>. The image file is then unmounted and removed.
7. If the partition layout uses multiple partitions, then <source inline>/dev/zero</source> is written to a single file, flushed to disk, and removed. This is to create large areas of null bytes on disk for better compression. The image file is then unmounted, an md5sum of the image file taken, compressed to an appropriately named file, e.g. <source inline>emmcimage.dd.xz</source>, and then an md5sum of the compressed image is taken.
8. After all images are captured, indicate that the process is done.


Note that when using this process, the USB drive used must be sized large enough to handle multiple compressed images as well as the uncompressed copy of the media image actively being worked with. If the image capture process runs out of space, the process will indicate a failure.
# If no valid images exist on the disk, image capture starts.
# For each valid media present on the unit, a bit for bit copy of the source is made.
# This image is mounted, sanitized (to remove unneeded files and allow safe copying of the image to other units), and saved as either a disk image or a tarball depending on the partition layout of the source disk.
# All images and tarballs are compressed, with both the output files having their MD5 hash saved as well as all of the files contained in the root partition having their MD5 hashes saved to a file for later verification.
 
The captured images and tarballs are named such that the USB Image Replicator disk can be immediately used to boot another unit and have it perform the Image Write process to write that unit's media with the captured images.
 
{{Note|When using this process, the USB drive used for the Image Replicator must be sized large enough to handle multiple compressed images as well as the uncompressed copy of the media image actively being worked with. If the image capture process runs out of space, the process will indicate a failure.}}
 
 
''' Image Write Process '''
 
The image write process performs the following steps. For more details information see the [[#Image_Write|Image Write section]] below.
 
# For each valid media present on the unit, find the first valid source image file for it.
# If a source image exists for a media that is not present on the unit, then the process indicates a failure.
# If the source image is a tarball, format the target disk with an appropriate filesystem, and unpack it to the target disk, verifying all files against the MD5 hash file list after they are written.
# If the source image is a disk image, write that to the target disk. If an MD5 file for the disk image exists, read back the written disk and compare it to the hash.

Latest revision as of 14:54, 18 May 2022

This platform supports our Image Replicator tool. The Image Replicator tool is intended for use by developers as a means to write bootable images or filesystems on to a device's media (SD / eMMC / SATA / etc.) as part of their production or preparation process. In addition to writing media, the Image Replicator tool is capable of capturing images from a device's media and preparing them to be written to other devices.

The Image Replicator tool is a USB disk image that can be booted on a target device to capture or write its media directly without the need for a host workstation. The USB disk image is based on Buildroot and contains a set of scripts which handle the capture and write process. The process and its scripts are flexible and can be used as-is or adapted in to larger production processes to format and load data on to devices. The single USB drive can be used to capture images from a device, and then can be inserted in to other devices to write those same images on to other devices. The capture process is not necessary if it is not needed. Images for the target device can be copied to the USB drive, booted on compatible units, and have the target images written to that unit's media.


Image Capture Process

The image capture process performs the following steps. For more detailed information, see the Image Capture section below.

  1. If no valid images exist on the disk, image capture starts.
  2. For each valid media present on the unit, a bit for bit copy of the source is made.
  3. This image is mounted, sanitized (to remove unneeded files and allow safe copying of the image to other units), and saved as either a disk image or a tarball depending on the partition layout of the source disk.
  4. All images and tarballs are compressed, with both the output files having their MD5 hash saved as well as all of the files contained in the root partition having their MD5 hashes saved to a file for later verification.

The captured images and tarballs are named such that the USB Image Replicator disk can be immediately used to boot another unit and have it perform the Image Write process to write that unit's media with the captured images.

Note: When using this process, the USB drive used for the Image Replicator must be sized large enough to handle multiple compressed images as well as the uncompressed copy of the media image actively being worked with. If the image capture process runs out of space, the process will indicate a failure.


Image Write Process

The image write process performs the following steps. For more details information see the Image Write section below.

  1. For each valid media present on the unit, find the first valid source image file for it.
  2. If a source image exists for a media that is not present on the unit, then the process indicates a failure.
  3. If the source image is a tarball, format the target disk with an appropriate filesystem, and unpack it to the target disk, verifying all files against the MD5 hash file list after they are written.
  4. If the source image is a disk image, write that to the target disk. If an MD5 file for the disk image exists, read back the written disk and compare it to the hash.
Retrieved from "http://docs.embeddedTS.com/index.php?title=Image_replicator_intro&oldid=15500"