Image replicator generic: Difference between revisions

From embeddedTS Manuals
(Fix heading depths)
(Remove headers but keep anchor links. Link back to creating a disk section. Remove deprecated source tags.)
 
(15 intermediate revisions by 2 users not shown)
Line 1: Line 1:
==== Image Capture ====
Once a USB drive is formatted with the Image Replicator tool (see [[#Creating_a_USB_Image_Replicator_Disk|Creating a USB Image Replicator Disk]] for the correct files and process), [[#Booting_From_USB|boot to this USB drive]] (note that the Image Replicator already sets up the correct U-Boot boot scripts to boot to the USB drive, see the aforementioned section for details on how to make U-Boot call the scripts on the USB drive). This will start either [[#Image_Capture|image capture]] if no disk images/tarballs are present on the USB drive, or [[#Image_Write|image write]] if there are disk images/tarballs present on the USB drive.


If no [[#Image_Write|valid images]] exist on the booted USB drive, the image capture process starts automatically. Startup scripts will check if the single partition of the USB drive can be expanded and do so if it is possible. If this process fails, image capture will be aborted and the red LED will blink to indicate a failure.
The Image Replicator tool, while in progress, will flash the green LED once per second while the red LED remains solidly lit. Upon completion, the red LED turns off and the green LED will slowly blink to indicate success, while a blinking red LED with the green LED off indicates a failure.


While in progress, the red LED will remain solid while the green LED flashes once per second. Upon completion, the red LED turns off and the green LED will slowly blink to indicate success, while a blinking red LED indicates a failure. Note that while in progress, the USB disk is mounted read-write. It is not recommended to power off the unit while the image capture is in progress.
On each boot, startup scripts will check if the single partition of the USB drive can be expanded and do so if possible. If this process fails, then any further operations will not be run and the LEDs will blink to indicate a failure.


To help diagnose failures, files in <source inline>/tmp/logs/</source> contain output from each capture process.


For each media present on the unit (SD/eMMC/SATA/etc.), the image capture process will do the following:
''' Image Capture ''' <span id="Image_Capture"></span>


# Copy the whole disk image to an appropriately named file on the USB drive, e.g. <source inline>sdimage.dd</source>, no data is written to the source media and it is never mounted. The source disk can follow our stock partition layout, or implement a customized one.
If no [[#Image_Write|valid images to write]] exist on the booted USB Image Replicator drive, the image capture process starts automatically.
# Perform an fsck on the Linux rootfs in the image file. Note that, if deviating from our standard partition layout, it may be necessary to modify the scripts handling the capture process.
# Perform a loopback mount of the Linux rootfs in the image file and sanitize the filesystem. The sanitization process removes temporary files (e.g. <source inline>/log/</source> files), unique files (e.g. ssh private key files, machine ID files), adds a file to indicate that it is a custom image with the date as its contents, etc. The full list of operations can be found in [https://github.com/embeddedTS/buildroot-ts/blob/main/technologic/board/usbprod-common/scripts/sanitize_linux_rootfs.sh this script]. It may be necessary to modify this file for unique situations.
# If the partition layout uses only a single partition, the filesystem is packed in to a tarball which is appropriately named and compressed, e.g. <source inline>sdimage.tar.xz</source>. The image file is then unmounted and removed.
# If the partition layout uses multiple partitions, then the image file is then unmounted, an md5sum of the image file taken, compressed to an appropriately named file, e.g. <source inline>emmcimage.dd.xz</source>, and then an md5sum of the compressed image is taken.


Note that when using this process, the USB drive used must be sized large enough to handle multiple compressed images as well as the uncompressed copy of the media image actively being worked with. If the image capture process runs out of space, the process will indicate a failure.
Note that while in progress, the USB Image Replicator drive is mounted read-write. It is not advised to remove power or disconnect the USB Image Replicator drive until the whole process has completed.


The images files captured are saved to the root folder of the USB disk. Upon completion, it is safe to remove power or unplug the USB drive.
To help diagnose failures, files in <code>/tmp/logs/</code> contain output from each capture process.
 
For each media present on the unit (SD / eMMC / SATA / etc.), the image capture process will do the following:
 
# Copy the entire media image to an appropriately named file on the USB Image Replicator drive, e.g. <code>sdimage.dd</code>. No data is written to the source media and it is never mounted. The source disk can follow the stock partition layout, or implement a customized one.
# Perform an fsck on the Linux rootfs partition in the image file. Note that, if deviating from the standard partition layout, it may be necessary to modify the scripts handling the capture process.
# Mount the Linux rootfs partition from the image file and sanitize the filesystem. The sanitization process removes temporary files (e.g. <code>/log/</code> files), unique files (e.g. ssh private key files, machine ID files), adds a file to indicate that it is a custom image with the date as its contents, etc. The full list of operations can be found in [https://github.com/embeddedTS/buildroot-ts/blob/main/technologic/board/usbprod-common/scripts/sanitize_linux_rootfs.sh this script]. It may be necessary to modify this file for unique situations.
# If the media's partition layout uses only a single partition, the filesystem is packed in to a tarball on the USB Image Replicator drive which is appropriately named and compressed, e.g. <code>sdimage.tar.xz</code>. The image file is then unmounted and removed from the USB Image Replicator drive.
# If the media's partition layout uses multiple partitions, the image file is then unmounted, an md5sum of the image file taken, it is compressed and appropriately named on the USB Image Replicator drive, e.g. <code>emmcimage.dd.xz</code>, and then an md5sum of the compressed image is taken.
 
Note that when using this process, the USB Image Replicator drive that is used must be sized large enough to handle multiple compressed images as well as the uncompressed copy of the media image actively being worked with. If the image capture process runs out of space, the process will indicate a failure via the LEDs.
 
The images files captured are saved to the root folder of the USB Image Replicator drive. Upon completion, it is safe to remove power or unplug the USB drive.


For more details on the image capture process, see [https://github.com/embeddedTS/buildroot-ts/blob/main/technologic/board/usbprod-common/scripts/blast_funcs.sh this script].
For more details on the image capture process, see [https://github.com/embeddedTS/buildroot-ts/blob/main/technologic/board/usbprod-common/scripts/blast_funcs.sh this script].




==== Image Write ====
''' Image Write ''' <span id="Image_Write"></span>


This process is used to write existing images to media on a target unit. If appropriately named disk images or tarballs (see table below) are present in the root folder of the USB drive when booted, then the startup scripts will start the image writing process. The latest disk images we provide can be downloaded from our FTP site, see the [[#Backup_.2F_Restore|backup and restore]] section for links to these files.
This process is used to write existing images to media on a target unit. If appropriately named disk images or tarballs (see table below) are present in the root folder of the USB Image Replicator drive when booted, then the startup scripts will start the image writing process. The latest disk images we provide for our platforms can be downloaded from our FTP site, see the [[#Backup_.2F_Restore|backup and restore]] section for links to these files.


While in progress, the red LED will remain solid while the green LED flashes once per second. Upon completion, the red LED turns off and the green LED will slowly blink to indicate success,;hile a blinking red LED indicates a failure. Note that the USB drive remains read-only through the entire process, but target devices may be mounted or actively written. It is not advised to remove power or the USB drive until the whole process has completed.
Note that the USB Image Replicator drive remains read-only through the entire process but target devices may be mounted or actively written. It is not advised to remove power or disconnect the USB Image Replicator drive until the whole process has completed.


To help diagnose failures, files in <source inline>/tmp/logs/</source> contain output from each writing process.
To help diagnose failures, files in <code>/tmp/logs/</code> contain output from each writing process.


Note that the script expects images and tarballs to have specific names. When using an ext2/3/4 filesystem on the USB drive, symlinks can be used. The following table is the list of valid file names:
The Image Replicator script expects disk images or tarballs to have specific names to match the target media. The script will attempt to match tarball and then disk image names (in the order they are listed in the table below) for each target media, using the first file that is found to write to the target media. Note that symlinks can be used on the USB Image Replicator disk if formatted with a filesystem that supports symlinks. This can be used, for example, to write the same tarball to both SD and eMMC from only a single copy of the source tarball.
 
{| class=wikitable
! Target device
! Accepted filenames
! Description
|-
! rowspan=2 | SD Card
|
* sdimage.tar.xz
* sdimage.tar.bz2
* sdimage.tar.gz
* sdimage.tar
| Tar of the filesystem.  This will repartition the SD card to a single partition and extract this tarball to the filesystem.  If present, a file named <source inline>/md5sums.txt</source> in the tarball will have its contents checked against the whole filesystem after the tarball is extracted. This <source inline>md5sums.txt</source> file is optional and can be omitted, but it must not be blank if present. This file is present in our official images and is created during image capture with the Image Replicator tool.
|-
|
* sdimage.dd.xz
* sdimage.dd.bz2
* sdimage.dd.gz
* sdimage.dd
| Disk image of the card. This will be written to the SD card block device directly. If present on the USB drive, a file named <source inline>sdimage.dd.md5</source> will be used to verify the data written to the SD card against this checksum. This file is provided with our official images and is created during image capture with the Image Replicator tool.
|-
! rowspan=2 | eMMC
|
* emmcimage.tar.xz
* emmcimage.tar.bz2
* emmcimage.tar.gz
* emmcimage.tar
| Tar of the filesystem.  This will repartition the eMMC to a single partition and extract this tarball to the filesystem. If present, a file named <source inline>/md5sums.txt</source> in the tarball will have its contents checked against the whole filesystem after the tarball is extracted. This <source inline>md5sums.txt</source> file is optional and can be omitted, but it must not be blank if present. This file is present in our official images and is created during image capture with the Image Replicator tool.
|-
|
* emmcimage.dd.xz
* emmcimage.dd.bz2
* emmcimage.dd.gz
* emmcimage.dd
| Disk image of the card. This will be written to the eMMC block device directly. If present on the USB drive, a file named <source inline>emmcimage.dd.md5</source> will be used to verify the data written to the SD card against this checksum. This file is provided with our official images and is created during image capture with the Image Replicator tool.
|-
! rowspan=2 | SATA
|
* sataimage.tar.xz
* sataimage.tar.bz2
* sataimage.tar.gz
* sataimage.tar
| Tar of the filesystem.  This will repartition the first SATA drive with a single partition and extract this tarball to the filesystem. If present, a file named <source inline>/md5sums.txt</source> in the tarball will have its contents checked against the whole filesystem after the tarball is extracted. This <source inline>md5sums.txt</source> file is optional and can be omitted, but it must not be blank if present. This file is present in our official images and is created during image capture with the Image Replicator tool.
|-
|
* sataimage.dd.xz
* sataimage.dd.bz2
* sataimage.dd.gz
* sataimage.dd
| Disk image of the card. This will be written to the first SATA block device directly. If present on the USB drive, a file named <source inline>sataimage.dd.md5</source> will be used to verify the data written to the SD card against this checksum. This file is provided with our official images and is created during image capture with the Image Replicator tool.
|}


Upon completion, it is safe to remove power or unplug the USB drive.
Upon completion, it is safe to remove power or unplug the USB drive.


For more details on the image write process, see [https://github.com/embeddedTS/buildroot-ts/blob/main/technologic/board/usbprod-common/scripts/blast_funcs.sh this script].
For more details on the image write process, see [https://github.com/embeddedTS/buildroot-ts/blob/main/technologic/board/usbprod-common/scripts/blast_funcs.sh this script].
The following table is the list of valid file names and how they are processed:
{| class=wikitable style="width: 100%;"
! style="width: 8%" | Target media
! style="width: 12%" | Accepted filenames
! style="width: 80%" | Description
|}

Latest revision as of 14:30, 12 June 2023

Once a USB drive is formatted with the Image Replicator tool (see Creating a USB Image Replicator Disk for the correct files and process), boot to this USB drive (note that the Image Replicator already sets up the correct U-Boot boot scripts to boot to the USB drive, see the aforementioned section for details on how to make U-Boot call the scripts on the USB drive). This will start either image capture if no disk images/tarballs are present on the USB drive, or image write if there are disk images/tarballs present on the USB drive.

The Image Replicator tool, while in progress, will flash the green LED once per second while the red LED remains solidly lit. Upon completion, the red LED turns off and the green LED will slowly blink to indicate success, while a blinking red LED with the green LED off indicates a failure.

On each boot, startup scripts will check if the single partition of the USB drive can be expanded and do so if possible. If this process fails, then any further operations will not be run and the LEDs will blink to indicate a failure.


Image Capture

If no valid images to write exist on the booted USB Image Replicator drive, the image capture process starts automatically.

Note that while in progress, the USB Image Replicator drive is mounted read-write. It is not advised to remove power or disconnect the USB Image Replicator drive until the whole process has completed.

To help diagnose failures, files in /tmp/logs/ contain output from each capture process.

For each media present on the unit (SD / eMMC / SATA / etc.), the image capture process will do the following:

  1. Copy the entire media image to an appropriately named file on the USB Image Replicator drive, e.g. sdimage.dd. No data is written to the source media and it is never mounted. The source disk can follow the stock partition layout, or implement a customized one.
  2. Perform an fsck on the Linux rootfs partition in the image file. Note that, if deviating from the standard partition layout, it may be necessary to modify the scripts handling the capture process.
  3. Mount the Linux rootfs partition from the image file and sanitize the filesystem. The sanitization process removes temporary files (e.g. /log/ files), unique files (e.g. ssh private key files, machine ID files), adds a file to indicate that it is a custom image with the date as its contents, etc. The full list of operations can be found in this script. It may be necessary to modify this file for unique situations.
  4. If the media's partition layout uses only a single partition, the filesystem is packed in to a tarball on the USB Image Replicator drive which is appropriately named and compressed, e.g. sdimage.tar.xz. The image file is then unmounted and removed from the USB Image Replicator drive.
  5. If the media's partition layout uses multiple partitions, the image file is then unmounted, an md5sum of the image file taken, it is compressed and appropriately named on the USB Image Replicator drive, e.g. emmcimage.dd.xz, and then an md5sum of the compressed image is taken.

Note that when using this process, the USB Image Replicator drive that is used must be sized large enough to handle multiple compressed images as well as the uncompressed copy of the media image actively being worked with. If the image capture process runs out of space, the process will indicate a failure via the LEDs.

The images files captured are saved to the root folder of the USB Image Replicator drive. Upon completion, it is safe to remove power or unplug the USB drive.

For more details on the image capture process, see this script.


Image Write

This process is used to write existing images to media on a target unit. If appropriately named disk images or tarballs (see table below) are present in the root folder of the USB Image Replicator drive when booted, then the startup scripts will start the image writing process. The latest disk images we provide for our platforms can be downloaded from our FTP site, see the backup and restore section for links to these files.

Note that the USB Image Replicator drive remains read-only through the entire process but target devices may be mounted or actively written. It is not advised to remove power or disconnect the USB Image Replicator drive until the whole process has completed.

To help diagnose failures, files in /tmp/logs/ contain output from each writing process.

The Image Replicator script expects disk images or tarballs to have specific names to match the target media. The script will attempt to match tarball and then disk image names (in the order they are listed in the table below) for each target media, using the first file that is found to write to the target media. Note that symlinks can be used on the USB Image Replicator disk if formatted with a filesystem that supports symlinks. This can be used, for example, to write the same tarball to both SD and eMMC from only a single copy of the source tarball.

Upon completion, it is safe to remove power or unplug the USB drive.

For more details on the image write process, see this script.

The following table is the list of valid file names and how they are processed:

Target media Accepted filenames Description