Custom Software Deployment with EMAC OE

TODO: {{#todo:SEOKWREV (12.17.13-01:25->MD+);(12.19.13-11:30->MD+);(12.19.13-15:02->KY+);(12.14.13-12:55->MG+);(03.04.14-16:30->BS-);(03.27.14-15:40->BS+);(04.02.14-16:05->BS+)|Mike Dean|oe 4,oe 5,md,ky,mg,bs,SEOKWREV}}

There are many considerations to take into account when deploying software. This guide was written to provide guidance regarding the decisions relevant to deployment on EMAC OE systems. This is by no means a complete guide, but it covers some of the issues that are easily forgotten or overlooked. This material should be considered a supplement to a full deployment guide.

File Locations

Picking the right locations in which to store files is very important for the following reasons:

• Many parts of an embedded filesystem are read-only during normal use, and are therefore unsuitable for storing files which will be written to at runtime.
• Following the standard filesystem organization ensures that files will be easy to find. For example, putting a configuration file in /etc will ensure that it's stored where someone experienced with Linux will first look for it; however, if the configuration file were instead stored in /bin, that same person may give up without ever finding it (even if that person is the person who wrote the software, perhaps a few years previous).
• Placing write-heavy files onto flash will wear out the flash quickly. These types of files should always be put on a RAM disk or magnetic media.
• Lock files should go in a predictable location in case a stale lock needs to be manually removed at some point.
• Files for a webserver should be in a predictable location so they can be easily found.
• Initialization scripts are expected to reside in a specific location, both for organizational and for security reasons.

Ensuring that software correctly utilizes the filesystem helps ensure reliability (by not prematurely wearing out the flash), portability and maintainability. More information is available in Linux Filesystem Organization.

Filesystem Considerations

There are various important considerations when it comes to the filesystem for an embedded system. Here is a list of some of those considerations, along with recommendations or guidelines for the decisions which need to be made regarding filesystems.

Keeping the Root Filesystem Read-Only

Ensuring that the root filesystem stays read-only the entire time the machine is running is one of the best ways to ensure reliability of the machine. A machine with a writeable filesystem may lose power suddenly while writing to disk, which could cause corruption to the filesystem. A read-only filesystem ensures this type of corruption will not occur.

When software on a machine needs to write data somewhere, there are two good alternatives to writing to the root filesystem:

• Write to a RAM disk.
• Write to a dedicated data storage device.

Data can be written to a RAM disk if it doesn't need to persist across boots. This is perfect for temporary information, such as a device's IP address as assigned by a DHCP server. Writing data to RAM provides the additional benefit of preventing wear of the flash.

When data needs to persist across boots, it can be written to a dedicated storage device. This storage device can be a separate partition on the main flash device, a partition on a removable flash device, a network filesystem, a remote database, a remote FTP server, or something else. Two such storage devices are described below.

Choosing a Filesystem for a Data Partition

There are many filesystems available for Linux. The correct filesystem to use depends upon the application. Here are some guidelines:

1. Determine whether the filesystem needs to be read/write, or if it can be read-only.

   • If read-only, any filesystem can be used so long as it meets the requirements of point 3.

   • If read-write, all other points in this list need to be considered.

2. Determine the type of physical storage which will be used.

   • If flash, then a flash friendly filesystem will be needed. A flash friendly filesystem will provide wear leveling, and possibly other features pertaining to flash storage devices. An example is JFFS2.

   • If a hard disk will be used, a traditional filesystem will be a better fit. An example is ext2.

3. Determine whether or not Linux filesystem permissions need to be preserved.

   • If so, avoid FAT and NTFS filesystems.

   • If not, any filesystem meeting the other requirements can be chosen.

4. Determine whether or not the machine needs to be capable of dealing with sudden power loss (as opposed to a battery backup which initiates auto-shutdown).

   • If so, a journalled filesystem will be required.

   • If not, a filesystem without journalling is an option which might provide higher performance (but not always).

Remember, reliability is usually more important in embedded systems than speed. The filesystem should usually be chosen for reliability first, then speed.

Using Removable Flash for Data Storage

Removable flash devices, such as SD cards and USB flash drives, can be used to store data. Using such a device provides the following benefits:

• Storage device can easily be swapped out when full.
• Data can be quickly and easily retrieved, even if no network connection to the board is available or the board is on a low bandwidth connection.
• Wear occurs on a removable and easily replaceable flash rather than on an on-board chip.
• Storage space can be easily upgraded as data demands grow.

For information on mounting a flash device temporarily, see Mounting a Flash Filesystem. When satisfied with a method of mounting the flash, an entry in the /etc/fstab can be created to ensure that the flash is mounted automatically at boot. Alternatively, udev can be configured to always mount the flash device at a specific location; this method allows hot-plugging the flash devices.

Preparing a Filesystem for a Snapshot

Prior to taking a snapshot of a filesystem, it must be prepared for deployment. This preparation process involves cleaning up any temporary files and settings, configuring the desired default settings for the machines produced, and removing any machine-specific information (such as SSH keys) from the filesystem. The Deployment Checklist section below provides a checklist of items to address prior to taking a snapshot of a filesystem.

Manufacturing Boards with a Filesystem Image

Once a custom distribution has been created for you, that distribution can be given a unique part number and kept as a disk image on our server. Subsequent purchases of that part number are guaranteed to contain software identical to the original and the image can be installed on subsequent orders at no charge. As a value-added service, we can even incorporate your application and utilities and combine them on the disk image, saving you time and effort. If you require an updated build, you would only pay once for the normal build fees and the image could then replace the previous version on our server. Additionally, EMAC can provide integration services for custom products. Please contact EMAC for information on obtaining these services.

Other Considerations

Launching Software Automatically on Boot

When software is created for an embedded system, it is often required for the software to start when the machine boots. The best approach to meeting this requirement is to use the robust initialization system which is part of the Linux filesystem. Configuring the initialization system to launch an application is performed by creating a simple script, placing it in the correct spot in the filesystem, then creating appropriate symbolic links to the script.

For more information on this process, see EMAC OE Boot Process Customization.

Automatically Sending Code to a Machine

During the development process, development versions of software will typically be sent to a machine many times. An IDE, when configured correctly, will perform this task automatically as part of the build process. An IDE may also be able to send additional files to the machine, such as configuration and data files, automatically as part of a customized build process.

When developing software without an IDE, the process of sending software to a machine can still be automated. The sshpass program can be used to automate entry of a password to ssh and related tools, such as scp. For instance, the following command would send the file, myprogram, to the board located at 10.0.1.20 and store it in the /usr/local/bin directory on that machine:

user@ldc:~$ sshpass -p 'emac_inc' scp myprogram root@10.0.1.20:/usr/local/bin/

The following command would then execute myprogram:

user@ldc:~$ sshpass -p 'emac_inc' ssh -l root 10.0.1.20 "/usr/local/bin/myprogram"

Here, the password is passed to sshpass as a commandline argument. Since this means it's possible for the password to show up in the output of a command like ps, there are other, more secure options as well. The sshpass command can also take the password from any of the following sources: a file, an open file descriptor, or a shell environment variable. Please see the manual for sshpass for more information.

Deployment Checklist

Here is a checklist of issues which need to be addressed prior to deployment:

1. Does the board boot into the desired state, reliably and repeatedly?

2. Have the ssh keys been removed? Look in /etc/ssh.

3. Can the image of the filesystem be put on a different machine (with a different MAC address) than the development machine, and boot into the desired state reliably and repeatedly?

4. Is the network configuration of the machine set to the desired default for production machines?

5. Are any errors or warnings appearing in the log files on the machine while it's running? If so, can they be stopped?

6. Does the root filesystem mount read-only after the machine is booted?

7. Has a method to test the boards coming off the production line been determined and accounted for?

8. Are all the passwords set to their desired defaults?

9. Have all of the temporary files from the development process been removed, such as .tar.gz files and saved work-in-progress configuration files?

10. Is the timezone configured to the desired default timezone?

11. If an sshd server is going to run on the machine, is it configured with adequate security? Have all preshared SSH keys on the machine been removed?

12. If a static IP is configured for the default, does the /etc/resolv.conf file specify appropriate defaults for the DNS servers and the search domain?

Conclusion

There are many considerations to take into account when preparing to deploy an embedded system. Following the recommendations and checklist above will help prevent a developer from forgetting details which need to be addressed prior to final deployment.