The best embedded Linux tool you've never heard of: Rugix

 

So you have a Linux-based application that you want to run on both cloud and bare metal devices with non-bricking Over The Air (OTA) update capability. You probably thought of a few popular options like: Mender, SWUpdate, fwup, fwupd, balenaCloud, Torizon, JFrog Artifact Manager, Memfault, OSTree, Vib, ABRoot, Airo, boot2container, Sidero Omni, Heroku, Netlify, Vercel, Coolify, gokrazy, BareMetal OS, Yocto, Buildroot, ACRN, RAUC, or even tried rolling your own solution, but those options are either complicated, expensive, don't support OTA updates, or don't support both cloud and bare metal scenarios. You may have also considered automatically running a cron job to update a regular Linux OS using its package manager, but that's a bad idea. So what's left? A little-known tool that rocks! Rugix

Initial bare-metal image setup:

1. Install Docker on a build/control computer

2. Create a directory to store your custom build: mkdir ~/example && cd ~/example

3. Install the Rugix tools: curl -sfSO https://raw.githubusercontent.com/silitics/rugix/v0.8/bakery/run-bakery && chmod +x ./run-bakery

4. Optional: if you're going to cross-compile for a target architecture that's not the build/control computer's architecture, follow these instructions.

5. Optional: if you need to add boot arguments export an environment variable: export rugpi_bootargs="pci=nommconf"

then edit layers/customized.toml and in the recipes array add this entry:

"core/alpine-grub-setup",

6. Download and initialize the base operating system. Rugix supports Debian, Alpine Linux, and Raspberry Pi OS. In this example, I'll use Alpine Linux: sudo ./run-bakery init alpine-grub-efi

7. Edit layers/customized.toml and in the recipes array add this entry:

"core/ssh",

Then add this section at the bottom with your public SSH key so you can later login to the new embedded OS:

[parameters."core/ssh"]

root_authorized_keys = """

<INSERT YOUR PUBLIC SSH KEY HERE>

"""

8. You would also normally add your application but for this demo I'm going to keep the default Hello World web page app.

9. Now build the OS: sudo ./run-bakery bake image customized-amd64

10. The image will now be ready to load onto a USB key: build/customized-amd64/system.img

Note: the image is large (~2.4 GB) because it includes redundant boot partitions and a config partition. Updates to the image (as shown in the next section) will be smaller.

11. Use balenaEtcher to load the image on to a USB thumb drive

12. Insert the thumb drive into your appliance hardware and boot (making any necessary BIOS or keystrokes to choose the USB drive to boot into). You should see an Alpine Linux login and you should also be able to login via SSH with the key you set in step 6. Lastly, you should be able to load the IP address into your browser and see a simple demo web page:

Update bare-metal image:

So far we've booted Alpine Linux and run a web page. Not that exciting. However, imagine you have a fleet of hundreds or thousands of remote machines running a complex appliance and a critical security vulnerability was just announced or you need to update your appliance to version 2. How do you do that securely and safely without risk of bricking the remote device? Rugix to the rescue!

1. On your original build machine, go to your original ~/example directory

2. Let's create a simple change for this example, but in reality it could be any number of changes you want to make: sudo sed -i 's/Hello/Hi/g' recipes/hello-world/html/index.html

3. Now, instead of creating a whole new image, we're going to create a smaller delta binary ("bundle") that we can push to the remote device(s): sudo ./run-bakery bake bundle customized-amd64

Important: that command will return a hash that looks something like the example below. Copy this to a secure location because you'll need it later to allow the upgrade to occur on the remote device.

sha512-256:ec3bb055bd5df8b7bcb31950d184da2cdcf02c41984c21d6e2c716df26363c02

4. The bundle will be created in build/customized-amd64/system.rugixb

Note: the bundle size (~700 MB) is obviously larger than the change we made because the bundle needs to contain all the partitions and upgrade config since Rugix implements the same A/B upgrade strategy as Android, Chrome OS, and many other software solutions.

5. After building the bundle, you can transfer it via SCP to your remote device(s): scp build/customized-amd64/system.rugixb root@<device IP goes here>:/root

6. Then you SSH into the remote device and install the bundle with the following command: rugix-ctrl update install --verify-bundle <hash from step 3> /root/system.rugixb

7. Rugix will automatically reboot the system after the bundle has been installed. If everything is working as expected (web page at IP address now says "Hi World!") you would run this command to make the update permanent: rugix-ctrl system commit

Otherwise, a simple reboot returns you to the prior working state. Nice!

Adaptive delta updates:

Although the bundle upgrade process above works, it's still not ideal. Transferring a 700 MB file to remote device requires a lot of bandwidth. Wouldn't it be nicer to just transfer the changes within the OS and not the entire OS bundle? Well, it turns out Rugix supports that as well!

1. You'll need a web server to host the system.rugixb file. For this quick demo I'll use the simple Spark web server:

sudo wget https://github.com/rif/spark/releases/download/v1.8.1/spark_1.8.1_linux_amd64.tar.gz

sudo tar -xvf spark_1.8.1_linux_amd64.tar.gz && sudo rm spark_1.8.1_linux_amd64.tar.gz

sudo ./spark -port 80 build/customized-amd64/system.rugixb

2. Then, SSH into the remote device and run the following command:

cat > install-update.sh <<EOF

#!/usr/bin/env bash

set -euo pipefail

URL=\$1

ACTIVE_SYSTEM=\$(rugix-ctrl system info | jq -r ".boot.activeGroup")

if [ "\$ACTIVE_SYSTEM" == "a" ]; then

    rugix-ctrl slots create-index boot-a casync-64 sha512-256

    rugix-ctrl slots create-index system-a casync-64 sha512-256

else

    rugix-ctrl slots create-index boot-b casync-64 sha512-256

    rugix-ctrl slots create-index system-b casync-64 sha512-256

fi

rugix-ctrl update install "\$URL"

EOF

apk update && apk add -f jq

bash ./install-update.sh http://<YOUR WEB SERVER IP HERE>/system.rugixb

Alternatives:

Rugix is nice but does have some downsides. For example, the images and bundles are large in filesize and you can't embed the hash in the bundle itself. If you plan on managing remote systems via SSH you might as well use something like Terraform, Ansible, Chef, Puppet, or Salt along with a system diff tool, like OSTree.