My OpenWrt observations
OpenWrt is a highly extensible Linux distribution for over 2,000 commercial-off-the-shelf devices, typically wireless routers.
Although incredibly popular and versatile, it's not without its challenges, especially for newcomers. Being one of those newbies, I'll share the observations I've collected along my journey in case they help others.
Observation #1: there's no installer
Almost every open source project has a quick start guide with a short, step-by-step tutorial or a one-liner script to install it. Not OpenWrt. Their quick start guide for OpenWrt installation is actually just a wiki introduction page that links to a factory installation page that tells you to "Follow the device-specific instructions of your manufacturer's user guide" and then links to more wiki pages, which also link to more wiki pages...
..which leads me to my next observation:
Observation #2: a vast ocean of wiki and forum pages
Documentation is great and a rare commodity in the tech world, but too much of a good thing can also be overwhelming. In short, OpenWrt is meant for technical people so don't expect a whole lot of hand holding. That said, the forum community is generally polite and will help you with a few questions, but beyond that you're pretty much on your own.
It also doesn't help that a while back the community split into two camps: OpenWrt and LEDE. They eventually made amends and merged back into one platform, but it is confusing to still find references to both projects. In addition, the popular device maker FriendlyElec created their own fork called FriendlyWrt and a China-based group created another fork called ImmortalWrt. Yeah, confusing.
Observation #3: releases vs snapshots
Let's say you just bought a shiny new NanoPi R5C. It supports a number of Linux operating systems, including Debian, Ubuntu, and FriendlyWrt. In this example, we'll assume you've installed FriendlyWrt to the device's eMMC drive and now you want to replace it with the official OpenWrt version.
The official OpenWrt instructions will tell you to find your device in the Table of Hardware: Firmware downloads page. Searching for "R5C" will provide you a link to the "Factory sysupgrade image" to flash (install) on the device:
Okay, so I've downloaded the "Factory sysupgrade image" file. Now what?
Heh, read more wiki pages ;)
You probably found their Sysupgrade wiki page like I did and followed their instructions to overwrite FriendlyWrt. Guess what, it doesn't work (or it least it didn't for me). You're left with an unresponsive device and you have to start the process all over again to get FriendlyWrt back on it (which is doubly annoying since the R5C will no longer boot from the SD card unless you insert a thin object into the MASK hole to press the button for 5 seconds upon boot). Ugh. I spent a ridiculous amount of time just trying to figure out how to install OpenWrt's R5C snapshot sysupgrade image. It turns out sysupgrade is the proper method IF you're already running OpenWrt. If you're running FriendlyWrt THEN you need to login to the web GUI at http://192.168.2.1/ and choose their custom image solution via System > eMMC Tools (NOT System > Backup / Flash Firmware):
You'll be notified when the installation is complete and instructed to unplug then plug in the device.
After a couple minutes, the device will be happily blinking lights and looking very healthy and active. However, http://192.168.2.1/ doesn't load. Some wiki reading later, you discover FriendlyWrt uses that address but OpenWrt uses http://192.168.1.1/ Bah.
So you try that address instead. No dice. Is the installation busted? Did I brick my device? Nope, it's just that OpenWrt tries to be so lean that it doesn't include the web UI by default. Umm.. thanks?
You need to login to the device via SSH by connecting your computer to the device's LAN Ethernet port and run this command at a terminal prompt: ssh root@192.168.1.1
You're in! Finally!
You'll want to set a root password by running the passwd command and then run this command to install the web UI: opkg update && opkg install luci
Now you can browse to http://192.168.1.1/
Observation #4 uci, luci, opkg, oh my!
Most computer systems today, such as Windows, use the "x86" architecture. It established a uniform set of rules for how a system should boot and interact with hardware. That structure and standardization provided consistency and ease for firmware developers (who write device drivers).
Fast forward a bit and a new kid on the block showed up: ARM. Although hardware for ARM devices was less powerful than their x86 siblings, it was significantly more efficient and less expensive. Unfortunately, its humble niche beginnings also meant very little standardization was introduced or enforced so most hardware vendors simply developed their own solutions. This untamed 'wild west' meant you were typically stuck with whatever operating systems and drivers the manufacturer provided, often without ongoing support, system upgrades, or security patches. Old devices basically were relegated to the scrap heap.
OpenWrt tried to solve that by introducing ARM standards called UCI (Unified Configuration Interface). That standardization empowered the open source community to develop firmware solutions for old and new commercial devices. It also allows device manufacturers to easily include OpenWrt as a supported operating system.
LuCI is completely unrelated. LOL, confused yet? LuCI is the default web interface for OpenWrt and supports multiple themes. There are other web engines, but LuCI is the most common.
Lastly, opkg is OpenWrt's package manager which you use to install software. If you've used apt on Ubuntu, yum on AlmaLinux, or apk on Alpine Linux, you're familiar with the concept.
Observation #5: squashfs vs ext4
After a router reboot, let's say you want to install the text editor Nano. You run opkg install nano and receive an unexpected error:
Unknown package 'nano'.
Collected errors:
* opkg_install_cmd: Cannot install package nano.
Hmm, that's odd. Is Nano not supported on OpenWrt? A quick package check shows it is supported so why the error? On a whim you decide to run opkg update and then try opkg install nano. This time it worked!
So what happened? I had already performed opkg update previously and I saw the results were stored in /var/opkg-lists/ so why did I have to do it again? The answer is immutable squashfs. Let me explain what that means.
Take a look at the output of this command after I reboot the R5C:
# ls -al /var
lrwxrwxrwx 1 root root 3 Apr 4 16:31 /var -> tmp
So everything you put in /var is actually being stored in /tmp, which gets cleared on every reboot. Now we know why after every reboot we have to run opkg update again to repopulate the local package database, but why would they set it up like that? Normally Linux stores /var on the permanent root file system that doesn't get cleared out with every reboot but this command tells us something interesting about OpenWrt's root file system:
# grep "[[:space:]]ro[[:space:],]" /proc/mounts
/dev/root /rom squashfs ro,relatime,errors=continue 0 0
It turns out OpenWrt has set the root file system to read-only (ro). That type of setup is called immutable, meaning you can't change it. This confuses some users, so why would they do that? There are many reasons, but I'll highlight two: system stability and resource constraints.
A filesystem that can't change usually means a filesystem that can't break. For Internet-of-Things (IOT) and remote edge devices, that stability is critical. Immutable systems typically replace the entire operating system when there are updates rather than updating packages, files, and other elements within the operating system.
In addition, most hardware for routers and other embedded systems is fairly minimal. Having a limited, dedicated job to perform means you can optimize the workload and get away with less CPU, RAM (memory), and disk space. For this reason OpenWrt is an incredibly small Linux distribution compared to other distros like Red Hat Enterprise Linux or Ubuntu. Storing a compressed operating system is one thing, but extracting and running it can pose a challenge if the disk space is limited. So, OpenWrt defaults to using a special filesystem called SquashFS.
SquashFS is a compressed read-only file system for Linux. OpenWrt uses it for the root file system. But if the entire system were read-only, it would be very limiting (no installation of addition packages, no logging, no storage of user configuration settings, etc.) so OpenWrt layers a read-write file system over the read-only root files system, called tmpfs that only exists in RAM (memory). OpenWrt then uses OverlayFS to "merge" those layers together in a way that to the end-user it appears to be a single coherent filesystem.
A good analogy is a stack of painted glass panes for an old-style Disney movie. If you look down on a stack from above it looks like a complete scene, whereas it's really a composite of multiple layers:
In the image above, the sun and clouds background would be the read-only SquashFS root file system and the panes above it would be read-write tmpfs filesystems stored in memory.
With that technical background, take a look at the output of this command:
# df -Th
Filesystem Type Size Used Available Use% Mount
/dev/root squashfs 4.0M 4.0M 0 100% /rom
tmpfs tmpfs 1.8G 1.3M 1.8G 0% /tmp
/dev/loop0 f2fs 98.1M 42.7M 55.4M 44% /overlay
overlayfs:/overlay overlay 98.1M 42.7M 55.4M 44% /
tmpfs tmpfs 512.0K 0 512.0K 0% /dev
The yellow read-only root filesystem is being "merged" with the blue temporary read-write filesystems to create a combined green overlay filesystem.
Since tmpfs only exists in memory, their contents vanish upon shutdown / reboot. This is why the opkg update command that stores results in /var (which actually points to /tmp) disappeared after a reboot and we needed to run the command again.
So what read-write part of OpenWrt is permanent (i.e. can survive a reboot)? The f2fs filesystem at /overlay (see the /dev/loop0 line above). Should I put my personal stuff there? You can, but it will get wiped on the next OpenWrt system upgrade (sysupgrade).
In summary, the root filesystem is read-only, almost all other areas are read-write but get wiped upon reboot, and /overlay is permanent until an operating system upgrade. If you want a permanent storage solution that survives sysupgrade, see Observation #9 below.
What if I don't want a read-only root filesystem? It turns out OpenWrt offers an alternate read-write filesystem called EXT4. However, you'll notice there was no mention of EXT4 in any of the pages or links above. For some reason, OpenWrt chooses to bury that option. You'll need to go to https://firmware-selector.openwrt.org/ , change the dropdown to SNAPSHOT (if your device, like our R5C, isn't yet officially supported), then type the model number of your device, select it, and click the download button for SYSUPGRADE (EXT4):
Note: if you plan to use virtualization on your OpenWrt device (Docker, LXC, etc.) you must use EXT4.
Observation #6: filesystem size restrictions
A few observant readers may have noticed that although my R5C has 4GB of RAM and 32GB of eMMC disk space, my permanent writeable disk space is only ~100MB and total disk space is only ~2GB. Why is that?
Regarding the permanent writeable disk space, since it's only designed to hold configuration files, SSH keys, passwd/shadow files, LuCI web files, and other small entries, OpenWrt purposely keeps it small. To increase it you'll need to build your own OpenWrt image with the ROOTFS_PARTSIZE environment variable. See Observation #8 below for how to do that.
Regarding the total disk space, since OpenWrt upgrades need to extract the entire operating system into memory to replace the existing running OpenWrt installation, it limits the total disk space to half your device memory. Why half? Since half the RAM is used as a pseudo-filesystem, the other half is needed for regular RAM functionality by the operating system.
Observation #7: no automatic updates
Regular operating system upgrades and software updates keep your system running smoothly and securely. All popular Linux distributions have either an automated or manual process for that. Similarly, OpenWrt provides sysupgrade for operating system upgrades and opkg upgrade for software updates.
However, OpenWrt strongly discourages upgrading your packages and offers no official method to do so. I'm sure this is due to many years of end users trying to upgrade their system, running out of disk space on their resource-poor hardware, bricking their device and blaming OpenWrt. That said, in my humble opinion simply saying 'don't upgrade' is a huge disservice to the community and a pretty big security risk.
Instead of upgrading packages as needed, OpenWrt recommends you upgrade the entire operating system each time you need to apply any update. This approach is called atomic. Although required for technical reasons, it often confuses new users.
Another word of caution: if you follow their official process for running sysupgrade, all your installed packages, configuration, and personal settings will be wiped out. Instead, use a new and improved upgrade method called Attended SysUpgrade (ASU).
Observation #8: custom build process poorly documented
While Attended SysUpgrade is pretty awesome, it does have some risk so another alternative is to build your own custom image. This is one of the coolest and most powerful features of OpenWrt but their official documentation comes across as complicated and overwhelming. There are two ways to do it. One is extremely easy and the other requires a bit more effort but is more powerful. I'll show you how to do both:
** Option #1 **
1. Go to https://firmware-selector.openwrt.org/
2. Use the steps covered above to find your device
3. Instead of immediately downloading the EXT4 or squashfs files provided, click the line that says "Customize installed packages and/or first boot script"
4. A textarea will appear and allow you to identify your own list of desired packages.
5. Once you've entered your list of packages, click the Request Build button
6. The page will automatically submit your request and build you a custom image according to your specifications. Isn't that awesome?!
7. Click the EXT4 or squashfs button in the "Custom Downloads" section to download your custom image.
Note: This option works great for most scenarios but if you need to modify your root file system size to support larger packages (like golang on the R5C, for example), you'll need to use the second option:
** Option #2 **
1. On an x86_64 Linux computer, install some prerequisites.
2. Browse to https://downloads.openwrt.org/ and click on the link for the latest OpenWrt stable release or, if your device isn't officially supported yet, the "Snapshot download" link.
3. Then click the link for your router's CPU manufacturer. This is typically listed on the device website or you can do a Google search. In the case of the NanoPi R5C, the CPU manufacturer is Rockchip so I clicked on "rockchip" then "armv8": https://downloads.openwrt.org/snapshots/targets/rockchip/armv8/
4. Scroll down to the "Supplementary Files" section and click the link that includes the word "imagebuilder". In my case this was openwrt-imagebuilder-rockchip-armv8.Linux-x86_64.tar.zst
5. Extract the contents on your x86_64 computer and open a terminal in that directory
6. We'll want to set four environment variables: PROFILE, PACKAGES, ROOTFS_PARTSIZE, and FILES
7. In the terminal run make info | grep r5c | head -n1 | sed 's/://' (replacing "r5c" with the model of your device). This will give you your PROFILE value. In my case it is friendlyarm_nanopi-r5c
8. In the terminal run PROFILE="friendlyarm_nanopi-r5c" (replacing the value with your device's profile)
9. PACKAGES will be up to you. In this example, let's assume I want the following packages (in addition to the default packages OpenWrt comes with):
PACKAGES="golang kmod-ath11k kmod-ath11k-pci ath11k-firmware-wcn6855 kmod-tun nano wget-ssl uvol autopart"
10. Finally, since the golang package is relatively large and the default OpenWrt root file system total is only 100MB, we would normally get this error if we tried to install it:Observation #9: no permanent storage area
As noted above, very little survives a sysupgrade and Attended SysUpgrade has some risk, so wouldn't it be nice to have a directory that reliably survives upgrades of any type where you can store things permanently without worrying it will be overwritten or lost? Fortunately, that option does exist but, once again, it isn't documented very well. Lots of forum posts mention fdisk, cfdisk, gfdisk, gparted, and other complicated methods. By far the easiest solution I found is uvol+autopart:
opkg update
opkg install uvol autopart
uvol create userdata $(uvol free) rw
uvol up userdata
Now, whatever you store in /tmp/run/uvol/userdata/ will be permanent. I know that seems a bit counter-intuitive since the path starts with /tmp but we can verify it works by running: echo test > /tmp/run/uvol/userdata/test.txt && reboot
After reboot, login and run: cat /tmp/run/uvol/userdata/test.txt
The file and contents will still be there!
After a sysupgrade, if uvol and autopart packages were included in the image, the userdata directory will automatically show up. Otherwise, it may initially look like our permanent directory is gone, but we just need to tell OpenWrt where to find it:
opkg update
opkg install uvol autopart
uvol up userdata
test -d /tmp/run/uvol/userdata || sed -i 's/^exit 0$//' /etc/rc.local && echo 'mkdir -p /tmp/run/uvol/userdata && mount /dev/$(uvol device userdata) /tmp/run/uvol/userdata' >> /etc/rc.local && /bin/sh /etc/rc.local
Observation #10: opkg issues without wget-ssl
OpenWrt would work fine for me for a couple days and then suddenly I would start getting "Failed to download the package list" errors. It turns out I needed the wget-ssl package but opkg couldn't get it because it wasn't installed -- a classic Catch 22 situation. Why did it work for a couple days and then stop working? No idea. Lesson learned: always install wget-ssl.
Observation #11: cron @reboot doesn't work
If you try to run a process on boot via cron, it won't work because @reboot is not possible with default busybox settings. You can change that with a custom build and the BUSYBOX_DEFAULT_FEATURE_CROND_SPECIAL_TIMES flag or, instead, you can add your command to the /etc/rc.local file. Note: most forum posts will tell you to make that file executable and add #!/bin/sh to the top of the file since that's what most Linux operating systems require but none of that is necessary with OpenWrt ..just be sure you add a 30-second sleep delay if your process relies on the network
Comments
Post a Comment
Keep it clean and professional...