Production-Grade Nebula

Nebula is a super-cool mesh VPN originally started at Slack. The creators later spun it off in their own company called Defined Networking. There are a number of useful articles online [1][2][3][4][5][6] so I won't go into detail, but here are some tips for making your installation production-grade:

• The best cloud deal for your lighthouse nodes is Oracle Cloud Free Tier (I use the Always-Free VM.Standard.E2.1.Micro Canonical-Ubuntu-20.04-Minimal compute instance type) -- make sure to remember to open the Nebula port in the firewall:

sudo apt install -y ed

printf "%s\n" '/dport 22/a' '-A INPUT -p udp -m state --state NEW -m udp --dport 4242 -j ACCEPT' . wq | sudo ed -s /etc/iptables/rules.v4

sudo iptables-restore < /etc/iptables/rules.v4

• Use a Reserved Public IP address to pair with your lighthouse instance since the instance public IP is ephemeral
1. Go to "Menu > Networking > Reserved IPs" and click "Reserve Public IP Address" and give it a name (e.g. lighthouse1)
2. Note the new public IP address for later configuration in Nebula
3. Go to "Menu > Compute > Instances" and select your lighthouse instance
4. Scroll to the bottom and in the left-hand Resources menu select "Attached VNICs"
5. Click the name of the VNIC associated with the compute instance
6. Scroll to the bottom and in the left-hand Resources menu select "IPv4 Addresses"
7. For the "Primary IP" row, click on the three dots to the far right and choose "Edit"
8. Select the radio button for "No Public IP" and click "Update"
9. For the "Primary IP" row, click on the three dots to the far right and choose "Edit"
10. Click the radio button for "Reserved Public IP" and in the dropdown that appears select the floating IP address you created above
11. Click "Update"
12. SSH into the instance to verify the new public IP address works

• To be truly production-grade, you'll want to upgrade to a paid account and create a lighthouse instance in a few different availability zones for disaster resistance and reduced latency if your network is geographically dispersed.

• disable tun to run Nebula as non-root:

wget https://raw.githubusercontent.com/slackhq/nebula/master/examples/config.yml

printf "%s\n" '/tun:/++c' '  disabled: true' . wq | ed -s config.yml

• You'll need to ensure the ubuntu user can access the binaries and certs and config when running non-root

mkdir /home/ubuntu/nebula

sed -i 's/ca: \/etc\/nebula\/ca.crt/ca: \/home\/ubuntu\/nebula\/ca.crt/' config.yml

sed -i 's/cert: \/etc\/nebula\/host.crt/cert: \/home\/ubuntu\/nebula\/lighthouse.crt/' config.yml

sed -i 's/key: \/etc\/nebula\/host.key/key: \/home\/ubuntu\/nebula\/lighthouse.key/' config.yml

• List all the lighthouses in the static_host_map section

• Enable some performance tweaks:

sed -i 's/10485760/20971520/g' config.yml

sed -i 's/tx_queue: 500/tx_queue: 5000/' config.yml

sed -i 's/mtu: 1300/mtu: 1310/' config.yml

echo 'net.core.rmem_default=20000000' | sudo tee -a /etc/sysctl.conf

echo 'net.core.wmem_default=20000000' | sudo tee -a /etc/sysctl.conf

sudo sysctl -p

• Use IP ranges and/or Ports to allow multiple tenants to use the lighthouses

• (optional) Switch the log format to JSON:

sed -i 's/level: info/level: warning/' config.yml

sed -i 's/ format: text/ format: json/' config.yml

• Remove sample firewall rules:

sed -i '/# Allow tcp\/443/,/^$/d' config.yml

sed -i 's/Allow icmp/Allow any inbound traffic/' config.yml

sed -i 's/proto: icmp/proto: any/' config.yml

• Set your groups (Nebula-Cert-Maker suggests Lighthouse, Server, and Workstation)

• Start Nebula as a user-level service:

mkdir -p /home/ubuntu/.local/share/systemd/user/

cat > /home/ubuntu/.local/share/systemd/user/nebula.service <<EOF

[Unit]

Description=Nebula Service

[Service]

Restart=always

RestartSec=1

ExecStart=nebula -config /home/ubuntu/nebula/config.yml

[Install]

WantedBy=default.target

EOF

systemctl --user start nebula.service

systemctl --user status nebula.service

systemctl --user enable nebula.service

sudo loginctl enable-linger ubuntu