Network Booting a Raspberry Pi 4 from a CentOS 7 Machine
First of all let, me state that the following instructions are specific for the Raspberry Pi 4 and a machine running CentOS 7. The instructions are based on two articles by Jonathan Bennett published on Hackaday: Hack My House: Running Raspberry Pi Without An SD Card and Network Booting The Pi 4. In addition to what Jonathan wrote in the articles, I also found several comments below them very helpful.
The CentOS specific parts of the instructions deal with its firewall and the required SELinux settings. These might not be needed for other distros, but are absolutely essential for CentOS.
Step 0: Prerequisites
I will use this setup in a long-term experiment where a CentOS machine is controlling all the test-gear, including the Raspberry Pis. To separate the test-gear from the general network this machine is equipped with two network interfaces, where one (namely enp0s3) is used exclusively to talk to an internal network connecting to the test-gear. This network uses the 10.0.0.0/24 IP address range, with the CentOS machine being at 10.0.0.1. It will be added to the zone internal in the firewall. The figure below illustrates this setup.
The network setup this guide is assuming. Note that the CentOS machine has two network interfaces.
Step 1: Bootloader Settings
The Raspberry Pi 4 has an on-board I2C boot EEPROM to store the boodloader and its settings. At the time of writing this, all Pis get shipped with a bootloader missing the PXE boot support. So the first step is to download, configure and install a beta version of the bootloader. The following commands are taken literally from Jonathan's article, the only change being the use of a more recent beta version. To run them on the Raspberry Pi you need a recent version of Raspbian or Raspbian Lite installed to an SD-card.
sudo apt-get update
sudo apt-get upgrade
wget https://github.com/raspberrypi/rpi-eeprom/raw/master/firmware/stable/pieeprom-2020-01-17.bin
rpi-eeprom-config pieeprom-2020-01-17.bin.bin > bootconf.txt
sed -i s/0x1/0x21/g bootconf.txt
rpi-eeprom-config --out pieeprom-2020-01-17.bin-netboot.bin --config bootconf.txt pieeprom-2020-01-17.bin.bin
sudo rpi-eeprom-update -d -f ./pieeprom-2020-01-17.bin-netboot.bin
cat /proc/cpuinfo
ip addr
The last two commands returns general information about the Pi, including its serial number and MAC address. These will be needed later and you should write them down.
If you plan on network booting multiple Pis you can now use this SD-card and just need to run the last two lines (flash the reconfigured bootloader and retrieve information).
Step 2: Preparing the NFS, DHCP and TFTP Servers
On the CentOS machine start by installing the packages for the NFS and DHCP server, namely nfs-utils and dnsmasq. You will also need a directory structure for both services. In accordance with the Filesystem Hierarchy Standard put these in the /srv/tftp``and ``/srv/nfs directories.
sudo yum -y install nfs-utils dnsmasq
sudo mkdir -p /srv/nfs/rpi4-3a205fc0
sudo mkdir -p /srv/tftp/3a205fc0
sudo chmod 777 /srv/tftp
Step 3: Preparing the Boot Image
Next you will need to prepare image the Raspberry Pi will boot. Start by downloading the latest version of Raspbian from the Raspberry Pi homepage. I use Raspbian Lite as the machine will run headless. Using the awesome kpartx tool you can mount the root and boot partitions contained in the image and copy their contents to the appropriate directories in the NFS server root. I named the directory based on the serial number of my Raspberry Pi.
mkdir bootmnt
mkdir rootmnt
wget -O raspbian_lite_latest.zip https://downloads.raspberrypi.org/raspbian_lite_latest
unzip raspbian_lite_latest.zip
sudo kpartx -a -v *.img # this will produce unpredictable behavior if there are multiple .img files!
sudo mount /dev/mapper/loop0p1 bootmnt/
sudo mount /dev/mapper/loop0p2 rootmnt/
sudo cp -a rootmnt/* /srv/nfs/rpi4-3a205fc0/
sudo cp -a bootmnt/* /srv/nfs/rpi4-3a205fc0/boot/
For PXE-boot support, two files in the boot directory need to be updated as well.
cd /srv/nfs/rpi4-3a205fc0/boot/
sudo rm start4.elf
sudo rm fixup4.dat
sudo wget https://github.com/Hexxeh/rpi-firmware/raw/stable/start4.elf
sudo wget https://github.com/Hexxeh/rpi-firmware/raw/stable/fixup4.dat
The boot settings for the Raspberry Pi need to be changed as well. First, the SSH server is enabled. Then the fstab is cleaned so that it does not interfere with booting from the NFS share. Finally, tell the Pi to boot over the network by changing its cmdline.txt.
sudo touch /srv/nfs/rpi4-3a205fc0/boot/ssh
sudo sed -i /UUID/d /srv/nfs/rpi4-3a205fc0/etc/fstab
echo "console=serial0,115200 console=tty root=/dev/nfs nfsroot=10.0.0.1:/srv/nfs/rpi4-3a205fc0,vers=4.1,proto=tcp rw ip=dhcp rootwait elevator=deadline" | sudo tee /srv/nfs/rpi4-3a205fc0/boot/cmdline.txt
Step 4: Firewall Settings
Before you continue with setting up the services, the firewall needs to be configured to make them available. As mentioned before I use a separate network interface to connect to the Pi. This interface is assigned to the internal zone of the firewall. Then all the required services will be enabled for this zone. To apply the settings, the firewall needs to be reloaded.
sudo firewall-cmd --permanent --zone internal --change-interface enp0s3
sudo firewall-cmd --permanent --zone internal --add-service nfs3
sudo firewall-cmd --permanent --zone internal --add-service mountd
sudo firewall-cmd --permanent --zone internal --add-service rpc-bind
sudo firewall-cmd --permanent --zone internal --add-service tftp
sudo firewall-cmd --permanent --zone internal --add-service dhcp
sudo firewall-cmd --permanent --zone internal --add-service dns
sudo firewall-cmd --reload
Note: The DNS service is necessary if you plan on assigning a hostname to the Pi via DHCP (see last step). Otherwise the Pi will not be able to lookup its own IP via DNS.
Step 5: Configuring the Services
Now comes the configuration for the two services. Before you continue, use a bind-mount to make the Pi's boot directory available to the TFTP server.
echo "/srv/nfs/rpi4-3a205fc0/boot /srv/tftp/3a205fc0 none defaults,bind 0 0" | sudo tee -a /etc/fstab
sudo mount /srv/tftp/3a205fc0/
The files are now in the root directory of the TFTP server. However, it will not be able to access them as SELinux will prevent this. You therefore need to add a rule so that files in the /srv/tftp directory get labelled correctly as files to be served via TFTP. After the rule is added it needs to be applied to the files already present.
sudo semanage fcontext -a -t tftpdir_rw_t "/srv/tftp(/.*)?"
sudo restorecon -R /srv/tftp
The following commands make the necessary changes to the config files of both the NFS server and dnsmasq. Here dnsmasq will provide a DHCP service (with IPs in the range of 10.0.0.100 to 10.0.0.200) and the TFTP server.
echo "/srv/nfs/rpi4-3a205fc0 *(rw,sync,no_subtree_check,no_root_squash)" | sudo tee -a /etc/exports
echo "interface=enp0s3" | sudo tee -a /etc/dnsmasq.d/pxe-boot
echo "dhcp-range=10.0.0.100,10.0.0.200,12h" | sudo tee -a /etc/dnsmasq.d/pxe-boot
echo "log-dhcp" | sudo tee -a /etc/dnsmasq.d/pxe-boot
echo "enable-tftp" | sudo tee -a /etc/dnsmasq.d/pxe-boot
echo "tftp-root=/srv/tftp" | sudo tee -a /etc/dnsmasq.d/pxe-boot
echo 'pxe-service=0,"Raspberry Pi Boot"' | sudo tee -a /etc/dnsmasq.d/pxe-boot
All that is left to do is enabling and starting the services.
sudo systemctl enable dnsmasq
sudo systemctl enable rpcbind
sudo systemctl enable nfs-server
sudo killall dnsmasq # see comment
sudo systemctl enable dnsmasq
sudo systemctl enable rpcbind
sudo systemctl enable nfs-server
Note: When trying to start dnsmasq I would get an error hinting that there already was an instance of it running in the background. Using killall this instance gets terminated and then dnsmasq is started again.
Step 6: Finishing Touches
Power up the Pi and connect to it over SSH after it is booted. If the Pi does not answer, look in /var/log/messages for dnsmasq's output.
My setup is supposed to be more permanent, so I want to assign a fixed IP to the Pi. This can be done with dnsmasq as well.
echo "dhcp-host=dc:a6:32:35:2d:06,rpi4-3a205fc0,10.0.0.31,infinite" | sudo tee -a /etc/dnsmasq.d/static-IPs
sudo systemctl restart dnsmasq
Now the Pi gets assigned the static IP 10.0.0.31 based on its MAC address, which we retrieved in the first step. In continuing the naming scheme, it also gets assigned the hostname rpi4-3a205fc0 based on its serial number. Before the Raspberry Pi will replace its default hostname raspberrypi with the one assigned via DHCP, you need to reset it to localhost (therefore, the following command needs to be run on the Pi).
sudo hostnamectl set-hostname localhost
After rebooting the Pi and logging in once more the prompt should greet you with the freshly assigned hostname.
Note: These last steps are especially useful when booting several Pis over the network, as each can be assigned a unique name.