For installing the servers and wifi-testbed nodes (apu boards) we have the following setup: Two RBG VMs have one service IP each in the network of the server/nodes. The service IPs are configured in the RBG DHCP neetbot entries. The netboot entry can be assigned in the StrukturDB on a per host basis. It allows the servers/nodes to receive the PXE address during boot. The host fetches the config from the PXE VM and install a new OS from scratch with the help of a preseed file.

PXE + iDrac Setup
-----------------

                     |--- 131.159.24.187  (il11), vm-address
vmott19 -------------|
(Testbed PXE)        |--- 10.200.110.1    (il11_6), service-ip [netboot:il11_wifi_testbed], pxe address il11_6
                                          [alias: pxe-apu.cm.in.tum.de; apu-maas.cm.in.tum.de]


                     |--- 10.200.120.9    (il11_2), vm-address
vmott22 -------------|
(Server PXE)         |--- 10.0.0.253      (il11_3), interface to server-mgmt network (iDrac)
                     |
                     |--- 10.200.120.7    (il11_2), service-ip [netboot:il11_server_pool], pxe address il11_2
                                          [alias: pxe.cm.in.tum.de; idrac-gw.cm.in.tum.de]
                     

The following chapters will explain the installation and configuration of that setup.

1. (Server) Make sure the the default (untagged) network is configured correctly for the brocade port the server is connected to. Use netbox to lookup the Brocade Port of the server. Write an E-Mail to the rbg-noc group to change/verify configuration

 fabric isl enable
 fabric trunk enable
 switchport
 switchport mode trunk
# chair(il11) - 83, server(il11_2) - 133, mwn (il11_5) - 240
 switchport trunk allowed vlan add 83,133,240
 no switchport trunk tag native-vlan
# server network default without tagging il11_2 (PXE)
 switchport trunk native-vlan 133
 spanning-tree shutdown
 no shutdown

1. Log in on the StrukturDB. Add a new host entry / edit host entry - configure netboot entry (il11_wifi_testbed / il11_server_pool), server must be in the right network (il11_6 / il11_2)
2. Reboot server in PXE mode - (Option1 - easy) Execute Template in AWX, select reboot and enable PXE. (Option2 - hard) Connect to server iDrac interface and reboot server with pxe - only once enabled.
3. Server should reboot and install itself.
4. (TODO) After server is up it executes an ansible callback against AWX and configures itself.

• Follow steps described in the section below.

To set up a PXE environment you need the following packages installed:

 # apt install tftpd-hpa inetutils-inetd nfs-kernel-server 

The tftp root directory is /var/lib/tftpboot. This is where all the files will be put for PXE booting. This is default directory as of the time of writing and thus does not need further configuration.

To allow the tftp daemon to serve the boot files an entry has to be made in /etc/inetd.conf, with following line:

 tftp dgram udp wait root /usr/sbin/in.tftpd /usr/sbin/in.tftpd -s /var/lib/tftpboot 

Reload the config for that service after making changes to that file with

 # service tftpd-hpa reload 

After a reboot the inetutils-inetd service sometimes grabs the tftp port and prevents tftpd-hpa from starting correctly. First stop inetutils and restart the tftp service.

sudo service inetutils-inetd stop
sudo service tftpd-hpa restart

With these settings in place we can start populating our PXE environment. We first need to have a PXE boot file which responds to boot requests. This file is specified in the netboot entry as the 'filename' (default: pxelinux.0). We therefore extract it and the ldlinux.c32 file from our downloaded .iso file in the install/netboot/ directory to the PXE root directory. We will provide a summary of the root directory structure in the end.

With the boot file in place we need to set up a boot config to specify where the PXE environment gets its kernel and initrd it should provide. This config can be specified on a MAC address basis and a default config file (which is mandatory!). The configs are by default located in the pxelinux.cfg directory in the tftp root directory. For more information read here.

We provide the two branches stable and experimental. The experimental branch is used for testing of new distributions and should only be moved to stable when everything is working as intended. A backup of the latest stable branch is never a bad idea in case things don't go as planned after overwriting it.

Our pxelinux.cfg/default config looks like the following:

DEFAULT stable

LABEL stable
	KERNEL 	stable/install/vmlinuz
	APPEND	netboot=nfs nfsroot=10.200.120.7:/var/lib/tftpboot/stable/ initrd=stable/install/netboot/ubuntu-installer/amd64/initrd.gz toram locale=en_US.UTF-8 bootkbd=us console-setup/layoutcode=us

LABEL experimental
	KERNEL 	experimental/linux
	APPEND	netboot=nfs nfsroot=10.200.120.7:/var/lib/tftpboot/experimental/ initrd=experimental/initrd.gz toram locale=en_US.UTF-8 bootkbd=us console-setup/layoutcode=us

The default entry specifies what LABEL is supposed to be booted from. Since we do not plan to choose an entry interactively on startup, we specify the branch to boot from with that entry. Change accordingly.

The KERNEL entry specifies where PXE gets the kernel image provided to the machines from.

The APPEND entry specifies multiple configration settings for the boot. netboot tells the machine, that it gets its root directory (which it mounts on kernel initiation) from a NFS server, followed by the path (nfsroot). The exact configration for that will follow in a second. initrd is the initial ramdisk the machine loads to boot and it contains all necessary modules for a working environment. With the toram directive we specify the system to load everything into RAM instead of the disk. The remaining tree entries are for locale settings.

For the netboot and nfsroot directive to work, we have to configure the NFS server, which is located on the same machine. Therefore, we have to edit the /etc/exports configuration file. We added the following lines so they are available with NFS:

/var/lib/tftpboot/stable 10.200.112.0/255.255.240.0(rw,no_root_squash,sync,no_subtree_check)
/var/lib/tftpboot/experimental 10.200.112.0/255.255.240.0(rw,no_root_squash,sync,no_subtree_check) 

The paths are also located in the tftp root directory for consistency and are simply the unpacked .iso images, containing the kernel and initrd.

With all these settings in place, we are ready to boot from PXE. To automate the installation we provide a custom initrd.gz ramdisk containing a preseed file. The tftpboot directory should look like the following:

| /var/lib/tftpboot
|
|-- pxelinux.0
|-- ldlinux.c32
|
|--| pxelinux.cfg
|  |-- default
|
|--| stable
|  |-- vmlinuz (kernel)
|  |-- initrd.gz (ramdisk)
|  |-- remaining.iso contents
|
|--| experimental
|  |-- vmlinuz (kernel)
|  |-- initrd.gz (ramdisk)
|  |-- remaining .iso contents

The filenames can of course differ depending on the image used. Some kernels are also called “linux”, for example.

1. Log in on the PXE VM/Machine
2. Install the required packages:

    sudo apt install p7zip-full fakeroot curl cpio 

3. Clone the gitlab repository:

    git clone https://gitlab.lrz.de/cm/admins/pxe-boot 

4. Basic: Adjust the build-pxe.sh script (default user, pub key, hashed password)
5. Advanced: Adjust the custom/preseed.cfg file, additional commands, include files/scripts in the custom directory
6. Execute the script:

   # experimental branch
   ./build-pxe.sh
   #stable branch
   ./build-pxe.sh stable 

There is a bash script that build the iso image and adds a preseed file to it. The script works like this:

• Download iso image if not already available
• Inject variables and commands into the preseed file
• Unpack the iso image and include the preseed file
• Load the image into the stable or experimental TFTP directory /var/lib/tftpboot/ stable OR experimental

At the top of the bash script several variables can be set or adjusted:

• SOURCE_ISO_URL = The url to download the iso image
• TFTPBOOT = directory to store the final image for pxe boot
• hostname = default hostname - the actual hostname is requested during installtion from the RBG dhcp server
• username = default user
• SSH_PUBLIC_KEY = pub key for the default user
• PASSWD_HASH = the hashed password for the default user

In the preseed file itself additional commands executed at the end of the PXE installation can be added at the bottom. To make files or scripts available on the installation target they just need to be put into the custom folder. During script execution they are included in the finished iso image and are available during the installation. During the installation the scripts can be executed or copied to the target machine.