This blog is the first in a series in which I install, configure and deploy a full GitOps based Kubernetes cluster, following best practises for production clusters. This part covers getting Talos Linux running on bare metal, the hardware behind it, and the problems I ran into along the way.
The goal for this cluster was to build something closer to production than my usual lab setups, mostly to keep my Kubernetes skills sharp for the CKA and KCNA certifications I am going for this semester. That meant treating things like secure boot and following the official best practices as requirements rather than nice to haves, which is most of what this post ends up being about.
Hardware
For hardware, I went with six HP EliteDesk 800 G3 Minis, three set up as control planes and three as workers, bought secondhand for their small physical and electrical footprint rather than raw power. Nothing about running Kubernetes needs a lot of horsepower, and small office PCs are cheap, quiet, and easy to rack.
The one place hardware choice actually mattered was storage. The control plane nodes needed SSDs with power loss protection, since etcd relies on fast, confirmed writes and is notoriously prone to giving up on disks that cannot guarantee that. The workers each got an extra disk set aside for Rook-Ceph later in this series, no PLP requirement there.
Planning the install
Before touching an installer I went through Sidero Labs’ security best practices and production cluster notes to figure out what the install actually needed to look like. Talos had just hit 1.14.0, so this cluster started out on a fresh release.
Part of that planning happened in UEFI, on every node: virtualization enabled, Wi-Fi disabled since it goes unused and is just extra surface area, and the machines set to never drop into S5 on power loss, since that state effectively kills the operating system. Once one node was configured correctly, exporting the settings to a USB stick and reapplying them to the rest saved a lot of repetition.
The other part happened at the Talos Image Factory, where I built a secure boot metal ISO with the extensions this hardware needed baked in from the start, instead of layering them on afterwards.
Secure boot
This is where most of the time went, spread across the better part of a week.
My first instinct was that the keys had to be staged manually, so I went looking for the keys to drop them into a /EFI/HP directory. That went nowhere, I had no real idea where Talos was actually storing its keys, so I was guessing.
Nearly every guide I found described the enrollment mode in UEFI, but skipped over the one thing that actually mattered: after booting the Talos installer USB, you get a few seconds before it boots into Talos itself, and the option down three ticks (on v1.14.0) in that window is what triggers key enrollment. Miss that window and you just boot straight past it. None of the guides I read said this outright, except for oct8l’s writeup for the EliteDesk 800 G5, which is likely why it took as long as it did.
The other inconsistency was around secure boot’s own state during enrollment. Almost every guide I found said to disable secure boot while importing custom keys. On my hardware that was backwards, it had to stay enabled, in what is effectively an “import mode” that starts on the first boot after selecting “import custom keys” under the secure boot section of the advanced tab. Also here was oct8l’s writeup the only one matching my experience, which finally got me unstuck. My G3 had no option to disable HP Sure Start, but it didn’t need one, since that never blocked my “import custom keys” button like it does on the G5.
A couple of real gotchas
With secure boot sorted, the nodes sat in maintenance mode on their DHCP addresses, waiting for a config. I generated a first version of the Novio cluster config, wrote a config patch per node for hostname, IP, and the rest of the node specific bits, and applied those to the three control planes to get Talos installed to disk. This is where the first control plane took hours instead of minutes, for no reason I could pin down, and never happened again on any of the other five, so I didn’t investigate further. I suspect a reboot would have fixed this. The patches themselves live in my homelab repository, under the Novio subfolder, alongside the rest of the host level configuration for this cluster.
I stopped there for the night, planning to bootstrap the cluster in the morning. While reading through the production notes before doing that, I found out I had it backwards, the secrets bundle is meant to be generated separately, before the cluster config, not left to be generated alongside it. So the next morning I regenerated the Novio config properly, this time from that dedicated secrets bundle and with the specific factory image URL included as Sidero recommends, cleared the ephemeral partition on every node to drop them back into maintenance mode, and reapplied the patches to the control planes and then the workers. Bootstrapping from a single control plane, as those same notes describe, went smoothly, and I finished by generating a separate kubeconfig for the cluster.
What is next
Novio is up now, three control planes, three workers, Talos with secure boot enabled throughout. Next in this series: Flux for GitOps, Cilium as CNI, and Rook-Ceph on those extra worker disks.