What is an immutable operating system for Kubernetes nodes?
An immutable operating system for Kubernetes is a minimal Linux whose system files are read-only at runtime and replaced as a whole image during updates, instead of being changed package by package. Nodes are configured through a declarative file or an API at boot, updates switch atomically between two image slots, and a failed update rolls back to the previous image. Talos Linux, Flatcar Container Linux, Bottlerocket, Fedora CoreOS and Google’s Container-Optimized OS are common examples.
How an immutable OS differs from a general-purpose one
On a general-purpose distribution such as Ubuntu or RHEL, administrators install packages, edit files in /etc and apply updates one package at a time. Over months, nodes that started identical drift apart. An immutable OS removes those options:
| Property | General-purpose Linux | Immutable OS |
|---|---|---|
| System files | Writable, changed by packages and admins | Read-only image, often verified at boot |
| Configuration | Edited on the node, by hand or with Ansible and similar tools | Supplied at provisioning or through an API, then reapplied |
| Updates | Package by package, in place | Whole image written to an inactive slot, activated by reboot |
| Rollback | Manual, rarely clean | Boot the previous image |
| Package manager | Yes | Usually none |
| Size | Hundreds or thousands of packages | Only what a container host needs |
The A/B scheme is the core idea. The node has two system partitions. An update is written to the inactive one, the node reboots into it, and if it fails to boot, the bootloader returns to the partition that worked.
Common immutable operating systems
| OS | Defining traits |
|---|---|
| Talos Linux | Built only for Kubernetes. Fewer than 50 binaries, no SSH, no shell and no package manager. Every interaction goes through a gRPC API with mutual TLS, usually with talosctl. |
| Flatcar Container Linux | Minimal, container-optimized successor to CoreOS Container Linux, with read-only system images and atomic updates. Runs any Kubernetes distribution. A CNCF Incubating project since 2024. |
| Bottlerocket | Open-source OS from AWS for EKS and ECS, with VMware variants. Root filesystem verified with dm-verity, settings changed through an API, no SSH or shell by default. |
| Fedora CoreOS | Automatically updating, minimal OS that combines Ignition from Container Linux with rpm-ostree from Fedora Atomic Host. |
| Container-Optimized OS | Google’s default GKE node image. No package manager and automatic weekly updates. Not supported outside Google Cloud. |
k3OS, a Rancher OS built around k3s, was archived in December 2023 and is no longer maintained.
Benefits
- No configuration drift. Every node runs the same image, so a problem on one node reproduces on another and a fix applies to all.
- Smaller attack surface. Fewer binaries and no package manager leave an attacker less to use. Without SSH or a shell, a stolen password does not open a login session.
- Tamper resistance. A read-only, verified system partition makes persistent changes to the OS hard.
- Safer updates. An update either applies completely or the node returns to the last working image. There is no half-patched state.
- Fits declarative operations. Node configuration lives in version control, in the same way as Kubernetes manifests.
Operational costs
Immutability moves work rather than removing it.
- Debugging without a shell. On Talos, you read logs and run diagnostics through the API. On Bottlerocket, you enable an admin container. Teams used to SSH sessions need new habits and tooling.
- Update coordination. Each update ends in a reboot, and a reboot must not take down too many Pods at once. Nodes need to be cordoned and drained one at a time. The Bottlerocket update operator drains nodes before updating them and limits concurrent updates. kured, a CNCF Sandbox project, does the same for operating systems that signal a needed reboot with a sentinel file. Talos cordons and drains a node itself during an OS upgrade.
- Agents and drivers. Security agents, storage drivers and GPU drivers that expect to install packages on the host must run as containers or as OS extensions instead.
- A second update stream. OS images and Kubernetes versions update on separate schedules on most immutable systems. On Talos, an OS upgrade does not change the Kubernetes version. Plan both, as described in how to upgrade a Kubernetes cluster.
Where an immutable OS fits best
Immutable systems pay off where nodes are many, remote or hard to reach. On bare metal, one image installed from PXE or USB makes every server identical. At edge sites, atomic updates with automatic rollback mean a failed update does not need a technician on site. See edge Kubernetes and running Kubernetes on bare metal.
They fit less well where nodes host software that expects a traditional Linux, such as vendor agents with no container version, or where the team depends on interactive access for troubleshooting.
Node operating systems on Cloudfleet
Cloudfleet provisions and replaces cloud nodes through node auto-provisioning and handles their lifecycle, including upgrades. Self-managed nodes in your own data center or another cloud run Ubuntu, Debian or the RHEL family. See self-managed nodes for the requirements.