Self-managed node issues

Self-managed nodes let you join any Linux server to a CFKE cluster. Unlike auto-provisioned nodes, you own the machine, its operating system, and its network. This page covers the failures we most often see, and where the responsibility boundary between you and Cloudfleet runs. For setup, see Self-managed nodes.

The join command fails

cloudfleet clusters add-self-managed-node connects to the machine over SSH, configures it, and waits for it to join the cluster. Run the command with --verbose to see every command it executes on the node. Adding a node is safe to repeat on the same machine, so once you have fixed the cause, run the same command again (or re-apply your Terraform configuration). The most common failures:

The SSH connection fails. The CLI reports the specific cause. Nothing listening for SSH on the address and port usually means a machine that was just created is still booting; check --host and --ssh-port. A host that cannot be reached or does not resolve points at the address or a firewall between you and the node. A refused login names the user and key it tried; check --ssh-username and --ssh-key, and that the public key is in that user’s authorized_keys. The CLI tries the key file from --ssh-key as well as the keys loaded into your SSH agent. A host key mismatch means the address was reused by a reinstalled or newly created machine; remove the stale entry with ssh-keygen -R HOST and run the command again.

Unsupported operating system. The CLI checks the operating system before changing anything and refuses releases it does not support. Self-managed nodes must run Ubuntu 22.04 or 24.04, Debian 12 or 13, or the RHEL family (RHEL, Rocky Linux, AlmaLinux, CentOS Stream) 9 or 10.

Package installation fails. The node needs egress internet access to download the Kubernetes packages. If the command fails while downloading from pkgs.k8s.io, test IPv4 and IPv6 reachability separately from the node:

curl -4 -sS -o /dev/null -w '%{http_code}\n' https://pkgs.k8s.io/core:/stable:/v1.33/deb/Release.key
curl -6 -sS -o /dev/null -w '%{http_code}\n' https://pkgs.k8s.io/core:/stable:/v1.33/deb/Release.key

A 403 only on IPv6 means your provider’s IPv6 range is blocked by the CDN serving the Kubernetes packages, and the package manager’s downloads prefer IPv6. This is a recurring problem on some VPS providers. Prefer a machine with a routed public IPv4 address; an IPv6-only or unrouted-IPv4 server also breaks image pulls from IPv4-only registries (such as ghcr.io) and pod egress later, with symptoms like dial tcp IP:443: i/o timeout even though kubectl still works.

After fixing the network, run the join command again.

The node is configured but never becomes Ready. The CLI waits up to three minutes (--wait-timeout) for the node to register and report Ready. If it times out:

• Run kubectl describe node NODE_NAME (or kubectl get nodes if the node has not appeared yet) to see whether the node has registered and what its conditions report.
• On the machine, sudo journalctl -u kubelet shows why kubelet cannot register with the control plane.
• Check that the machine can reach the internet and the cluster endpoint (cloudfleet clusters describe CLUSTER_ID shows it), and that the firewall rules listed under Requirements are in place.
• Once the cause is fixed, run the join command again.

SSH access is not possible. If you cannot reach the machine over SSH from your workstation, generate the node’s join configuration as cloud-init data with the Terraform provider and apply it yourself when the machine boots:

resource "cloudfleet_cfke_node_join_information" "node" {
  cluster_id = CLUSTER_ID
  region     = "datacenter-1"
  zone       = "rack-a"
}

See the Terraform documentation and the provider-specific guides for examples.

The node freezes when workloads start

Symptom: the machine becomes unreachable over SSH whenever kubelet and the container runtime start, or shortly after a specific workload is scheduled.

Cause seen in practice: a workload without CPU limits (for example CPU-based LLM inference) consumed every core, starving kubelet, the container runtime, and SSH itself. The node appears dead although the hardware is fine.

Fix: set CPU limits on heavy workloads so cores remain for the system (for example, cap a workload at 12 of 16 cores), and set realistic requests everywhere.

This failure mode is not specific to self-managed nodes: a workload without resource requests and limits can starve the system components on any node and destabilize the whole cluster. Missing resource requests are the single most common root cause behind unstable clusters we see in support. On self-managed nodes the impact is worse because no provider replaces a starved machine automatically, and on auto-provisioned nodes missing requests additionally break node sizing (see Pods stuck in Pending).

Pods stop starting, existing pods go Unknown

Symptom: the node reports healthy, but no new container starts and existing pods drift into Unknown.

Cause seen in practice: /etc/resolv.conf disappeared on the host (systemd-resolved stopped or the symlink was removed). The container runtime requires it to create every pod sandbox.

Fix: restore DNS resolution on the host, then re-add the node:

sudo systemctl restart systemd-resolved
# or recreate the symlink:
sudo ln -sf /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf

The node is unreachable and logs fail with “No agent available”

Symptom: the node is NotReady, kubectl logs for pods on it fails with No agent available, and SSH times out.

Cause: the machine itself is down or cut off, often a provider incident on the physical host.

What to know: Cloudfleet cannot see, escalate, or resolve incidents in your provider account; check the provider’s status page and open a ticket with them. This is the key operational difference from auto-provisioned nodes, which Cloudfleet replaces automatically when a host fails.

Do not install a cloud provider integration

Do not install a cloud controller manager (for example the Hetzner cloud-controller-manager) or a provider node agent on CFKE nodes. CFKE ships its own cloud integration that manages load balancers (including PROXY protocol support) with its own annotations; a second controller conflicts with it and has caused cluster-wide networking outages. For persistent volumes, follow the provider-specific CSI guidance instead (see Persistent volume issues).

Also leave the host firewall permissive for the cluster’s overlay traffic; custom firewall rules that block it cut pods off from the API server (dial tcp 10.96.0.1:443: connect: no route to host from pod-network pods while host-network pods still work).

Upgrades

Self-managed nodes are synchronized with the control plane version by re-adding them. See Kubernetes versions and upgrades.