Self-managed nodes

For supported cloud providers (AWS, GCP, and Hetzner Cloud), Cloudfleet provides node auto-provisioning that automatically creates and manages compute nodes based on your workload requirements. For on-premises infrastructure and other cloud providers, you manually provision compute nodes and register them with Cloudfleet. This guide explains how to add self-managed nodes to your CFKE cluster.

Not sure which approach to use? See How Cloudfleet provisions nodes for a comparison of node auto-provisioning versus self-managed nodes.

Having problems with an existing node? See Self-managed node issues for join failures, frozen nodes, and other common problems.

Self-managed nodes are Linux machines that you provision and give to CFKE to allow it to run workloads on them. This allows you to use your own hardware or third-party cloud resources to expand your cluster. Self-managed nodes act as equal members of the cluster and can run any workload that their hardware supports. Bare-metal self-managed nodes are also a natural fit for Kata Containers, which run untrusted workloads inside lightweight VMs and require hardware virtualization on the node.

Requirements

To be able to add self-managed nodes to your cluster, your node must meet the following requirements:

• Your node must run one of the supported operating systems:
  • Ubuntu 22.04 or 24.04
  • Debian 12 or 13
  • Red Hat Enterprise Linux 9 or 10, or a rebuild such as Rocky Linux, AlmaLinux, or CentOS Stream
• You must have SSH access from a bastion host or your local workstation with root access
• Your node must have egress internet access
• Your firewall must allow your server to initiate TCP connections to *:443, UDP from :41641 to : and UDP to *:3478.
• Your node can be behind NAT and does not need a public IP address. Cloudfleet is able to establish a secure tunnel behind NAT.

Any other release is refused up front with the list of supported operating systems. The cloud-init method described below is currently available for Ubuntu and Debian; use the CLI or the Terraform SSH resource for nodes in the RHEL family.

Please note that egress internet access is required for the self-managed nodes to be able to communicate with the CFKE control plane and establish VPN connections with other nodes in the cluster. If two self-managed nodes are in the same network and have a connection via a private IP, they communicate with each other over the private network. In this case, the VPN between nodes is still established and node-to-node communication is encrypted.

How a node is provisioned

Whether you use the CLI or the Terraform provider, you point Cloudfleet at a machine and Cloudfleet configures it: the Kubernetes components are installed at the versions your cluster runs, the host is prepared, the node joins the cluster’s secure overlay network, and it registers with the control plane. The node is typically Ready about 30 seconds after the command on Ubuntu and Debian, and 40 to 80 seconds on the RHEL family.

Adding a node is safe to repeat on the same machine. If a run is interrupted or fails partway, run the same command (or apply the same Terraform configuration) again.

Adding self-managed nodes with Terraform

For production deployments, infrastructure-as-code workflows, or when adding multiple nodes at once, use the Cloudfleet Terraform provider. The provider supports two methods:

• Cloud-init (preferred): Generates configuration that automatically registers nodes when they boot. Use this method when your target platform supports cloud-init, as it enables fully automated provisioning without requiring SSH access during setup.
• SSH: Connects to existing machines via SSH to install and configure them. Use this for bare-metal servers, existing VMs, or platforms that do not support cloud-init.

Using Terraform with cloud-init

The Cloudfleet Terraform provider generates cloud-init userdata that you can use with any cloud provider or virtualization platform that supports cloud-init. This is the preferred method when your infrastructure supports it, as nodes automatically register themselves during boot without requiring SSH connectivity from your Terraform environment:

terraform {
  required_providers {
    cloudfleet = {
      source = "terraform.cloudfleet.ai/cloudfleet/cloudfleet"
    }
  }
}

provider "cloudfleet" {}

# Reference your existing cluster
data "cloudfleet_cfke_cluster" "cluster" {
  id = "YOUR_CLUSTER_ID"
}

# Generate cloud-init configuration for self-managed nodes
resource "cloudfleet_cfke_node_join_information" "node" {
  cluster_id = data.cloudfleet_cfke_cluster.cluster.id
  region     = "your-region"    # e.g., "eu-west-1", "on-premises"
  zone       = "your-zone"      # e.g., "datacenter-1", "rack-a"

  node_labels = {
    "cfke.io/provider" = "custom"  # Optional: identify node origin
  }

  # Optional: taints applied when the node first registers
  node_taints = [
    {
      key    = "dedicated"
      value  = "batch"
      effect = "NoSchedule"
    }
  ]
}

# The rendered cloud-init configuration is available as:
# cloudfleet_cfke_node_join_information.node.rendered
#
# Pass this to your VM's user_data field

The region and zone values become Kubernetes node labels (topology.kubernetes.io/region and topology.kubernetes.io/zone), allowing you to schedule workloads to specific locations using standard node selectors. node_labels adds further labels, and node_taints registers the node with taints (NoSchedule, PreferNoSchedule, or NoExecute) from the moment it joins, so no workload lands on it before you apply your tolerations. node_taints requires Terraform provider 1.5.0 or later.

Using the cloud-init output

The rendered attribute contains the cloud-init userdata. Pass this to your infrastructure provider’s user_data or cloud-init field:

# Example: Generic VM resource (syntax varies by provider)
resource "your_provider_instance" "node" {
  # ... other configuration ...

  user_data = cloudfleet_cfke_node_join_information.node.rendered
}

When the VM boots, cloud-init executes the configuration and the machine configures itself and joins your CFKE cluster without any further action on your side. The cloud-init output is generated for Ubuntu and Debian images.

Provider-specific guides

For complete Terraform examples including VM provisioning, firewall configuration, and security best practices, see the provider-specific guides:

For other providers that support cloud-init (DigitalOcean, Linode, VMware, OpenStack, etc.), adapt the examples above using your provider’s Terraform resources.

Adding GPU support with cloud-init

To enable NVIDIA GPU support on cloud-init provisioned nodes, add the install_nvidia_drivers option:

resource "cloudfleet_cfke_node_join_information" "gpu_node" {
  cluster_id            = data.cloudfleet_cfke_cluster.cluster.id
  region                = "your-region"
  zone                  = "your-zone"
  install_nvidia_drivers = true

  node_labels = {
    "cfke.io/provider"         = "custom"
    "cfke.io/accelerator-name" = "V100"  # Optional: specify GPU model
  }
}

Using Terraform with SSH

The cloudfleet_cfke_self_managed_node resource connects directly to existing machines via SSH and configures them as Kubernetes nodes. Use this method for bare-metal servers, existing VMs, or platforms that do not support cloud-init:

resource "cloudfleet_cfke_self_managed_node" "server" {
  cluster_id = cloudfleet_cfke_cluster.example.id
  region     = "datacenter-1"
  zone       = "rack-a"

  ssh {
    host             = "192.168.1.100"
    user             = "ubuntu"
    private_key_path = "~/.ssh/id_rsa"
    port             = 22  # Optional, defaults to 22
  }

  node_labels = {
    "environment" = "production"
  }

  # Optional: taints applied when the node first registers
  node_taints = [
    {
      key    = "dedicated"
      value  = "database"
      effect = "NoSchedule"
    }
  ]
}

The region and zone values become Kubernetes node labels (topology.kubernetes.io/region and topology.kubernetes.io/zone), allowing you to schedule workloads to specific locations using standard node selectors. node_labels and node_taints work the same way as on the cloud-init resource. The SSH resource supports every operating system listed under Requirements, including the RHEL family.

For GPU nodes with SSH provisioning:

resource "cloudfleet_cfke_self_managed_node" "gpu_server" {
  cluster_id             = cloudfleet_cfke_cluster.example.id
  region                 = "datacenter-1"
  zone                   = "rack-b"
  install_nvidia_drivers = true

  ssh {
    host             = "192.168.1.101"
    user             = "ubuntu"
    private_key_path = "~/.ssh/id_rsa"
  }

  node_labels = {
    "cfke.io/accelerator-name" = "RTX-4090"
  }
}

For complete SSH resource documentation, see the Terraform resources reference.

Adding self-managed nodes with the CLI

For quick testing, single-node additions, or environments where Terraform is not available, use the Cloudfleet CLI.

Basic CLI usage

1. Install the Cloudfleet CLI on a workstation by following the instructions here and configure it for your account.

2. Run the following command to add a self-managed node to your cluster:

cloudfleet clusters add-self-managed-node CLUSTER_ID \
  --host HOST_IP \
  --ssh-username SSH_USERNAME \
  --ssh-key SSH_KEY_LOCATION \
  --ssh-port SSH_PORT \
  --region DATACENTER_REGION \
  --zone DATACENTER_ZONE

region and zone are mandatory flags that specify the location of the node. These values become Kubernetes node labels (topology.kubernetes.io/region and topology.kubernetes.io/zone), allowing you to schedule workloads to specific locations using standard node selectors. You can use any string that makes sense for your infrastructure. The best practice is to use the datacenter region as the region value and the failure domain (e.g., rack) as the zone value.

• You can omit the --ssh-username flag if your node uses the default username root.
• You can omit the --ssh-key flag if you have an SSH agent running and the key is added to the agent.
• You can omit the --ssh-port flag if your node uses the default SSH port 22.

The CLI connects to the machine, configures it, and then waits for the node to register with the cluster and report Ready. Progress is shown on stderr, and when the node is ready the CLI prints a summary of the node:

NAME                 STATUS  READY AFTER  OS                              KERNEL          ARCH   KUBELET   RUNTIME           CPU (ALLOCATABLE)  MEMORY (ALLOCATABLE)  REGION  ZONE
cfke-test-debian-12  Ready   1m7s         Debian GNU/Linux 12 (bookworm)  6.1.0-52-amd64  amd64  v1.34.11  containerd 2.1.5  2 (1830m)          3.7 GiB (2.3 GiB)     test    debian-12

The summary respects --output-format json|yaml|table (-o). With -o json or -o yaml the node is printed as a document with the fields name, status, os, kernel, architecture, kubelet, container_runtime, cpu, cpu_allocatable, memory, memory_allocatable, region, zone, and ready_after, which scripts can consume while progress stays on stderr.

• --no-wait returns as soon as the node is configured, without waiting for it to report Ready.
• --wait-timeout sets how long to wait for the node to become Ready (defaults to 3m).
• --verbose shows every command the CLI runs on the node over SSH, which helps when a step fails.

If the SSH connection fails, the CLI reports the specific cause: nothing listening on the port, an unreachable or unresolvable host, a refused login for the given user and key, or a host key that does not match ~/.ssh/known_hosts (with the ssh-keygen -R command to clear the old entry).

3. Verify that the node is added by running:

kubectl get nodes

Adding labels and taints with the CLI

Use --label to add Kubernetes labels and --taint to register the node with taints (requires CLI 1.5.0 or later). Both flags can be repeated. Labels are merged with the topology labels, and a label with the same key overrides them. Taints use the kubectl taint syntax, key=value:Effect or key:Effect, with NoSchedule, PreferNoSchedule, or NoExecute as the effect, and are applied when the node first registers so nothing is scheduled on it before your tolerations are in place:

cloudfleet clusters add-self-managed-node CLUSTER_ID \
  --host HOST_IP \
  --region DATACENTER_REGION \
  --zone DATACENTER_ZONE \
  --label environment=production \
  --label cfke.io/provider=custom \
  --taint dedicated=database:NoSchedule

Adding GPU support with the CLI

If you have a self-managed node with an NVIDIA GPU, enable GPU support by adding the --install-nvidia-drivers flag:

cloudfleet clusters add-self-managed-node CLUSTER_ID \
  --host HOST_IP \
  --ssh-username SSH_USERNAME \
  --ssh-key SSH_KEY_LOCATION \
  --region DATACENTER_REGION \
  --zone DATACENTER_ZONE \
  --install-nvidia-drivers

This installs the NVIDIA driver and configures the NVIDIA container runtime. See NVIDIA GPU configuration.

NVIDIA GPU configuration

When GPU support is enabled (via Terraform or CLI), Cloudfleet:

• Installs the NVIDIA driver that matches the cluster’s release. On Ubuntu and the RHEL family 9 this takes about as long as a node without a GPU; on Debian and the RHEL family 10 it adds a few minutes to the first boot, and Debian gets the newest driver it publishes.
• Installs and configures the NVIDIA container runtime
• Labels the node with cfke.io/accelerator-manufacturer: NVIDIA
• Updates the node capacity nvidia.com/gpu field with the number of GPUs

Please note that the usage of the NVIDIA drivers is subject to the NVIDIA Driver License Agreement. By using the NVIDIA drivers, you agree to the terms of the NVIDIA Driver License Agreement.

Verify GPU configuration:

kubectl get node -o custom-columns=NAME:.metadata.name,CAPACITY:.status.capacity

After the node is initialized, give it a few seconds until the capacity value is updated.

Adding GPU model labels

When you have different GPU models in the cluster, add extra labels to identify them:

Option 1: Manual labeling

kubectl label node NODE_NAME cfke.io/accelerator-name=V100

Option 2: Automated labeling with Node Feature Discovery

The Node Feature Discovery project can automatically label nodes based on their hardware features. This Kubernetes SIG project discovers hardware features and advertises them as node labels.

For more information about GPU workloads, see GPU-based workloads and the Kubernetes GPU documentation.

Removing a self-managed node

To remove a node from a cluster:

1. Run the following commands on the node. On Ubuntu and Debian:

sudo apt remove -y kubelet
sudo rm -rf /etc/kubernetes/
sudo rm -rf /var/lib/kubelet/

On the RHEL family:

sudo dnf remove -y kubelet
sudo rm -rf /etc/kubernetes/
sudo rm -rf /var/lib/kubelet/

2. Remove the node from the cluster:

kubectl delete node NODE_NAME

Even if you do not run this last command, the cluster garbage collector will delete the node after it becomes NotReady. However, removing the node manually immediately deletes the pods scheduled on it, causing them to be recreated on other nodes.

Moving a node to another cluster

To add a self-managed node to a different cluster, first remove it from the current cluster following the steps above. This ensures the node is properly unregistered and does not retain any configuration from the previous cluster. After removal, follow the steps in this guide to add the node to the new cluster.