What is a Kubernetes CNI plugin, and how do you choose one?
A CNI plugin is the component that connects Pods to the cluster network. When a Pod starts, the container runtime calls the plugin, which creates a network interface in the Pod, assigns it an IP address and sets up the routes that let it reach every other Pod. Kubernetes ships no Pod network of its own, so every cluster runs one: Cilium, Calico and Flannel are common open source choices, and managed services often default to the provider’s plugin, such as the Amazon VPC CNI.
What a CNI plugin does
CNI, the Container Network Interface, is a specification for how a container runtime asks a plugin to set up networking. The runtime runs the plugin binary with an operation (ADD, DEL, CHECK, GC, VERSION or STATUS) and passes the network configuration as JSON. On ADD, the plugin creates the interface inside the Pod’s network namespace, and an IP address management (IPAM) plugin picks the address, gateway and routes.
Kubernetes requires a plugin compatible with CNI spec v0.4.0 or later and recommends v1.0.0. Since Kubernetes 1.24, the container runtime loads CNI plugins; the kubelet no longer does. Configuration files usually live in /etc/cni/net.d and binaries in /opt/cni/bin.
The specification covers only connectivity. Most projects called “CNI plugins” are full networking stacks with an agent on every node that also enforces NetworkPolicy, load-balances Services, encrypts traffic and records flows. The Kubernetes documentation notes that if the plugin does not implement network policy, the API is still there but has no effect.
Overlay, native routing or cloud network IPs
| Mode | How Pod packets travel | Strengths | Limits |
|---|---|---|---|
| Overlay (VXLAN, Geneve, IP-in-IP) | Encapsulated inside packets between node IPs | Works on any network where nodes reach each other | 20 to 70 bytes of overhead per packet; Pod IPs invisible outside the cluster |
| Native routing | The network routes Pod address ranges directly, often learned through BGP | No encapsulation; Pods addressable from the data center | The network has to learn and carry Pod routes |
| Cloud network CNI | Pods get IPs from the provider’s subnet | Pods addressable in the VPC; provider firewall rules apply | Consumes subnet addresses; tied to one provider |
Overlays are the default on most self-installed clusters. Native routing with BGP is common on-premises. Cloud network plugins, such as the Amazon VPC CNI, are the default on the matching managed services.
Data plane: iptables, nftables or eBPF
The data plane is the code in the Linux kernel that forwards, filters and load-balances each packet. Traditional setups use iptables rules, written by the CNI plugin for policy and by kube-proxy for Services. Rule lists grow with the number of Services and policies. nftables is the newer kernel framework that replaces iptables.
eBPF data planes load programs into the kernel that make the same decisions with hash-table lookups. They can also replace kube-proxy entirely, as Cilium’s kube-proxy replacement and Calico’s eBPF data plane do, and they make per-flow observability cheap.
Checklist for choosing
| Requirement | What to check |
|---|---|
| Network policy | Standard NetworkPolicy support, plus Layer 7, domain-name and cluster-wide rules if you need them. See Kubernetes network policies. |
| Encryption | WireGuard® or IPsec between nodes, and which traffic it leaves out. See whether Pod traffic is encrypted. |
| Egress control | A fixed outbound IP per workload. See egress gateways. |
| Observability | Flow logs with policy verdicts, not only packet counters |
| Routing | BGP peering with data center routers for on-premises clusters |
| Platform | Windows nodes, IPv6 or dual stack, and support on your managed service |
| Operations | Who installs and upgrades it, and how upgrades affect running traffic |
Three common plugins compared
| Cilium | Calico (open source) | Flannel | |
|---|---|---|---|
| Data planes | eBPF | iptables, nftables, eBPF, Windows, VPP | Kernel routing with VXLAN or other backends |
| Network policy | Kubernetes policy plus Layer 7 and domain-name rules | Kubernetes policy plus Calico policy resources | Not built in; an optional controller or another plugin adds it |
| Encryption | WireGuard®, IPsec | WireGuard® | WireGuard® backend |
| kube-proxy replacement | Yes | In eBPF mode | No |
| Egress gateway | Yes | Commercial editions only | No |
| BGP | Yes | Yes | No |
| Project | CNCF graduated project | Open source project led by Tigera | Open source project under flannel-io |
Flannel is the smallest of the three. It only moves packets between hosts, and its agent does not enforce network policy; its Helm chart can deploy the Kubernetes SIGs network policy controller, or you can add Calico or Cilium for policy. Calico has the widest choice of data planes and a long history of BGP networking. Cilium is built around eBPF and includes flow observability through Hubble.
Why the choice is hard to change
Every node holds interfaces, routes and Pod IP addresses created by the current plugin. Switching means re-creating every Pod and usually rolling every node, while Pods on old and new nodes still need to reach each other. Policies written as plugin-specific resources, such as CiliumNetworkPolicy, have to be rewritten. Cilium documents a migration procedure that runs both plugins side by side during the move, but plan for the plugin you will need in two years, not the one that is quickest to install today.
The CNI on Cloudfleet
Cloudfleet Kubernetes Engine (CFKE) uses Cilium as its CNI plugin and installs, configures and upgrades it on every node, including self-managed nodes. Cilium runs on top of an encrypted WireGuard® network between nodes across clouds and data centers. See the networking architecture for details.