Kubernetes networking can be complex, but its networking model simplifies things by assigning each pod its own IP address. Containers within a pod share the same IP, enabling seamless communication between pods in a cluster without needing NAT. This means pods can directly communicate with each other using their IP addresses, making it easier to manage networking within the cluster.

In this document, we will explore how Cloudfleet Kubernetes Engine (CFKE) manages networking and security within a Kubernetes cluster, ensuring efficient communication between pods and secure data transfer across your infrastructure.

We recommend reading the Kubernetes Networking Guide to understand the basics of Kubernetes networking before diving into this document.

Node-to-Node Communication on CFKE

Kubernetes is designed to run on a flat network. This means all nodes in a cluster can communicate with each other directly, without needing NAT. This is a challenge when building a Kubernetes cluster that runs across multiple cloud providers or on-premises infrastructure. Cloudfleet Kubernetes Engine (CFKE) solves this problem by creating a secure, encrypted network between nodes using WireGuard, a modern VPN solution that encrypts all traffic between nodes, protecting your cluster data from unauthorized access.

When a new node joins a CFKE cluster, it establishes a secure connection with the other nodes and the CFKE control plane using WireGuard®. Once these tunnels are established, the nodes get a secondary IP address and start behaving like they are on the same flat network.

To establish WireGuard® tunnels, CFKE uses a combination of public and private keys. The public keys are used to encrypt the traffic, while the private keys are used to decrypt it. Just before the node joins the cluster, it shares its public key and gets the information about the other nodes from the control plane. Later, it establishes the tunnels with the other nodes using their public keys.

To create a tunnel with the control plane, exchange keys, and finally create tunnels with other instances, a CFKE node needs egress Internet access. The node can be behind NAT and does not need to have a public IP address. CFKE is designed to overcome the most complicated NAT setups and is typically able to establish tunnels directly with other nodes. If a direct connection is not possible, CFKE uses a relay server to establish the connection with performance degradation.

The WireGuard® tunnels typically run over the public Internet. However, if two nodes are already on the same flat network, CFKE understands this and establishes the connection over the private network. This is the case when two nodes are in the same network segmentation in a datacenter or the same VPC in a cloud provider. In this case, the Internet connection is only used initially to fetch the metadata to establish the VPN connection, but the communication between nodes is performed over the private network. This brings a great benefit for classical datacenter setups where the nodes are in the same network but there is no default encryption between them. CFKE always ensures that the communication between nodes is encrypted.

The WireGuard® tunnels are highly performant, but it is worth noting that they consume some node resources. However, they are tested to handle up to 10 Gbps of traffic per tunnel.

Container Network Interface (CNI)

The Container Network Interface (CNI) is a standard for creating plugins that configure network interfaces for Linux containers. CNI focuses solely on network connectivity for containers and ensures resources are properly removed when containers are deleted. Its simple specification makes it widely supported and easy to implement.

In Cloudfleet Kubernetes Engine (CFKE), Cilium is the default CNI plugin. Cilium is a cutting-edge, open-source networking and security solution designed for Kubernetes clusters. Powered by eBPF (Extended Berkeley Packet Filter) technology, Cilium ensures efficient load balancing, fast data processing, and secure communication between workloads. It’s built to handle large-scale, dynamic cloud-native environments, capable of supporting clusters with thousands of nodes and hundreds of thousands of pods, making it ideal for high-performance environments.

Cilium also enhances security with identity-based policies that go beyond traditional IP-based ACLs. This fine-grained control allows you to define who can communicate within the cluster, effectively reducing your attack surface and ensuring robust protection for cloud-native applications.

On CFKE, Cilium works on the overlay network created by WireGuard®. CFKE customers are fully able to use Cilium’s advanced features like network policies, load balancing, and service mesh capabilities with CFKE.

Frequently Asked Questions

Q: My on-premise nodes are already on a private network. Why do I need WireGuard® tunnels between them?

Our experience in regulated industries has taught us the importance of a zero-trust networking architecture. Even within a “private” network, assuming inherent trust can be a vulnerability. Designing Cloudfleet with secure defaults out-of-the-box means encrypting data in transit is standard practice, regardless of the network.

Here’s why this is crucial, even for your on-premise nodes:

• Defense in Depth: Relying solely on network segmentation isn’t enough. If a perimeter is breached, or an insider threat exists, unencrypted internal traffic is exposed. Encrypting traffic between nodes provides an essential additional layer of security.

• Evolving Threat Landscape: Today, many organizations don’t own their networking infrastructure end-to-end, even within their own data centers. This can introduce unseen vulnerabilities. “Private” networks can still be susceptible to physical taps or misconfigurations that could expose data.

• Lightweight & Efficient: WireGuard® is a modern, high-performance VPN protocol. It’s lightweight and uses minimal system resources, meaning the security benefits come with negligible performance overhead. This makes it an ideal solution for securing inter-node communication without impacting your applications.

By implementing WireGuard® tunnels, you’re adopting a proactive security posture that aligns with modern best practices and protects your data from a wider range of potential threats.

Q: The public clouds I use have exorbitant egress bandwidth fees. How can I avoid high bandwidth costs when running workloads across different clouds?

We recommend architecting your deployments with careful consideration of how the placement of different applications will affect bandwidth usage. Firstly, when one pod communicates with another, CFKE (Cloudfleet Kubernetes Engine) does its best to route traffic to the nearest target pod using its locality load balancing feature. To leverage this feature as much as possible, you can use Kubernetes affinity rules. With affinity rules, you can co-locate pods of services that frequently communicate with each other. CFKE’s auto node-provisioning respects these affinity rules and will create compute nodes to fulfill these location requirements, thereby helping to minimize cross-cloud bandwidth consumption.