Egress Gateways

Cloudfleet Kubernetes Engine (CFKE) provides the ability to define specific egress points for your workloads. This is achieved using Cilium’s Egress Gateway feature, which is integrated into CFKE. This document explains how to configure and use Egress Gateways to ensure that outbound traffic from selected pods originates from a predictable, static IP address.

This is particularly useful when your applications need to access external services that have IP-based whitelisting or when you need a consistent source IP for auditing or logging purposes.

Another use case is to allow your cloud instances to access protected on-premise resources. In this case, you can set up a self-managed on-premise node with access to these on-premise resources. Your pods running on cloud instances can then access these resources via the self-managed node.

How It Works

The Cilium Egress Gateway feature works by routing outbound traffic from specified pods through a designated “gateway” node (or one of a set of gateway nodes). On this gateway node, the traffic is then Source NAT’d (SNAT’d) to a specific IP address (the egressIP) before it leaves the cluster.

Policy Definition: You create a CiliumEgressGatewayPolicy resource.
Pod Selection: This policy selects specific pods based on labels and/or namespaces.
Destination Matching: The policy also specifies the external destination CIDRs for which the egress rule should apply.
Gateway Node Selection: You define which node(s) in your cluster will act as the egress gateway, typically by using node labels.
Egress IP Assignment: You specify the static IP address (egressIP) that should be used as the source IP for this outbound traffic. This IP must be configured on an interface of the selected gateway node(s).
Traffic Routing: Cilium then ensures that traffic from the selected pods, destined for the specified CIDRs, is routed via the chosen gateway node and masqueraded with the egressIP.

Crafting the CiliumEgressGatewayPolicy Manifest

To implement an Egress Gateway, you need to create a YAML manifest for a CiliumEgressGatewayPolicy custom resource. This resource tells Cilium how to handle the egress traffic.

Here’s a breakdown of the key fields in the CiliumEgressGatewayPolicy manifest:

apiVersion: cilium.io/v2
kind: CiliumEgressGatewayPolicy
metadata:
  name: my-app-egress-policy # A unique name for your policy
spec:
  # 1. Select the source pods this policy applies to
  selectors:
    - podSelector:
        matchLabels:
          app: my-app
          # To select pods in a specific namespace, use the 'io.kubernetes.pod.namespace' label:
          # io.kubernetes.pod.namespace: my-namespace
      # You can also select entire namespaces:
      # namespaceSelector:
      #   matchLabels:
      #     team: backend-team

      # To select all pods, you can use NotIn with a non-existing label:
      #    matchExpressions:
      #        # All
      #        - {key: not-existing, operator: NotIn, values: [not-existing]}

  # 2. Specify the external destination CIDRs this policy applies to
  destinationCIDRs:
    - "192.0.2.0/24"   # Example external service CIDR
    - "203.0.113.10/32" # Example specific external IP
    # To route all traffic from selected pods through the egress gateway, use:
    # - "0.0.0.0/0"

  # 3. Configure the Egress Gateway node and IP
  egressGateway:
    # Select the node(s) that will act as the egress gateway
    nodeSelector:
      matchLabels:
        # Ensure this label exists on your intended gateway node(s)
        kubernetes.io/hostname: self-managed-egress-node

    # (Optional) The static IP address to be used for egress traffic.
    # This IP MUST be configured on a network interface of the selected gateway node(s).
    egressIP: "10.0.1.100" # Replace with your desired static egress IP

    # (Optional) Alternatively, specify the interface name on the gateway node.
    # Cilium will use the first IPv4 address assigned to this interface.
    # interface: "ethX"

spec.selectors: This array defines which pods are subject to this egress policy.
podSelector: Uses standard Kubernetes label selectors (matchLabels or matchExpressions) to identify pods.
- To target pods within a specific namespace, you can use the io.kubernetes.pod.namespace: <namespace-name> label within the podSelector.
namespaceSelector: Uses standard Kubernetes label selectors to identify all pods within matching namespaces.
spec.destinationCIDRs: An array of strings, where each string is an IP CIDR notation (e.g., 1.2.3.0/24, 4.5.6.7/32). The egress policy will only apply to traffic from the selected pods that is destined for an IP address within these CIDRs. For all other destinations, traffic will follow standard routing.
spec.egressGateway: This object configures the actual gateway node and the IP address to be used.
nodeSelector: A standard Kubernetes label selector that identifies the node(s) to be used as egress gateways.
- You must ensure that at least one node in your CFKE cluster has the labels specified here.
- If multiple nodes match, Cilium (in its standard configuration) typically selects the first node in lexical order by name. For High Availability (HA) setups, refer to the advanced Cilium documentation, as the basic CiliumEgressGatewayPolicy focuses on a primary gateway.
egressIP (Optional): This is the static IP address that will be used as the source IP for outbound traffic matching this policy.
- Crucial: This IP address must be manually configured by you on a network interface of the node(s) selected by nodeSelector. CFKE and Cilium will not provision this IP for you; they will only use it if it’s present on the gateway node.
interface (Optional Alternative): If you do not specify egressIP, you can specify interface. Cilium will then use the first IPv4 address assigned to this named network interface on the gateway node as the egress IP. If neither egressIP nor interface is specified, Cilium defaults to using the first IPv4 address on the interface holding the default route on the gateway node.

Key Considerations

Node Labeling: Ensure your intended gateway node(s) are correctly labeled so that the nodeSelector in your policy can identify them. You can use kubernetes.io/hostname as a label to specify one single node, preferably a self-managed one with a static name.
Egress IP Provisioning and Configuration: This is the most critical step.
- The egressIP you specify in the policy must be an IP address that you have provisioned and configured on a network interface of the chosen gateway node(s) within your CFKE environment.
- This IP must be routable from your CFKE network to the external destinations.
- Consult your network administrator or CFKE support if you are unsure how to assign an additional IP to a node’s interface or ensure its routability.
External Firewalls: If you are using the Egress Gateway to communicate with external services protected by firewalls, ensure that the chosen egressIP is whitelisted in those external firewalls.
Cilium Enablement: The Egress Gateway feature is enabled by default in CFKE’s Cilium installation. However, ensure your Cilium version supports it and that BPF masquerading is enabled (CFKE typically manages this).

Basic Example Manifest

Here’s a complete example of a CiliumEgressGatewayPolicy:

apiVersion: cilium.io/v2
kind: CiliumEgressGatewayPolicy
metadata:
  name: webapp-prod-egress-to-payment-gw
spec:
  selectors:
    - podSelector:
        matchLabels:
          app: webapp
          env: prod
          io.kubernetes.pod.namespace: production-apps
  destinationCIDRs:
    - "203.0.113.42/32" # IP of an external payment gateway
  egressGateway:
    nodeSelector:
      matchLabels:
        role: egress-node # Ensure this label is on your designated egress node
    egressIP: "172.16.100.50" # This IP must be configured on 'egress-node'

To apply the policy, save the manifest to a file (e.g., my-egress-policy.yaml) and apply it using kubectl:

kubectl apply -f my-egress-policy.yaml

Verification

To verify that your Egress Gateway policy is working:

Ensure the policy is applied and the gateway node is correctly configured with the egressIP.
Exec into a pod that matches the podSelector and (if applicable) namespaceSelector of your policy.
From within the pod, attempt to connect to an external service whose IP falls within the destinationCIDRs. A common way to check your public-facing IP is:
```
curl -4 ifconfig.me
```

The IP address returned should match the egressIP specified in your policy. If it shows the pod’s IP or the the node IP where the Pod is running, the policy is not being applied correctly for that traffic.

Troubleshooting

Use kubectl describe ciliumegressgatewaypolicy to see the status and events associated with the policy.
Verify network connectivity from the gateway node to the external destinations.
Ensure the egressIP is correctly configured on the gateway node’s interface and is not conflicting with other IPs.
- ip addr show on the gateway node can help verify IP configuration.

Known Limitations

Setting destinationCIDRs to 0.0.0.0/0 (to route all traffic through the egress gateway) causes Pods to stop accessing to the Kubernetes API server. There is a workaround to solve this issue by excluding the Kubernetes API server CIDR from CiliumEgressGatewayPolicy. Please reach out to Cloudfleet support who can assist you with this configuration.