Accessing cloud APIs securely

Accessing cloud APIs from workloads running in the Cloudfleet Kubernetes Engine (CFKE) is common. For example, you may need to access AWS S3, GCP Cloud Storage, or Azure Blob Storage from your workloads. You can use CFKE’s Kubernetes Service Accounts to authenticate your workloads to access these cloud APIs without hardcoding keys.

Overview

Like standard Kubernetes deployments, CFKE mounts an OIDC token to every pod running in the cluster. By default, this token is mounted at the path /var/run/secrets/kubernetes.io/serviceaccount/token.

The token has the following payload:

{
  "aud": [
    "https://api.cloudfleet.ai/v1/clusters/CLUSTER_ID"
  ],
  "exp": 1762601362,
  "iat": 1731065362,
  "iss": "https://api.cloudfleet.ai/v1/clusters/CLUSTER_ID",
  "kubernetes.io": {
    "namespace": "POD_NAMESPACE",
    "pod": {
      "name": "POD_NAME",
      "uid": "POD_UID"
    },
    "serviceaccount": {
      "name": "SERVICE_ACCOUNT_NAME",
      "uid": "SERVICE_ACCOUNT_UID"
    },
    "warnafter": 1731068969
  },
  "nbf": 1731065362,
  "sub": "system:serviceaccount:SERVICE_ACCOUNT_NAMESPACE:SERVICE_ACCOUNT_NAME"
}

Note that the JWT issuer https://api.cloudfleet.ai/v1/clusters/CLUSTER_ID serves as an OIDC discovery endpoint that third-party services can use to validate this token. This means, if a third-party service has ability to validate and trust external OIDC tokens, it can use the token assigned to every pod in your CFKE cluster for authentication. This is a strong security feature that allows you to authenticate your workloads to access cloud APIs without hardcoded keys.

Creating a Kubernetes Service Account

To configure this feature, you can start by creating a Kubernetes Service Account in your cluster. Each namespace has a default service account, and if you don’t specify a service account in your pod definition, the default one is used. However, it is best practice to create a dedicated service account for your workloads.

An example Kubernetes Service Account definition is as follows:

apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: default
  name: example-application

In the following steps, you will assign this service account to your application pods.

Example: Amazon Web Services (AWS)

You can configure your AWS account to trust the OIDC token issued by CFKE. This way, you can authenticate your workloads to access AWS services without hardcoded keys.

Create an OpenID Connect (OIDC) identity provider in IAM.

Use the AWS CLI to create an OIDC identity provider. Replace CLUSTER_ID with your cluster ID.
```
aws iam create-open-id-connect-provider --url https://api.cloudfleet.ai/v1/clusters/CLUSTER_ID --client-id-list https://api.cloudfleet.ai/v1/clusters/CLUSTER_ID
```
For more information about creating an OIDC provider, see the AWS documentation.

Create an IAM role.

The role must have the following Trust Policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::AWS_ACCOUNT_ID:oidc-provider/api.cloudfleet.ai/v1/clusters/CLUSTER_ID"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "api.cloudfleet.ai/v1/clusters/CLUSTER_ID:aud": "https://api.cloudfleet.ai/v1/clusters/CLUSTER_ID",
                    "api.cloudfleet.ai/v1/clusters/CLUSTER_ID:sub": "system:serviceaccount:KUBERNETES_SERVICE_ACCOUNT_NAMESPACE:KUBERNETES_SERVICE_ACCOUNT_NAME"
                }
            }
        }
    ]
}

Replace AWS_ACCOUNT_ID, CLUSTER_ID, KUBERNETES_SERVICE_ACCOUNT_NAMESPACE, and KUBERNETES_SERVICE_ACCOUNT_NAME with your AWS account ID, CFKE Cluster ID, Kubernetes service account namespace, and Kubernetes service account name, respectively.

Attach an IAM policy to the role. For example, you can use the following command to attach the managed AmazonS3ReadOnlyAccess policy to this IAM role:
```
aws iam attach-role-policy --role-name ROLE_NAME --policy-arn arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess
```
Replace ROLE_NAME with the role name you created in the previous step.

Create a pod that uses the service account.

apiVersion: v1
kind: Pod
metadata:
    name: example-application
    namespace: default
spec:
    serviceAccountName: example-application
    containers:
        -   name: awscli
            image: amazon/aws-cli:latest
            command:
                - "sleep"
                - "infinity"
            env:
                -   name: AWS_WEB_IDENTITY_TOKEN_FILE
                    value: /var/run/secrets/kubernetes.io/serviceaccount/token
                -   name: AWS_ROLE_SESSION_NAME
                    value: cfke-session
                -   name: AWS_ROLE_ARN
                    value: arn:aws:iam::AWS_ACCOUNT_ID:role/ROLE_NAME

Note that AWS_WEB_IDENTITY_TOKEN_FILE, AWS_ROLE_SESSION_NAME, and AWS_ROLE_ARN are supported environment variables by the AWS SDK to automatically assume a role using an OIDC token. Behind the scenes, the AWS SDK will fetch the token from its mount path and use AssumeRoleWithWebIdentity to assume the IAM role we created. These steps can also be implemented in a custom way in your application code.

Verify that the pod can list the AWS S3 buckets in the AWS account.

kubectl exec -n default example-application -- aws s3 ls

Example: Google Cloud platform

On Google Cloud Platform, you can configure GCP to trust the OIDC tokens issued by your Cloudfleet cluster and authenticate your workloads to access GCP services using the Workload Identity Federation.

For that, you should first create a Workload Identity Pool and configure it to trust the OIDC tokens issued by your CFKE cluster. Below you find an example Terraform code to configure this:

variable "cfke_cluster_id" {
  type        = string
  description = "The cluster ID for the CFKE cluster"
  default     = "CLUSTER_ID"
}

resource "google_iam_workload_identity_pool" "cfke" {
  workload_identity_pool_id = "cfke"
  display_name              = "CFKE"
  disabled                  = false
}


resource "google_iam_workload_identity_pool_provider" "cfke" {
  workload_identity_pool_id          = google_iam_workload_identity_pool.cfke.workload_identity_pool_id
  workload_identity_pool_provider_id = "cfke"
  display_name                       = "CFKE"
  attribute_mapping = {
    "google.subject" = "assertion.sub"
  }
  oidc {
    allowed_audiences = [
      "https://api.cloudfleet.ai/v1/clusters/${var.cfke_cluster_id}"
    ]
    issuer_uri = "https://api.cloudfleet.ai/v1/clusters/${var.cfke_cluster_id}"
  }
}

output "cfke_workload_identity_pool_provider-id" {
    value = google_iam_workload_identity_pool.cfke.id
}

Once you create the Workload Identity Pool, you can now refer to it in your IAM policies using direct resource access. For example, you can create a GCP service account and assign it the roles/storage.objectViewer role to allow access to GCP Cloud Storage:

resource "google_storage_bucket_iam_member" "example-bucket" {
    bucket = "example-bucket" # Replace with your bucket name
    role   = "roles/storage.objectViewer"
    member = "principal://iam.googleapis.com/${google_iam_workload_identity_pool.cfke.id}/subject/system:serviceaccount:K8S_NAMESPACE:K8S_SERVICE_ACCOUNT_NAME"
}

Replace K8S_NAMESPACE and K8S_SERVICE_ACCOUNT_NAME with the namespace and name of the Kubernetes service account you created earlier.

You can apply the same principle to other cloud providers or third-party services that support OIDC tokens. The key is to create a trust relationship between the OIDC provider (CFKE) and the cloud provider or third-party service, allowing your workloads to authenticate securely without hardcoded keys.

Previous
← Private container registries

Next
GPU-based workloads →