Manage Kubernetes secrets with External Secrets Operator

Managing secrets in Kubernetes can quickly become a headache. You might start by copying database passwords into YAML files, which end up in Git. Then you realize that’s not secure, so you base64-encode them (which doesn’t actually encrypt anything). Eventually you’re managing dozens of secrets across multiple environments, manually updating them whenever a password changes, and wondering if you just accidentally pushed credentials to your public repository.

Secret management systems like AWS Secrets Manager, Google Secret Manager, Azure Key Vault, and HashiCorp Vault solve many of these problems by providing a centralized place to store secrets with proper encryption, access controls, and audit logging. But there’s still a gap: how do you get those secrets into Kubernetes without manually copying them over? And how do you authenticate without storing credentials as Kubernetes secrets (which brings you back to the same problem)?

This is where External Secrets Operator comes in. It bridges secret management systems and Kubernetes, automatically syncing secrets into your cluster. More importantly, it uses OIDC authentication so you don’t need to store any credentials in Kubernetes at all. Your cluster authenticates directly using its built-in identity.

In this guide, you’ll set up External Secrets Operator on your CFKE cluster using AWS Secrets Manager as an example. The same approach works with Google Secret Manager, Azure Key Vault, HashiCorp Vault, and other secret management systems with no hardcoded credentials anywhere.

Prerequisites

Before getting started, ensure you have:

  • A running Cloudfleet Kubernetes Engine (CFKE) cluster (see the getting started guide if you need to set one up)
  • kubectl configured to interact with your cluster (see the Cloudfleet CLI configuration guide if needed)
  • Helm installed on your local machine
  • Terraform installed on your local machine
  • AWS account with permissions to create IAM roles and manage Secrets Manager
  • Your CFKE cluster ID

How OIDC authentication works

The crucial part of this setup is that you never need to create or manage AWS credentials in Kubernetes. The authentication happens through OIDC (OpenID Connect) federation, where your CFKE cluster’s built-in identity is trusted by AWS.

This means no secret is needed to manage secrets. You’re not trading one secret management problem for another.

Every pod in your CFKE cluster automatically receives a JWT token that AWS can validate. External Secrets Operator uses this token to authenticate with AWS and fetch secrets. The tokens are short-lived and automatically rotated by Kubernetes.

For a detailed explanation of how OIDC authentication works with CFKE, including the token format and validation process, see the accessing cloud APIs securely documentation.

Step 1: Set up AWS IAM role with Terraform

First, you’ll create the AWS infrastructure needed for OIDC authentication. This includes an OIDC provider, IAM role, and policies that grant access to AWS Secrets Manager.

We’re using Terraform here because it provides a straightforward way to set up the required infrastructure. If you prefer to use the AWS CLI or Console, or need more details about the OIDC setup, refer to the accessing cloud APIs securely documentation.

Use the following Terraform configuration. This assumes your Terraform is already configured with the AWS provider:

variable "cfke_cluster_id" {
  type    = string
  default = "YOUR_CLUSTER_ID"
}

data "aws_caller_identity" "current" {}
data "aws_region" "current" {}

variable "kubernetes_service_account_namespace" {
  type        = string
  description = "Kubernetes service account namespace"
  default     = "external-secrets"
}

variable "kubernetes_service_account_name" {
  type        = string
  description = "Kubernetes service account name"
  default     = "external-secrets"
}

resource "aws_iam_openid_connect_provider" "default" {
  url = "https://api.cloudfleet.ai/v1/clusters/${var.cfke_cluster_id}"

  client_id_list = [
    "https://api.cloudfleet.ai/v1/clusters/${var.cfke_cluster_id}",
  ]
}

data "aws_iam_policy_document" "eso_trust_policy" {
  statement {
    effect = "Allow"

    principals {
      type        = "Federated"
      identifiers = ["arn:aws:iam::${data.aws_caller_identity.current.account_id}:oidc-provider/api.cloudfleet.ai/v1/clusters/${var.cfke_cluster_id}"]
    }

    actions = ["sts:AssumeRoleWithWebIdentity"]

    condition {
      test     = "StringEquals"
      variable = "api.cloudfleet.ai/v1/clusters/${var.cfke_cluster_id}:aud"
      values   = ["https://api.cloudfleet.ai/v1/clusters/${var.cfke_cluster_id}"]
    }

    condition {
      test     = "StringEquals"
      variable = "api.cloudfleet.ai/v1/clusters/${var.cfke_cluster_id}:sub"
      values   = ["system:serviceaccount:${var.kubernetes_service_account_namespace}:${var.kubernetes_service_account_name}"]
    }
  }
}

resource "aws_iam_role" "external_secrets" {
  name               = "external-secrets-role"
  assume_role_policy = data.aws_iam_policy_document.eso_trust_policy.json
}

data "aws_iam_policy_document" "external_secrets_policy" {
  statement {
    effect = "Allow"
    actions = [
      "secretsmanager:ListSecrets",
      "secretsmanager:BatchGetSecretValue"
    ]
    resources = ["*"]
  }

  statement {
    effect = "Allow"
    actions = [
      "secretsmanager:GetResourcePolicy",
      "secretsmanager:GetSecretValue",
      "secretsmanager:DescribeSecret",
      "secretsmanager:ListSecretVersionIds"
    ]
    resources = [
      "arn:aws:secretsmanager:${data.aws_region.current.region}:${data.aws_caller_identity.current.account_id}:secret:*"
    ]
  }
}

resource "aws_iam_role_policy" "external_secrets_policy" {
  name   = "external-secrets-policy"
  role   = aws_iam_role.external_secrets.id
  policy = data.aws_iam_policy_document.external_secrets_policy.json
}

output "external_secrets_iam_role_arn" {
  value = aws_iam_role.external_secrets.arn
}

Important configuration notes:

  • Replace YOUR_CLUSTER_ID with your actual CFKE cluster ID
  • The trust policy ensures only the specific service account in your cluster can assume this role
  • The permissions policy grants access to all secrets in your AWS account (you can restrict this to specific secret prefixes if needed)

Apply the Terraform configuration:

terraform apply

This will create:

  • An OIDC provider for your Cloudfleet cluster
  • An IAM role with trust policy for OIDC authentication
  • IAM policies granting access to AWS Secrets Manager

After applying, Terraform will output the IAM role ARN. Copy this ARN as you’ll need it in the next step:

Outputs:

external_secrets_iam_role_arn = "arn:aws:iam::AWS_ACCOUNT_ID:role/external-secrets-role"

Step 2: Add the Helm repository

Add the External Secrets Operator Helm repository:

helm repo add external-secrets https://charts.external-secrets.io
helm repo update

Step 3: Configure Helm values

Create a helm-values.yaml file with the following configuration:

extraEnv:
  - name: AWS_WEB_IDENTITY_TOKEN_FILE
    value: /var/run/secrets/kubernetes.io/serviceaccount/token
  - name: AWS_ROLE_SESSION_NAME
    value: cfke-session
  - name: AWS_ROLE_ARN
    value: arn:aws:iam::AWS_ACCOUNT_ID:role/external-secrets-role

Important configuration notes:

  • Replace the AWS_ROLE_ARN value with the actual IAM role ARN from the Terraform output in Step 1
  • The AWS_WEB_IDENTITY_TOKEN_FILE points to the service account token that Kubernetes automatically mounts

Step 4: Install External Secrets Operator

Install ESO using Helm with your custom values:

helm install external-secrets \
  external-secrets/external-secrets \
  -n external-secrets \
  --create-namespace \
  -f helm-values.yaml

Verify the installation:

kubectl get pods -n external-secrets

You should see three pods running: the controller, webhook, and cert-controller.

Step 5: Create a test secret in AWS Secrets Manager

Before setting up the External Secrets resources, create a test secret in AWS Secrets Manager:

aws secretsmanager create-secret \
  --name test \
  --secret-string '{"test":"hello-from-aws"}' \
  --region us-west-2

You can also create the secret through the AWS Console if you prefer.

Step 6: Create ESO objects

Now you’ll create the resources that define how secrets should be synced from AWS Secrets Manager.

Create an eso-objects.yaml file:

apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: aws
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-west-2
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: example
  namespace: default
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws
    kind: ClusterSecretStore
  target:
    name: example
    creationPolicy: Owner
  data:
    - secretKey: secret-key-to-be-managed
      remoteRef:
        key: test
        property: test

This configuration creates:

  1. ClusterSecretStore – Defines how to connect to AWS Secrets Manager (region, authentication method)
  2. ExternalSecret – Specifies which AWS secret to sync and how to map it to a Kubernetes secret

Apply the configuration:

kubectl apply -f eso-objects.yaml

Step 7: Verify secret synchronization

Check that the ExternalSecret was created successfully:

kubectl get externalsecrets -n default

You should see the example ExternalSecret with status SecretSynced.

Verify the Kubernetes secret was created:

kubectl get secrets -n default example

View the secret data:

kubectl get secret example -n default -o jsonpath='{.data.secret-key-to-be-managed}' | base64 -d

This should output: hello-from-aws

Step 8: Test automatic synchronization

To verify that secrets automatically sync when changed in AWS, update the secret in AWS Secrets Manager:

aws secretsmanager update-secret \
  --secret-id test \
  --secret-string '{"test":"updated-value"}' \
  --region us-west-2

Wait for the refresh interval (1 hour by default, or force a refresh by deleting and recreating the ExternalSecret):

kubectl delete externalsecret example -n default
kubectl apply -f eso-objects.yaml

Check the updated value:

kubectl get secret example -n default -o jsonpath='{.data.secret-key-to-be-managed}' | base64 -d

You should now see: updated-value

Using secrets in your applications

Now that secrets are automatically synced, you can use them in your applications just like any Kubernetes secret:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: app
        image: my-app:latest
        env:
        - name: SECRET_VALUE
          valueFrom:
            secretKeyRef:
              name: example
              key: secret-key-to-be-managed

When the secret updates in AWS Secrets Manager, External Secrets Operator will automatically update the Kubernetes secret. However, note that pods need to be restarted to pick up the new values unless your application watches for secret changes.

Troubleshooting

If secrets aren’t syncing:

  1. Check ExternalSecret status:

    kubectl describe externalsecret example -n default

  2. Check External Secrets Operator logs:

    kubectl logs -n external-secrets -l app.kubernetes.io/name=external-secrets

  3. Verify IAM role permissions: Ensure the IAM role has the correct trust policy and permissions to access Secrets Manager

  4. Check service account token: Verify the External Secrets Operator pod has the service account token mounted at /var/run/secrets/kubernetes.io/serviceaccount/token

Conclusion

You’ve successfully set up External Secrets Operator on your CFKE cluster with secure OIDC authentication to AWS. This setup provides:

  • Centralized secret management in AWS Secrets Manager
  • Automatic synchronization of secrets into Kubernetes
  • Secure authentication without storing long-lived credentials
  • Audit logging through AWS CloudTrail
  • Automatic secret rotation and refresh

Your applications can now consume secrets from AWS Secrets Manager as native Kubernetes secrets, while maintaining security best practices and centralized secret management.

For advanced configurations including multiple secret stores, secret templating, other cloud providers (Google Secret Manager, Azure Key Vault, HashiCorp Vault), and production considerations, refer to the External Secrets Operator documentation.

Previous
← Install and Configure Istio service mesh
Next
Use persistent volumes with Cloudfleet on Hetzner →
On this page