How should you manage secrets in Kubernetes?
Manage Kubernetes secrets in layers: encrypt the cluster data store, restrict who can read Secret objects, and keep the source of truth either encrypted in Git or in an external secret manager instead of in plain manifests. A Kubernetes Secret is only base64-encoded, and by default it is stored unencrypted in etcd. Where a cloud API is involved, the best secret is no secret: workload identity replaces the stored key with a short-lived token.
What a Kubernetes Secret protects and what it does not
A Secret is an API object for small confidential values, up to 1 MiB each. Its data field is base64, which the secrets good-practices guide states plainly “is not an encryption method”. Separating Secrets from ConfigMaps still helps, because RBAC, audit rules and the kubelet treat them differently. The value itself is only as safe as three things around it.
The data store. The Secret documentation warns that Secrets are “stored unencrypted in the API server’s underlying data store (etcd)” unless you configure encryption at rest. An EncryptionConfiguration on the API server encrypts them with a local key (aescbc, aesgcm, secretbox) or through an external key management service with the KMS v2 provider, which has been stable since Kubernetes 1.29. A local key sits in a file on the control plane host, so KMS is the stronger option. Existing Secrets stay unencrypted until they are written again.
RBAC. get on a Secret reads one value, but list and watch read all of them: kubectl get secrets -A -o yaml prints every value the caller can list. The built-in view ClusterRole deliberately excludes Secrets for this reason. See Kubernetes RBAC for how to scope it.
Pod creation. Anyone who can create a Pod, or a Deployment, in a namespace can mount any Secret in that namespace and print it. Namespace boundaries therefore matter more than the permissions on the Secret objects themselves.
Four ways to manage secret values
The open question is where the real value lives and how it reaches the cluster. There are four common patterns.
| Pattern | Source of truth | Tools | Ends up as a Kubernetes Secret | Fits when |
|---|---|---|---|---|
| Encrypted in Git | Ciphertext in the Git repository | Sealed Secrets, SOPS | Yes | GitOps teams without a central secret manager |
| Synced from an external store | Cloud secret manager or Vault | External Secrets Operator | Yes | A secret manager already exists and apps expect Secrets |
| Mounted at runtime | Cloud secret manager or Vault | Secrets Store CSI Driver | Only if you enable sync | Values should stay out of etcd |
| Injected by an agent | HashiCorp Vault | Vault Agent Injector | No | You need dynamic, per-Pod credentials from Vault |
Encrypted in the repository
Sealed Secrets encrypts a Secret with the cluster controller’s public key into a SealedSecret that is safe to commit. Only the controller can decrypt it, so neither developers nor CI hold a decryption key. SOPS encrypts the values inside any YAML or JSON file with age, PGP or a cloud KMS key, and works outside Kubernetes too, but whoever deploys needs the decryption key. The tutorial Store Kubernetes secrets in Git with Sealed Secrets compares the two in more depth and walks through sealing, scopes, key backup and rotation.
Synced from an external store
External Secrets Operator reads values from AWS Secrets Manager, Google Secret Manager, Azure Key Vault, HashiCorp Vault and many other providers, and writes them into a regular Kubernetes Secret. An ExternalSecret names the keys to fetch, and a SecretStore or ClusterSecretStore holds the connection. Applications need no changes. The tutorial Manage Kubernetes secrets with External Secrets Operator shows a full setup with AWS Secrets Manager and OIDC authentication.
Mounted at runtime
The Secrets Store CSI Driver is a DaemonSet that mounts values from an external store as files in the Pods you authorize, through providers for AWS, Azure, Google Cloud, Vault and others. By default nothing is written to etcd. Syncing to a Kubernetes Secret and automatic rotation exist, but the project marks both as alpha.
Injected by an agent
The Vault Agent Injector is a mutating webhook. It adds a Vault Agent init container and sidecar to annotated Pods, authenticates with the Pod’s service account, and renders secrets into a shared in-memory volume at /vault/secrets. It pairs well with Vault’s dynamic secrets, such as database credentials generated per Pod.
Remove the secret where you can
Many secrets in a cluster are cloud credentials: an access key for object storage, a service account key file for a registry. With workload identity federation, the Pod exchanges its own short-lived service account token for temporary cloud credentials, so there is nothing to store, sync or rotate. The same applies to the bootstrap problem of the patterns above: External Secrets Operator and the CSI driver can authenticate to the cloud secret manager through workload identity instead of through another stored key.
How each pattern handles rotation
Rotation has two parts: changing the value at its source, and getting the new value into running Pods.
- Sealed Secrets and SOPS: change the value, encrypt it again and commit. Sealed Secrets also renews its own sealing key every 30 days by default, which does not rotate the values it protects.
- External Secrets Operator: change the value in the external store. The operator refreshes the Kubernetes Secret on its
refreshInterval. - Secrets Store CSI Driver: change the value in the store. Mounted files update only with the alpha rotation feature enabled.
- Vault Agent: the sidecar keeps authenticating and rendering for the life of the Pod, so renewed or dynamic credentials reach the files without a restart.
On the Pod side, Kubernetes updates mounted Secret volumes eventually, after the kubelet’s sync period. A container that mounts a Secret with subPath gets no updates, and environment variables are read once at container start, so those workloads need a rolling restart after every rotation.
A sensible default
For most teams: turn on encryption at rest with KMS where the platform allows it, grant list and watch on Secrets to almost nobody, use workload identity for every cloud API, and pick one pattern for the rest. Choose Sealed Secrets when Git is the source of truth and no secret manager exists. Choose External Secrets Operator when one does. Keep application secrets in their own namespaces and audit reads of Secret objects at the Metadata level, which records who read what without logging the value.
Secrets on Cloudfleet
Cloudfleet Kubernetes Engine (CFKE) stores cluster data, including Secrets, encrypted at rest. On Enterprise clusters, Confidential Control Plane also encrypts every Kubernetes object with keys held in your own HSM or cloud KMS, and every cluster supports keyless access to cloud APIs.