Secrets at runtime: Vault, Kubernetes Secrets, CSI driver, External Secrets and SOPS

Start from where each copy of the secret will live and how a rotation will reach the process. A secret manager such as Vault, AWS Secrets Manager, Google Secret Manager or Azure Key Vault holds the source of truth; the patterns below differ in whether a second copy lands in etcd, whether the value can change without a restart, and what has to be trusted on every node.

Mount from the store when values must not land in etcd or when you want in-place updates; sync into Kubernetes Secrets when applications or Helm charts expect native Secrets and you can encrypt etcd and restrict RBAC. Whatever you pick, authenticate the workload with its own identity (a Kubernetes service account, EKS Pod Identity, GKE Workload Identity Federation or Microsoft Entra Workload ID) rather than a static credential for the secret manager itself, which only moves the problem.

A Kubernetes Secret is base64-encoded, which is encoding, not encryption. By default the API server stores Secrets unencrypted in etcd, so anyone who can read etcd or its backups reads every credential. Encryption at rest fixes that layer only: the API server still returns plaintext to any caller RBAC allows.

Use the kms provider with apiVersion: v2, stable since Kubernetes 1.29. It is envelope encryption: the API server derives a single-use data key for each write and a plugin wraps the seed with a key encryption key held in your cloud KMS or HSM. KMS v1 has been deprecated since 1.28 and is disabled by default from 1.29. The local providers (aescbc, aesgcm, secretbox) keep their key in the configuration file on the control-plane nodes, so anyone who can read that file and an etcd backup has everything. Kubernetes recommends rotating the key encryption key at least every 90 days.

The first provider in the list encrypts new writes; the rest only decrypt. Writing the configuration encrypts nothing that already exists, so rewrite every Secret afterwards (the verification commands below include the command). Managed services do this for you to different degrees: Amazon EKS has envelope-encrypted all Kubernetes API data with KMS v2 by default since March 2025 for clusters on 1.28 or later, while GKE and AKS offer encryption with your own Cloud KMS or Key Vault key as an option you turn on.

# /etc/kubernetes/enc/enc.yaml, passed to kube-apiserver with
#   --encryption-provider-config=/etc/kubernetes/enc/enc.yaml
#   --encryption-provider-config-automatic-reload=true
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      # First provider encrypts new writes: envelope encryption with a key in your KMS.
      - kms:
          apiVersion: v2
          name: cloud-kms # cannot be changed once set
          endpoint: unix:///var/run/kms-plugin/socket.sock
          timeout: 3s
      # Later providers only read. identity lets the API server read Secrets written before
      # encryption was on; remove it once every Secret has been rewritten.
      - identity: {}

RBAC decides who can read Secrets through the API, and its verbs are easy to over-grant. list and watch on Secrets return the contents of every Secret in scope, so a monitoring agent or developer role with list secrets across a namespace can read all of them. Grant get with resourceNames instead, and give people no standing access to production Secrets at all.

Pod creation is an indirect read. Anyone who can create a Pod, or a Deployment, Job or CronJob that creates one, in a namespace can mount any Secret in that namespace and print it. Separate namespaces by trust, keep deploy rights as narrow as Secret rights, and alert on one identity reading many Secrets in a short time. Mark Secrets that never change as immutable: true so they cannot be edited in place.

# The billing API may read exactly one Secret. No list, no watch: either verb returns
# every Secret's value in the namespace.
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: read-billing-db
  namespace: billing
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    resourceNames: ["billing-db"]
    verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: billing-api-reads-billing-db
  namespace: billing
subjects:
  - kind: ServiceAccount
    name: billing-api
    namespace: billing
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: read-billing-db

The Secrets Store CSI Driver is a Kubernetes SIG project that mounts values from an external store into a Pod as files on an in-memory volume, through a provider plugin for AWS, Azure, Google Cloud, Vault and others. The kubelet asks the driver to mount the volume when the Pod starts, and the provider fetches the values with the Pod's own identity, so nothing is written to etcd unless you also ask it to sync a Kubernetes Secret through secretObjects.

Rotation is an alpha feature and off by default. With enableSecretRotation: true in the Helm chart (the --enable-secret-rotation flag), from v1.6.0 the driver re-fetches on the kubelet's periodic republish calls, no more often than rotationPollInterval (2 minutes by default). It updates the files and any synced Secret but never restarts your Pods, so the application must re-read the file. The driver's DaemonSet runs privileged on every node, which is the price of keeping values out of etcd.

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: billing-db
  namespace: billing
spec:
  provider: aws
  parameters:
    region: eu-west-1
    usePodIdentity: "true" # EKS Pod Identity; leave out to use IRSA
    objects: |
      - objectName: "prod/billing/db"
        objectType: "secretsmanager"
        jmesPath:
          - path: "password"
            objectAlias: "db_password"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: billing-api
  namespace: billing
spec:
  replicas: 2
  selector:
    matchLabels:
      app: billing-api
  template:
    metadata:
      labels:
        app: billing-api
    spec:
      serviceAccountName: billing-api # the identity AWS allows to read prod/billing/db
      containers:
        - name: api
          image: registry.example/billing-api:1.4.2
          volumeMounts:
            - name: secrets
              mountPath: /mnt/secrets # the app reads /mnt/secrets/db_password
              readOnly: true
      volumes:
        - name: secrets
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: billing-db

External Secrets Operator reads values from an external store and writes them into ordinary Kubernetes Secrets, which suits applications and charts that already expect a Secret. A SecretStore (one namespace) or ClusterSecretStore (all namespaces) says how to reach the store; an ExternalSecret says which values to copy and how often. The current API is external-secrets.io/v1.

The trade-off is the copy: values now live in etcd as well, so the encryption and RBAC sections above apply in full. Prefer a namespaced SecretStore with its own cloud role, so one team's operator configuration cannot read another team's secrets, and authenticate it with short-lived service account tokens (the jwt block below, using IRSA) rather than an access key stored in yet another Secret; with EKS Pod Identity the store has no auth block and uses the controller's role, so set a per-team role for it to assume. A refreshed Secret updates mounted files, but environment variables still need a restart. HashiCorp's Vault Secrets Operator follows the same sync model for Vault, and can roll Deployments when a value changes (rolloutRestartTargets).

apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: aws-secrets
  namespace: billing
spec:
  provider:
    aws:
      service: SecretsManager
      region: eu-west-1
      auth:
        jwt:
          serviceAccountRef:
            name: billing-secrets-reader # IRSA: annotated with eks.amazonaws.com/role-arn
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: billing-db
  namespace: billing
spec:
  refreshInterval: 15m # how often the operator reads the store again
  secretStoreRef:
    name: aws-secrets
    kind: SecretStore
  target:
    name: billing-db # the Kubernetes Secret the operator creates, owns and updates
    creationPolicy: Owner
  data:
    - secretKey: password
      remoteRef:
        key: prod/billing/db
        property: password

A static database password shared by every replica is valid until someone rotates it. Vault's database secrets engine instead creates a new database user for each request, with a lease: default_ttl sets its lifetime, the holder can renew it up to max_ttl, and Vault drops the user when the lease ends or is revoked. A leaked credential then expires on its own, and vault lease revoke -prefix removes every user a role ever issued in one step.

The Kubernetes auth method lets a Pod trade its service account token for a Vault token bound to the role's policy, so there is no bootstrap secret to manage. Rotate the root credential right after configuring the connection so that only Vault knows it, and grant the role the narrowest SQL privileges the application needs. The commands below were run against Vault 2.1.2 and PostgreSQL 17: each read created a new database user with a one-hour lease, and after vault lease revoke PostgreSQL reported that the role no longer existed. Vault 2.x warns when a Kubernetes auth role has no audience; setting one makes Vault accept only service account tokens issued for that audience, so the Agent must then be given such a projected token.

# 1. Pods log in to Vault with their Kubernetes service account token.
vault auth enable kubernetes
# Vault running in the cluster reviews tokens with its own service account and CA.
vault write auth/kubernetes/config \
    kubernetes_host="https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_SERVICE_PORT"
vault policy write billing-db - <<'EOF'
path "database/creds/billing-api" { capabilities = ["read"] }
EOF
vault write auth/kubernetes/role/billing-api \
    bound_service_account_names=billing-api \
    bound_service_account_namespaces=billing \
    token_policies=billing-db \
    token_ttl=1h

# 2. Every read of database/creds/billing-api creates a new PostgreSQL user with a lease.
vault secrets enable database
vault write database/config/billing \
    plugin_name="postgresql-database-plugin" \
    allowed_roles="billing-api" \
    connection_url="postgresql://{{username}}:{{password}}@db.internal:5432/billing?sslmode=verify-full" \
    username="vault_admin" password="$INITIAL_ADMIN_PASSWORD" \
    password_authentication="scram-sha-256"
vault write -force database/rotate-root/billing # afterwards only Vault knows vault_admin's password
vault write database/roles/billing-api \
    db_name="billing" \
    creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT, INSERT, UPDATE ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
    default_ttl="1h" \
    max_ttl="24h"

# 3. Check by hand, and the incident lever: revoke every user this role ever issued.
vault read database/creds/billing-api # lease_id, lease_duration, username, password
vault lease revoke -prefix database/creds/billing-api

The Vault Agent Injector is a mutating admission webhook, installed by the Vault Helm chart, that adds a Vault Agent init container and sidecar to Pods carrying its annotations. The Agent logs in with the Pod's service account, fetches the secrets and renders them through a template into files under /vault/secrets/ on a shared in-memory volume, so the application needs no Vault client code.

The Agent keeps the files current: it renews renewable leases after two thirds of their duration and fetches a replacement near the end of a lease it cannot renew, then re-renders the file. The application must re-open the file or connection pool when it changes; the vault.hashicorp.com/agent-inject-command-<name> annotation can run a command after each render. For Jobs and CronJobs set vault.hashicorp.com/agent-pre-populate-only: "true" so no sidecar keeps the Pod alive. HashiCorp's comparison of its three Kubernetes integrations is a good guide: the injector for templates and many auth methods, the CSI provider for vendor-neutral mounts, the Secrets Operator for native Kubernetes Secrets.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: billing-api
  namespace: billing
spec:
  replicas: 2
  selector:
    matchLabels:
      app: billing-api
  template:
    metadata:
      labels:
        app: billing-api
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "billing-api" # the Kubernetes auth role
        # Rendered to /vault/secrets/db on an in-memory volume shared with the app.
        vault.hashicorp.com/agent-inject-secret-db: "database/creds/billing-api"
        vault.hashicorp.com/agent-inject-template-db: |
          {{- with secret "database/creds/billing-api" -}}
          postgresql://{{ .Data.username }}:{{ .Data.password }}@db.internal:5432/billing?sslmode=verify-full
          {{- end }}
    spec:
      serviceAccountName: billing-api
      containers:
        - name: api
          image: registry.example/billing-api:1.4.2
          env:
            - name: DATABASE_URL_FILE # a path, not the value: the app re-reads the file when it changes
              value: /vault/secrets/db

On 10 August 2023 HashiCorp announced that future releases of its products would move from the Mozilla Public License 2.0 to the Business Source License 1.1. For Vault that means 1.15.0 and later, and the 1.14.9 and 1.13.13 patch releases onward. The license's additional use grant allows production use, including inside your organization's own platform, unless you offer Vault to third parties as a hosted or embedded product that competes with the vendor's paid versions. Each version converts to MPL 2.0 four years after it was published. IBM completed its acquisition of HashiCorp on 27 February 2025; releases up to 2.1 still name HashiCorp as licensor, and the terms are unchanged. The Vault Helm chart and the Agent Injector (vault-k8s) remain MPL 2.0; the Vault Secrets Operator and Vault CSI provider are under the BSL.

OpenBao is a community fork of Vault's last MPL-licensed code, governed as a sandbox project of the Linux Foundation's OpenSSF and released under MPL 2.0; version 2.7 was current in October 2026. It keeps Vault's API, CLI concepts and Kubernetes auth method, so the commands above carry over with the bao binary, but features added to Vault since the fork, and Vault Enterprise features, may differ. Choose on support, features and license terms, and have legal review the additional use grant if you build a product on Vault.

SOPS, a CNCF sandbox project, encrypts the values in a YAML, JSON, ENV or INI file and leaves the keys readable, so diffs and reviews still work. Each value is encrypted with AES-256-GCM under one data key per file, the key names are bound in as associated data, and a MAC over the whole tree detects added, removed or edited values, including plaintext fields such as a Secret's namespace. The data key is then encrypted to every recipient in .sops.yaml: age or PGP public keys, AWS KMS, Google Cloud KMS, Azure Key Vault or a Vault transit key.

Use one age key or KMS key per environment and restrict encrypted_regex to data and stringData, as Flux expects. The commands below were run with SOPS 3.13.3 and age 1.3.2. Removing a recipient with sops updatekeys does not take back access: the old commits stay decryptable with the old key, so rotate the secret values themselves when someone leaves or a key leaks. Since SOPS 3.9 the commands are sops encrypt, sops decrypt and sops edit.

# Once per environment: an age key pair. The private key goes to the cluster, never to Git.
age-keygen -o prod.agekey # prints the public key, age1...

cat > .sops.yaml <<'EOF'
creation_rules:
  - path_regex: clusters/prod/.*\.yaml$
    encrypted_regex: ^(data|stringData)$ # values only; kind, name and namespace stay readable
    age: age1gxuttyn8k7jvxzmj5523m4npu9fcx8cj8pankq7lytldfdxaeuyq33yhwv
EOF

sops encrypt --in-place clusters/prod/billing-db.yaml # commit only the encrypted file
SOPS_AGE_KEY_FILE=prod.agekey sops decrypt clusters/prod/billing-db.yaml # needs the private key

# Flux decrypts in the cluster with the private key from this Secret (the key must end in .agekey).
kubectl create secret generic sops-age --namespace=flux-system \
  --from-file=age.agekey=prod.agekey

With GitOps, the cluster rather than a person decrypts. Flux's kustomize-controller reads the private key from the sops-age Secret in flux-system and decrypts files during reconciliation, so the plaintext exists only as the Kubernetes Secret it applies. That Secret is now the most valuable object in the cluster: encrypt etcd, restrict it to the controller, and prefer a cloud KMS key that the controller's workload identity may use for decryption only, which leaves an audit trail and needs no key file.

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: billing
  namespace: flux-system
spec:
  interval: 10m
  sourceRef:
    kind: GitRepository
    name: platform
  path: ./clusters/prod
  prune: true
  decryption:
    provider: sops
    secretRef:
      name: sops-age

The three large cloud secret managers version secrets and integrate with workload identity, but only AWS Secrets Manager runs the rotation itself, through managed rotation or a Lambda rotation function; Google Cloud and Azure tell your code when it is time.

Rotating a credential that one user shares across all replicas leaves a short window in which the database has the new password and some clients still send the old one. AWS calls the fix the alternating-users strategy: keep two database users and rotate the one not currently in use, so every client always holds a valid credential. The same idea works anywhere: issue the new credential, move every consumer, then revoke the old one at the issuer.

A rotation is complete only when every running process uses the new value and the issuer rejects the old one. Kubernetes updates the files of a mounted Secret volume after the kubelet's sync period plus its cache delay, but a container that mounts the Secret through subPath never sees the update, and an environment variable keeps its value for the life of the process. The CSI driver and the external-secrets operators do not restart Pods either.

Pick one mechanism per workload and test it. The application re-reads the file on change, or on an authentication failure, then retries once; a controller such as Stakater Reloader, or Vault Secrets Operator's rolloutRestartTargets, rolls the Deployment when the Secret changes; or the Vault Agent runs a command after it re-renders. Keep the old value valid until the slowest consumer has moved, as the rotation-ordering guide at /domains/identity-crypto/secrets/secrets-lifecycle/ explains, and if a value has leaked, revoke first and follow /domains/identity-crypto/secrets/leaked-secret-response/.

Run these checks after setting up a namespace and after every change to its secrets plumbing. The etcd read works only where you operate etcd yourself; on a managed control plane, rely on the provider's encryption setting and audit logs instead.

ns=billing sa=billing-api

# Workloads read only the Secrets they need: expect "no", then "yes".
kubectl auth can-i list secrets -n "$ns" --as="system:serviceaccount:$ns:$sa"
kubectl auth can-i get secrets/billing-db -n "$ns" --as="system:serviceaccount:$ns:$sa"

# On a cluster whose etcd you run: the stored value must start with k8s:enc:kms:v2:
ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key \
  get /registry/secrets/$ns/billing-db | hexdump -C | head -3

# Rewrite every Secret so ones stored before encryption was on get encrypted too.
kubectl get secrets --all-namespaces -o json | kubectl replace -f -

# Which secret versions each CSI-mounted pod currently holds.
kubectl get secretproviderclasspodstatus -n "$ns" -o yaml

# Rotation reached the app: the old credential must now fail at the database.
vault lease revoke -prefix database/creds/billing-api

Sources & further reading

The standards, specifications and documentation behind the guidance above. Each is the evidence for a claim on this page; consult it for detail beyond the defaults and versions listed here.

1. Kubernetes: good practices for Kubernetes Secrets. Kubernetes v1.37.
2. Kubernetes: using a KMS provider for data encryption (KMS v2). Kubernetes v1.37.
3. Kubernetes: encrypting confidential data at rest. Kubernetes v1.37.
4. Kubernetes: Secrets (volume updates, subPath and immutable Secrets). Kubernetes v1.37.
5. AWS: Amazon EKS envelope encrypts all Kubernetes API data by default (March 2025)
6. Secrets Store CSI Driver: auto rotation of mounted contents
7. AWS Secrets and Configuration Provider for the Secrets Store CSI Driver
8. External Secrets Operator: AWS authentication
9. Vault: PostgreSQL database secrets engine
10. Vault: Agent Injector annotations
11. Vault: Kubernetes integrations comparison
12. Vault Agent: renewals and updating secrets
13. HashiCorp adopts the Business Source License (10 August 2023)
14. Vault LICENSE: BSL 1.1 parameters and change date
15. OpenBao: a fork of Vault managed by the Linux Foundation's OpenSSF
16. SOPS documentation: age and encryption details
17. Flux: manage Kubernetes secrets with SOPS
18. AWS Secrets Manager: Lambda rotation functions
19. AWS Secrets Manager: rotation strategies
20. Google Cloud: Secret Manager rotation schedules
21. Microsoft: monitoring Key Vault with Azure Event Grid

Reference information

Tags

Secrets managementKubernetes and containersAI agents and tools

SDLC phase

DesignDevelopmentMaintenance

Role

DevelopersPlatform engineers and SREs

ASVS 5.0.0 requirements

v5.0.0-13.3.1 · v5.0.0-13.3.2 · v5.0.0-13.3.4 · v5.0.0-13.2.1

CAPEC attack patterns

• CAPEC-37 at MITRE
• CAPEC-560 at MITRE

Download this guide as Markdown

Continue exploring