---
key: "secrets-in-kubernetes-and-runtime"
title: "Secrets at runtime: Vault, Kubernetes Secrets, CSI driver, External Secrets and SOPS"
url: "https://softwaresecurity.com/domains/identity-crypto/secrets/secrets-in-kubernetes-and-runtime/"
api: "https://softwaresecurity.com/api/v1/articles/secrets-in-kubernetes-and-runtime.json"
domain: "identity-crypto"
topic: "Identity \u0026 cryptography"
type: "Guide"
identifiers: ["CWE-798","CWE-522","CWE-312","CWE-732","A02:2025","A04:2025","V13.3.1","V13.3.2","V13.3.4","V13.2.1","CAPEC-37","CAPEC-560"]
tags: ["Secrets management","Kubernetes and containers","AI agents and tools"]
stages: ["design","build","operate"]
roles: ["developer","platform-sre"]
license: "CC-BY-4.0"
license_url: "https://creativecommons.org/licenses/by/4.0/"
code_license: "MIT"
---

# Secrets at runtime: Vault, Kubernetes Secrets, CSI driver, External Secrets and SOPS

How a running workload should get its credentials: encrypt Kubernetes Secrets with a KMS v2 provider and lock down RBAC, mount from an external store with the Secrets Store CSI Driver, sync with External Secrets Operator, issue dynamic credentials with Vault and its Agent Injector, keep Git copies encrypted with SOPS, and make rotation reach the process.

Canonical URL: https://softwaresecurity.com/domains/identity-crypto/secrets/secrets-in-kubernetes-and-runtime/

## In short

Give each workload its own identity and let it fetch short-lived secrets at runtime from a secret manager, instead of baking values into images, Git or long-lived environment variables. On Kubernetes, either mount secrets straight from the external store (Secrets Store CSI Driver or Vault Agent Injector), or sync them into Kubernetes Secrets (External Secrets Operator or Vault Secrets Operator) and then encrypt etcd with a KMS v2 provider and grant `get` on named Secrets only. Keep secrets in Git only when SOPS has encrypted them, and prove that each rotation reaches the running process.

- Kubernetes Secrets are base64-encoded, not encrypted: etcd holds them in plaintext until you configure encryption at rest, preferably a KMS v2 provider (stable since Kubernetes 1.29).
- `list` or `watch` on Secrets returns every value, and anyone who can create a Pod in a namespace can mount any Secret in it.
- Mounting from the external store keeps values out of etcd; syncing into Kubernetes Secrets is simpler for apps but brings the etcd and RBAC duties with it.
- Prefer dynamic, leased credentials such as Vault's database secrets engine, or managed rotation, to a static value with a calendar reminder.
- Mounted files can change in place; environment variables and `subPath` mounts never do, so rotation needs a reload or a rolling restart.
- Vault releases since 1.15 are under the Business Source License (announced 10 August 2023); OpenBao is the MPL 2.0 fork hosted by the Linux Foundation's OpenSSF.

## Recommended defaults

| Setting | Value | Note | Source |
| --- | --- | --- | --- |
| Kubernetes encryption at rest | A `kms` provider with `apiVersion: v2` listed first in the EncryptionConfiguration | KMS v1 is deprecated since 1.28 and disabled by default since 1.29. | Kubernetes: KMS provider for data encryption |
| KMS key encryption key rotation | At least every 90 days | Kubernetes recommends it because you do not control how many writes use each data key. | Kubernetes: KMS provider for data encryption |
| RBAC for workloads | `get` on named Secrets (`resourceNames`); no `list` or `watch` | — | Kubernetes: good practices for Secrets |
| Delivery into the container | Read-only files on an in-memory volume, not environment variables | Files can be updated in place; a `subPath` mount cannot. | — |
| Vault dynamic database credentials | `default_ttl` of about 1 hour, `max_ttl` of 24 hours | The values in Vault's PostgreSQL example; shorter is better if your connection pool reconnects. | Vault: PostgreSQL database secrets engine |
| Secrets Store CSI Driver rotation | `enableSecretRotation: true`; `rotationPollInterval` default 2 minutes | Alpha; off by default. | Secrets Store CSI Driver: auto rotation |
| SOPS for Kubernetes manifests | age recipients; `encrypted_regex: ^(data\|stringData)$`; one key per environment | — | Flux: Manage Kubernetes secrets with SOPS |
| Fastest cloud rotation schedules | AWS Secrets Manager every 4 hours; Google Secret Manager every hour | Azure Key Vault signals `SecretNearExpiry` 30 days before a secret's expiry date. | AWS, Google Cloud and Microsoft documentation |

## Verify it

1. Run `kubectl auth can-i list secrets --as=system:serviceaccount:<namespace>:<account>` for every workload account and confirm each answer is no.
2. Read one Secret straight from etcd and confirm the stored value starts with `k8s:enc:kms:v2:` rather than readable text.
3. Rotate a value in the external store and confirm the mounted file changes within the refresh interval and the application uses it without a deploy, or that the planned restart happens.
4. After rotation, call the database or API with the old credential and confirm it is rejected.
5. Edit a plaintext field such as `metadata.namespace` in a SOPS-encrypted file and confirm `sops decrypt` fails with a MAC mismatch; make CI fail on any unencrypted Secret manifest.
6. Revoke a Vault lease and confirm the database no longer accepts that username.

## What you will learn

- Choose a delivery pattern for a workload and state where each copy of the secret lives.
- Configure KMS v2 encryption at rest and least-privilege RBAC for Kubernetes Secrets.
- Write SecretProviderClass, ExternalSecret, Vault and SOPS configuration that keeps values out of Git and images.
- Prove that a rotation reaches the running process and that the old value stops working.

Audience: Platform engineers and developers who deliver credentials to workloads on Kubernetes or a cloud platform, and reviewers who check how those credentials are stored and rotated.

Before you start:

- Read Kubernetes YAML for Deployments, Roles and Secrets.
- Know the difference between where a secret is stored, how it reaches the process and how it is rotated.
- For the commands: kubectl, and optionally the vault, sops and age CLIs; no cluster is needed for the exercise.

Time: About 36 minutes: 14 to read and 22 for the exercise

## Which delivery pattern should a workload use?

Start from where each copy of the secret will live and how a rotation will reach the process. A secret manager such as Vault, AWS Secrets Manager, Google Secret Manager or Azure Key Vault holds the source of truth; the patterns below differ in whether a second copy lands in etcd, whether the value can change without a restart, and what has to be trusted on every node.

Mount from the store when values must not land in etcd or when you want in-place updates; sync into Kubernetes Secrets when applications or Helm charts expect native Secrets and you can encrypt etcd and restrict RBAC. Whatever you pick, authenticate the workload with its own identity (a Kubernetes service account, EKS Pod Identity, GKE Workload Identity Federation or Microsoft Entra Workload ID) rather than a static credential for the secret manager itself, which only moves the problem.

| Pattern | Where the value lives | How a rotation reaches the app | Watch for |
| --- | --- | --- | --- |
| Kubernetes Secret as an environment variable | etcd, then the process environment | Only when the Pod restarts | Child processes inherit it and crash dumps can include it; it never updates. |
| Kubernetes Secret as a volume | etcd, then an in-memory file | The kubelet rewrites the file after its sync delay; never with `subPath` | etcd encryption and RBAC are still required. |
| Secrets Store CSI Driver | The external store, then an in-memory file in the Pod | Re-fetched on republish when the alpha rotation feature is on | The driver runs privileged on every node; optional sync creates a Kubernetes Secret again. |
| External Secrets Operator or Vault Secrets Operator | The external store, copied into a Kubernetes Secret | The operator's refresh interval, then a file update or a restart | The copy in etcd needs encryption and RBAC; the operator can read every secret its store allows. |
| Vault Agent Injector | Vault, rendered into an in-memory volume | The Agent renews leases and re-renders the file | A sidecar per Pod; new Pods wait for Vault to answer before the app starts. |
| SOPS-encrypted manifest in Git | Git, encrypted; decrypted by Flux or CI into a Kubernetes Secret | A new commit | Whoever holds the age key or KMS decrypt right can read every value in the history. |
| Application calls the secret manager's API | The secret manager, then process memory | The application re-fetches or caches with a TTL | Needs workload identity; cache expiry is the application's job. |

## Encrypt Kubernetes Secrets at rest with a KMS v2 provider

A Kubernetes Secret is base64-encoded, which is encoding, not encryption. By default the API server stores Secrets unencrypted in etcd, so anyone who can read etcd or its backups reads every credential. Encryption at rest fixes that layer only: the API server still returns plaintext to any caller RBAC allows.

Use the `kms` provider with `apiVersion: v2`, stable since Kubernetes 1.29. It is envelope encryption: the API server derives a single-use data key for each write and a plugin wraps the seed with a key encryption key held in your cloud KMS or HSM. KMS v1 has been deprecated since 1.28 and is disabled by default from 1.29. The local providers (`aescbc`, `aesgcm`, `secretbox`) keep their key in the configuration file on the control-plane nodes, so anyone who can read that file and an etcd backup has everything. Kubernetes recommends rotating the key encryption key at least every 90 days.

The first provider in the list encrypts new writes; the rest only decrypt. Writing the configuration encrypts nothing that already exists, so rewrite every Secret afterwards (the verification commands below include the command). Managed services do this for you to different degrees: Amazon EKS has envelope-encrypted all Kubernetes API data with KMS v2 by default since March 2025 for clusters on 1.28 or later, while GKE and AKS offer encryption with your own Cloud KMS or Key Vault key as an option you turn on.

```yaml
# /etc/kubernetes/enc/enc.yaml, passed to kube-apiserver with
#   --encryption-provider-config=/etc/kubernetes/enc/enc.yaml
#   --encryption-provider-config-automatic-reload=true
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      # First provider encrypts new writes: envelope encryption with a key in your KMS.
      - kms:
          apiVersion: v2
          name: cloud-kms # cannot be changed once set
          endpoint: unix:///var/run/kms-plugin/socket.sock
          timeout: 3s
      # Later providers only read. identity lets the API server read Secrets written before
      # encryption was on; remove it once every Secret has been rewritten.
      - identity: {}

```

## Grant get on named Secrets, never list or watch

RBAC decides who can read Secrets through the API, and its verbs are easy to over-grant. `list` and `watch` on Secrets return the contents of every Secret in scope, so a monitoring agent or developer role with `list secrets` across a namespace can read all of them. Grant `get` with `resourceNames` instead, and give people no standing access to production Secrets at all.

Pod creation is an indirect read. Anyone who can create a Pod, or a Deployment, Job or CronJob that creates one, in a namespace can mount any Secret in that namespace and print it. Separate namespaces by trust, keep deploy rights as narrow as Secret rights, and alert on one identity reading many Secrets in a short time. Mark Secrets that never change as `immutable: true` so they cannot be edited in place.

```yaml
# The billing API may read exactly one Secret. No list, no watch: either verb returns
# every Secret's value in the namespace.
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: read-billing-db
  namespace: billing
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    resourceNames: ["billing-db"]
    verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: billing-api-reads-billing-db
  namespace: billing
subjects:
  - kind: ServiceAccount
    name: billing-api
    namespace: billing
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: read-billing-db

```

## Mount secrets from the store with the Secrets Store CSI Driver

The Secrets Store CSI Driver is a Kubernetes SIG project that mounts values from an external store into a Pod as files on an in-memory volume, through a provider plugin for AWS, Azure, Google Cloud, Vault and others. The kubelet asks the driver to mount the volume when the Pod starts, and the provider fetches the values with the Pod's own identity, so nothing is written to etcd unless you also ask it to sync a Kubernetes Secret through `secretObjects`.

Rotation is an alpha feature and off by default. With `enableSecretRotation: true` in the Helm chart (the `--enable-secret-rotation` flag), from v1.6.0 the driver re-fetches on the kubelet's periodic republish calls, no more often than `rotationPollInterval` (2 minutes by default). It updates the files and any synced Secret but never restarts your Pods, so the application must re-read the file. The driver's DaemonSet runs privileged on every node, which is the price of keeping values out of etcd.

```yaml
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: billing-db
  namespace: billing
spec:
  provider: aws
  parameters:
    region: eu-west-1
    usePodIdentity: "true" # EKS Pod Identity; leave out to use IRSA
    objects: |
      - objectName: "prod/billing/db"
        objectType: "secretsmanager"
        jmesPath:
          - path: "password"
            objectAlias: "db_password"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: billing-api
  namespace: billing
spec:
  replicas: 2
  selector:
    matchLabels:
      app: billing-api
  template:
    metadata:
      labels:
        app: billing-api
    spec:
      serviceAccountName: billing-api # the identity AWS allows to read prod/billing/db
      containers:
        - name: api
          image: registry.example/billing-api:1.4.2
          volumeMounts:
            - name: secrets
              mountPath: /mnt/secrets # the app reads /mnt/secrets/db_password
              readOnly: true
      volumes:
        - name: secrets
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: billing-db

```

## Sync into Kubernetes Secrets with External Secrets Operator

External Secrets Operator reads values from an external store and writes them into ordinary Kubernetes Secrets, which suits applications and charts that already expect a Secret. A `SecretStore` (one namespace) or `ClusterSecretStore` (all namespaces) says how to reach the store; an `ExternalSecret` says which values to copy and how often. The current API is `external-secrets.io/v1`.

The trade-off is the copy: values now live in etcd as well, so the encryption and RBAC sections above apply in full. Prefer a namespaced `SecretStore` with its own cloud role, so one team's operator configuration cannot read another team's secrets, and authenticate it with short-lived service account tokens (the `jwt` block below, using IRSA) rather than an access key stored in yet another Secret; with EKS Pod Identity the store has no `auth` block and uses the controller's role, so set a per-team `role` for it to assume. A refreshed Secret updates mounted files, but environment variables still need a restart. HashiCorp's Vault Secrets Operator follows the same sync model for Vault, and can roll Deployments when a value changes (`rolloutRestartTargets`).

```yaml
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: aws-secrets
  namespace: billing
spec:
  provider:
    aws:
      service: SecretsManager
      region: eu-west-1
      auth:
        jwt:
          serviceAccountRef:
            name: billing-secrets-reader # IRSA: annotated with eks.amazonaws.com/role-arn
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: billing-db
  namespace: billing
spec:
  refreshInterval: 15m # how often the operator reads the store again
  secretStoreRef:
    name: aws-secrets
    kind: SecretStore
  target:
    name: billing-db # the Kubernetes Secret the operator creates, owns and updates
    creationPolicy: Owner
  data:
    - secretKey: password
      remoteRef:
        key: prod/billing/db
        property: password

```

## Issue short-lived database credentials with Vault

A static database password shared by every replica is valid until someone rotates it. Vault's database secrets engine instead creates a new database user for each request, with a lease: `default_ttl` sets its lifetime, the holder can renew it up to `max_ttl`, and Vault drops the user when the lease ends or is revoked. A leaked credential then expires on its own, and `vault lease revoke -prefix` removes every user a role ever issued in one step.

The Kubernetes auth method lets a Pod trade its service account token for a Vault token bound to the role's policy, so there is no bootstrap secret to manage. Rotate the root credential right after configuring the connection so that only Vault knows it, and grant the role the narrowest SQL privileges the application needs. The commands below were run against Vault 2.1.2 and PostgreSQL 17: each read created a new database user with a one-hour lease, and after `vault lease revoke` PostgreSQL reported that the role no longer existed. Vault 2.x warns when a Kubernetes auth role has no `audience`; setting one makes Vault accept only service account tokens issued for that audience, so the Agent must then be given such a projected token.

```bash
# 1. Pods log in to Vault with their Kubernetes service account token.
vault auth enable kubernetes
# Vault running in the cluster reviews tokens with its own service account and CA.
vault write auth/kubernetes/config \
    kubernetes_host="https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_SERVICE_PORT"
vault policy write billing-db - <<'EOF'
path "database/creds/billing-api" { capabilities = ["read"] }
EOF
vault write auth/kubernetes/role/billing-api \
    bound_service_account_names=billing-api \
    bound_service_account_namespaces=billing \
    token_policies=billing-db \
    token_ttl=1h

# 2. Every read of database/creds/billing-api creates a new PostgreSQL user with a lease.
vault secrets enable database
vault write database/config/billing \
    plugin_name="postgresql-database-plugin" \
    allowed_roles="billing-api" \
    connection_url="postgresql://{{username}}:{{password}}@db.internal:5432/billing?sslmode=verify-full" \
    username="vault_admin" password="$INITIAL_ADMIN_PASSWORD" \
    password_authentication="scram-sha-256"
vault write -force database/rotate-root/billing # afterwards only Vault knows vault_admin's password
vault write database/roles/billing-api \
    db_name="billing" \
    creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT, INSERT, UPDATE ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
    default_ttl="1h" \
    max_ttl="24h"

# 3. Check by hand, and the incident lever: revoke every user this role ever issued.
vault read database/creds/billing-api # lease_id, lease_duration, username, password
vault lease revoke -prefix database/creds/billing-api

```

## Deliver Vault secrets with the Vault Agent Injector

The Vault Agent Injector is a mutating admission webhook, installed by the Vault Helm chart, that adds a Vault Agent init container and sidecar to Pods carrying its annotations. The Agent logs in with the Pod's service account, fetches the secrets and renders them through a template into files under `/vault/secrets/` on a shared in-memory volume, so the application needs no Vault client code.

The Agent keeps the files current: it renews renewable leases after two thirds of their duration and fetches a replacement near the end of a lease it cannot renew, then re-renders the file. The application must re-open the file or connection pool when it changes; the `vault.hashicorp.com/agent-inject-command-<name>` annotation can run a command after each render. For Jobs and CronJobs set `vault.hashicorp.com/agent-pre-populate-only: "true"` so no sidecar keeps the Pod alive. HashiCorp's comparison of its three Kubernetes integrations is a good guide: the injector for templates and many auth methods, the CSI provider for vendor-neutral mounts, the Secrets Operator for native Kubernetes Secrets.

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: billing-api
  namespace: billing
spec:
  replicas: 2
  selector:
    matchLabels:
      app: billing-api
  template:
    metadata:
      labels:
        app: billing-api
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "billing-api" # the Kubernetes auth role
        # Rendered to /vault/secrets/db on an in-memory volume shared with the app.
        vault.hashicorp.com/agent-inject-secret-db: "database/creds/billing-api"
        vault.hashicorp.com/agent-inject-template-db: |
          {{- with secret "database/creds/billing-api" -}}
          postgresql://{{ .Data.username }}:{{ .Data.password }}@db.internal:5432/billing?sslmode=verify-full
          {{- end }}
    spec:
      serviceAccountName: billing-api
      containers:
        - name: api
          image: registry.example/billing-api:1.4.2
          env:
            - name: DATABASE_URL_FILE # a path, not the value: the app re-reads the file when it changes
              value: /vault/secrets/db

```

## Vault, OpenBao and the license change

On 10 August 2023 HashiCorp announced that future releases of its products would move from the Mozilla Public License 2.0 to the Business Source License 1.1. For Vault that means 1.15.0 and later, and the 1.14.9 and 1.13.13 patch releases onward. The license's additional use grant allows production use, including inside your organization's own platform, unless you offer Vault to third parties as a hosted or embedded product that competes with the vendor's paid versions. Each version converts to MPL 2.0 four years after it was published. IBM completed its acquisition of HashiCorp on 27 February 2025; releases up to 2.1 still name HashiCorp as licensor, and the terms are unchanged. The Vault Helm chart and the Agent Injector (vault-k8s) remain MPL 2.0; the Vault Secrets Operator and Vault CSI provider are under the BSL.

OpenBao is a community fork of Vault's last MPL-licensed code, governed as a sandbox project of the Linux Foundation's OpenSSF and released under MPL 2.0; version 2.7 was current in October 2026. It keeps Vault's API, CLI concepts and Kubernetes auth method, so the commands above carry over with the `bao` binary, but features added to Vault since the fork, and Vault Enterprise features, may differ. Choose on support, features and license terms, and have legal review the additional use grant if you build a product on Vault.

## Keep secrets in Git only encrypted, with SOPS and age

SOPS, a CNCF sandbox project, encrypts the values in a YAML, JSON, ENV or INI file and leaves the keys readable, so diffs and reviews still work. Each value is encrypted with AES-256-GCM under one data key per file, the key names are bound in as associated data, and a MAC over the whole tree detects added, removed or edited values, including plaintext fields such as a Secret's namespace. The data key is then encrypted to every recipient in `.sops.yaml`: age or PGP public keys, AWS KMS, Google Cloud KMS, Azure Key Vault or a Vault transit key.

Use one age key or KMS key per environment and restrict `encrypted_regex` to `data` and `stringData`, as Flux expects. The commands below were run with SOPS 3.13.3 and age 1.3.2. Removing a recipient with `sops updatekeys` does not take back access: the old commits stay decryptable with the old key, so rotate the secret values themselves when someone leaves or a key leaks. Since SOPS 3.9 the commands are `sops encrypt`, `sops decrypt` and `sops edit`.

```bash
# Once per environment: an age key pair. The private key goes to the cluster, never to Git.
age-keygen -o prod.agekey # prints the public key, age1...

cat > .sops.yaml <<'EOF'
creation_rules:
  - path_regex: clusters/prod/.*\.yaml$
    encrypted_regex: ^(data|stringData)$ # values only; kind, name and namespace stay readable
    age: age1gxuttyn8k7jvxzmj5523m4npu9fcx8cj8pankq7lytldfdxaeuyq33yhwv
EOF

sops encrypt --in-place clusters/prod/billing-db.yaml # commit only the encrypted file
SOPS_AGE_KEY_FILE=prod.agekey sops decrypt clusters/prod/billing-db.yaml # needs the private key

# Flux decrypts in the cluster with the private key from this Secret (the key must end in .agekey).
kubectl create secret generic sops-age --namespace=flux-system \
  --from-file=age.agekey=prod.agekey

```

## Let Flux decrypt SOPS files inside the cluster

With GitOps, the cluster rather than a person decrypts. Flux's kustomize-controller reads the private key from the `sops-age` Secret in `flux-system` and decrypts files during reconciliation, so the plaintext exists only as the Kubernetes Secret it applies. That Secret is now the most valuable object in the cluster: encrypt etcd, restrict it to the controller, and prefer a cloud KMS key that the controller's workload identity may use for decryption only, which leaves an audit trail and needs no key file.

```yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: billing
  namespace: flux-system
spec:
  interval: 10m
  sourceRef:
    kind: GitRepository
    name: platform
  path: ./clusters/prod
  prune: true
  decryption:
    provider: sops
    secretRef:
      name: sops-age

```

## How do cloud secret managers rotate secrets?

The three large cloud secret managers version secrets and integrate with workload identity, but only AWS Secrets Manager runs the rotation itself, through managed rotation or a Lambda rotation function; Google Cloud and Azure tell your code when it is time.

Rotating a credential that one user shares across all replicas leaves a short window in which the database has the new password and some clients still send the old one. AWS calls the fix the alternating-users strategy: keep two database users and rotate the one not currently in use, so every client always holds a valid credential. The same idea works anywhere: issue the new credential, move every consumer, then revoke the old one at the issuer.

| Service | What performs the rotation | Schedule | How consumers get the new value |
| --- | --- | --- | --- |
| AWS Secrets Manager | Managed rotation for services such as Amazon RDS, or a Lambda function that runs `createSecret`, `setSecret`, `testSecret` and `finishSecret` | `rate()` or `cron()` expressions, as often as every 4 hours, inside a rotation window | Staging labels: the new version is `AWSPENDING` until tested, then `AWSCURRENT`; `AWSPREVIOUS` keeps the last good version |
| Google Cloud Secret Manager | Your code: at `next_rotation_time` it publishes a `SECRET_ROTATE` message to Pub/Sub topics | `rotation_period` of at least 1 hour | Your subscriber adds a version; consumers read `latest` or a pinned version |
| Azure Key Vault | Your code, triggered by Event Grid events | `Microsoft.KeyVault.SecretNearExpiry` fires 30 days before the expiry date you set | A function or job writes a new secret version |

## Make each rotation reach the running process

A rotation is complete only when every running process uses the new value and the issuer rejects the old one. Kubernetes updates the files of a mounted Secret volume after the kubelet's sync period plus its cache delay, but a container that mounts the Secret through `subPath` never sees the update, and an environment variable keeps its value for the life of the process. The CSI driver and the external-secrets operators do not restart Pods either.

Pick one mechanism per workload and test it. The application re-reads the file on change, or on an authentication failure, then retries once; a controller such as Stakater Reloader, or Vault Secrets Operator's `rolloutRestartTargets`, rolls the Deployment when the Secret changes; or the Vault Agent runs a command after it re-renders. Keep the old value valid until the slowest consumer has moved, as the rotation-ordering guide at /domains/identity-crypto/secrets/secrets-lifecycle/ explains, and if a value has leaked, revoke first and follow /domains/identity-crypto/secrets/leaked-secret-response/.

## Verify encryption, access and rotation

Run these checks after setting up a namespace and after every change to its secrets plumbing. The etcd read works only where you operate etcd yourself; on a managed control plane, rely on the provider's encryption setting and audit logs instead.

```bash
ns=billing sa=billing-api

# Workloads read only the Secrets they need: expect "no", then "yes".
kubectl auth can-i list secrets -n "$ns" --as="system:serviceaccount:$ns:$sa"
kubectl auth can-i get secrets/billing-db -n "$ns" --as="system:serviceaccount:$ns:$sa"

# On a cluster whose etcd you run: the stored value must start with k8s:enc:kms:v2:
ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key \
  get /registry/secrets/$ns/billing-db | hexdump -C | head -3

# Rewrite every Secret so ones stored before encryption was on get encrypted too.
kubectl get secrets --all-namespaces -o json | kubectl replace -f -

# Which secret versions each CSI-mounted pod currently holds.
kubectl get secretproviderclasspodstatus -n "$ns" -o yaml

# Rotation reached the app: the old credential must now fail at the database.
vault lease revoke -prefix database/creds/billing-api

```

## Guided practice

You are reviewing a namespace before it goes to production. Find every path by which the database password can be read or outlive a rotation, then write the configuration that closes them. Produce the YAML and commands, not just a list of findings.

### Supplied material

```text
Cluster: self-managed Kubernetes 1.36; kube-apiserver runs without --encryption-provider-config.
Namespace billing:
  Secret billing-db (type Opaque) with key password, created by a CI job from a value in the CI variables.
  Deployment billing-api: env DB_PASSWORD from secretKeyRef billing-db/password; serviceAccountName billing-api.
  Deployment report-worker: mounts billing-db with subPath password at /etc/app/db-password.
  Role dev-read: apiGroups [""], resources [secrets, pods, pods/log], verbs [get, list, watch]; bound to group billing-devs.
  Role deployer: apps/deployments create, patch; bound to the CI service account and to group billing-devs.
Policy: the database password rotates every 30 days through AWS Secrets Manager (secret prod/billing/db).
```

### 1. List the read paths

Name every identity or component that can read the password today, and how.

Worked answer: Anyone who can read etcd or its backups (no encryption at rest); group billing-devs through list and watch on secrets, which return values; billing-devs and the CI account through creating or patching a Deployment that mounts or prints the Secret; and the billing-api process environment, which child processes and crash dumps inherit. The CI variable is a further copy outside the cluster.

### 2. Write the encryption configuration

Write the `EncryptionConfiguration` and the two follow-up commands that make existing data encrypted.

Worked answer: An `EncryptionConfiguration` for `resources: [secrets]` with a `kms` provider (`apiVersion: v2`, a name, a unix socket endpoint) first and `identity: {}` last; set `--encryption-provider-config` on every kube-apiserver and restart; then run `kubectl get secrets --all-namespaces -o json | kubectl replace -f -` and confirm in etcd that the value starts with `k8s:enc:kms:v2:`. Remove `identity` once nothing unencrypted remains.

### 3. Rewrite the RBAC

Replace dev-read and deployer so that billing-devs can debug without reading secret values.

Worked answer: dev-read keeps `pods` and `pods/log` with `get`, `list`, `watch` but drops `secrets` entirely. billing-api gets its own Role with `get` on `resourceNames: [billing-db]`. Deployment create and patch move to the CI account only, behind review, because the ability to deploy a Pod is the ability to read every Secret in the namespace.

### 4. Make rotation reach both workloads

Decide what happens to billing-api and report-worker when the password rotates, and fix each.

Worked answer: Neither sees the new value: billing-api reads an environment variable and report-worker uses a `subPath` mount, which the kubelet never updates. Mount the Secret as a whole volume without `subPath` (or use the CSI driver or External Secrets Operator), have both re-read the file on change or on an authentication failure, or add a controller that restarts them; until then the 30-day rotation breaks them or forces the old password to stay valid.

### 5. Remove the copy in CI

Write the `ExternalSecret` or `SecretProviderClass` that replaces the CI job, and say what happens to the CI variable.

Worked answer: An `ExternalSecret` (`external-secrets.io/v1`) reading `prod/billing/db` property `password` into Secret `billing-db` with a `refreshInterval`, through a namespaced `SecretStore` that authenticates with a service account (IRSA or Pod Identity); or a `SecretProviderClass` mounting it directly. Delete the CI variable and rotate the password once, because the old value has lived in CI logs and settings.

Deliverable: An `EncryptionConfiguration`, two Roles and a RoleBinding, an `ExternalSecret` or `SecretProviderClass`, and a short note per workload saying how a rotated password reaches it, each checked with `kubectl auth can-i` or a dry-run apply.

Scope: The cluster and its objects are described, not provided; check your YAML with `kubectl apply --dry-run=server` against a test cluster. The exercise does not cover node compromise, where an attacker on a node can read the Secrets of Pods scheduled there.

Articles and teaching material: CC BY 4.0. Original executable code: MIT. Attribution: Software Security.

## Sources

- [Kubernetes: good practices for Kubernetes Secrets](https://kubernetes.io/docs/concepts/security/secrets-good-practices/). Kubernetes v1.37.
- [Kubernetes: using a KMS provider for data encryption (KMS v2)](https://kubernetes.io/docs/tasks/administer-cluster/kms-provider/). Kubernetes v1.37.
- [Kubernetes: encrypting confidential data at rest](https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/). Kubernetes v1.37.
- [Kubernetes: Secrets (volume updates, subPath and immutable Secrets)](https://kubernetes.io/docs/concepts/configuration/secret/). Kubernetes v1.37.
- [AWS: Amazon EKS envelope encrypts all Kubernetes API data by default (March 2025)](https://aws.amazon.com/about-aws/whats-new/2025/03/amazon-eks-envelope-encrypts-kubernetes-api-data-default/)
- [Secrets Store CSI Driver: auto rotation of mounted contents](https://secrets-store-csi-driver.sigs.k8s.io/topics/secret-auto-rotation.html)
- [AWS Secrets and Configuration Provider for the Secrets Store CSI Driver](https://github.com/aws/secrets-store-csi-driver-provider-aws)
- [External Secrets Operator: AWS authentication](https://external-secrets.io/latest/provider/aws-access/)
- [Vault: PostgreSQL database secrets engine](https://developer.hashicorp.com/vault/docs/secrets/databases/postgresql)
- [Vault: Agent Injector annotations](https://developer.hashicorp.com/vault/docs/deploy/kubernetes/injector/annotations)
- [Vault: Kubernetes integrations comparison](https://developer.hashicorp.com/vault/docs/deploy/kubernetes/comparisons)
- [Vault Agent: renewals and updating secrets](https://developer.hashicorp.com/vault/docs/agent-and-proxy/agent/template)
- [HashiCorp adopts the Business Source License (10 August 2023)](https://www.hashicorp.com/blog/hashicorp-adopts-business-source-license)
- [Vault LICENSE: BSL 1.1 parameters and change date](https://github.com/hashicorp/vault/blob/main/LICENSE)
- [OpenBao: a fork of Vault managed by the Linux Foundation's OpenSSF](https://openbao.org/)
- [SOPS documentation: age and encryption details](https://getsops.io/docs/)
- [Flux: manage Kubernetes secrets with SOPS](https://fluxcd.io/flux/guides/mozilla-sops/)
- [AWS Secrets Manager: Lambda rotation functions](https://docs.aws.amazon.com/secretsmanager/latest/userguide/rotate-secrets_lambda-functions.html)
- [AWS Secrets Manager: rotation strategies](https://docs.aws.amazon.com/secretsmanager/latest/userguide/rotation-strategy.html)
- [Google Cloud: Secret Manager rotation schedules](https://docs.cloud.google.com/secret-manager/docs/secret-rotation)
- [Microsoft: monitoring Key Vault with Azure Event Grid](https://learn.microsoft.com/en-us/azure/key-vault/general/event-grid-overview)