Runtime secrets

Do this

1. Inventory each secret and its consumers

   For every credential, record its owner, the issuer, which workloads use it, where each copy lives and how it rotates. Copies in CI variables, Helm values and .env files count. The list shows which secrets you can replace with workload identity and which must move into a secrets manager; Secrets out of code keeps them out of the repository.

2. Prefer workload identity to stored keys

   When the workload calls a cloud API, let the platform issue it short-lived credentials instead of storing a key: EKS Pod Identity or IAM roles for service accounts, Workload Identity Federation for GKE, or Microsoft Entra Workload ID on AKS. Each maps a Kubernetes service account to a cloud identity, so give that identity only the resources the workload uses, and give each workload its own service account.

3. Keep the rest in a secrets manager

   Store remaining secrets in a manager such as HashiCorp Vault or OpenBao, AWS Secrets Manager, Google Secret Manager or Azure Key Vault, as ASVS 5.0 requirement V13.3.1 asks, and authenticate to it with the workload's own identity, not a static token. Prefer dynamic credentials where possible: Vault's database secrets engine creates a database user per request, with a lease that expires.

4. Deliver secrets as mounted files

   Never put a secret in an image, ARG or ENV line. Mount secrets read-only on an in-memory volume, through the Secrets Store CSI Driver, a Vault Agent sidecar or a Kubernetes Secret volume, rather than environment variables, which other processes can read and logs or crash dumps can capture. If secrets land in Kubernetes Secrets, encrypt etcd at rest, preferably with a KMS v2 provider.

5. Let each workload read only its own secrets

   In Kubernetes, grant get on named Secrets with resourceNames, never list or watch, which return the contents of every Secret in scope. Anyone who can create a pod in a namespace can mount any Secret there, so keep deploy rights as narrow as Secret rights and separate namespaces by trust. Scope secrets manager policies the same way: one path per workload.

6. Make rotation reach the running process

   Pick one mechanism per workload and test it: the application re-reads the mounted file when it changes or after an authentication failure, or a controller restarts the deployment. Environment variables and subPath mounts never change in a running container. Rotate by issuing the new credential, moving every consumer, then revoking the old one at the issuer and proving it fails.

7. Audit access and rehearse revocation

   Turn on the secrets manager's audit log and alert on unusual reads, such as one identity reading many secrets in a short time, which the Kubernetes good-practice guide also suggests. In a test environment, revoke a credential on purpose and confirm the service recovers with a new one, so you know the steps before a leak forces them.

Common mistakes

• Giving the workload a static key to log in to the secrets manager. That only moves the secret one step; authenticate with the platform's workload identity instead.
• Granting list on Secrets to a monitoring agent or developer role. Kubernetes returns the contents of every Secret in scope, not just their names.
• Rotating a value the application reads from an environment variable. The process keeps the old value until it restarts, so either it breaks or the old credential has to stay valid.
• Treating a Kubernetes Secret as encrypted. Base64 is an encoding, and by default the API server stores Secrets unencrypted in etcd, so anyone with etcd or its backups can read them.
• Leaving the node's credentials reachable from pods. On EKS, unless access to the instance metadata service is restricted, a container can also reach the node's IAM role.

How a secret can reach the process

Choose by where each copy of the value lives and how a rotation reaches the running process. Mounting from the store keeps values out of etcd; syncing into Kubernetes Secrets brings the etcd encryption and RBAC duties with it.

Tools and guides

• Secrets at runtime: Vault, Kubernetes Secrets, CSI driver, External Secrets and SOPS

  How a running workload should get its credentials: encrypt Kubernetes Secrets with a KMS v2 provider and lock down RBAC, mount from an external store with the Secrets Store CSI Driver, sync with External Secrets Operator, issue dynamic credentials with Vault and its Agent Injector, keep Git copies encrypted with SOPS, and make rotation reach the process.

• Workload identity federation: short-lived, scoped identities for services and CI

  Trace how a running workload obtains a short-lived identity, write cloud trust policies that match immutable claims instead of names or wildcards, and keep the resulting credential limited by audience and authorization.

• Secrets management: storage, rotation and leaked-secret response

  Follow one application credential from issuance through caching, rotation, revocation and restored backups, prove that its old value stops working, and respond correctly when a secret leaks.

Standards

OWASP SAMM 2

Secure Deployment

Sources

1. OWASP Secrets Management Cheat Sheet
2. Kubernetes: good practices for Kubernetes Secrets. Kubernetes v1.37.
3. Kubernetes: Secrets (volume updates and subPath). Kubernetes v1.37.
4. Kubernetes: using a KMS provider for data encryption (KMS v2). Kubernetes v1.37.
5. Amazon EKS: Pod Identity
6. Google Cloud: Workload Identity Federation for GKE
7. Microsoft Learn: Microsoft Entra Workload ID on AKS
8. Vault: PostgreSQL database secrets engine
9. Secrets Store CSI Driver: auto rotation of mounted contents
10. OWASP ASVS 5.0.0: V13 Configuration (V13.3 Secret Management). Version 5.0.0.
11. OWASP Top 10:2025 A02 Security Misconfiguration. OWASP Top 10:2025.
12. OWASP SAMM: Secure Deployment (Secret Management stream)