Authenticate workloads and authorize their service actions

Mutual TLS can authenticate both ends of a connection and protect traffic. It does not determine whether the caller may invoke an administrative method or read another tenant's record. A service mesh adds infrastructure-level identity and policy, while application logic still owns business permissions.

Map the identity issuance path: which workload attributes are attested, which identity is issued, how trust bundles are distributed and who can change those mappings. SPIFFE provides a model for workload identifiers and identity documents, which the workload identity models guide at /domains/identity-crypto/identity/workload-identity-models/ compares with cloud-native options; deployments must still protect the attestation and issuance components. A compromised issuer can undermine every policy that trusts its identities.

The billing service trusts certificates from a particular workload trust domain. Both reports and support hold valid identities from that domain, but only reports may invoke ExportInvoice. If billing accepts any certificate from the domain as authorization for that method, support inherits a capability it was never intended to have. Verify the peer’s issued identity against the service/method rule after validating the transport credentials.

Even reports must not choose arbitrary tenants. In this example its capability is restricted to North, and the application loads the invoice’s tenant from authoritative storage. The required checks are therefore transport peer identity, method permission and object relationship. An mTLS access log proving reports connected successfully does not establish which tenant the returned invoice belonged to.

Now consider a function fed by a queue. For SQS, Lambda itself polls the queue through an event source mapping, using the function's execution role, so the invocation control is who may create or change that mapping and which queue the role may read. For SNS, S3 or EventBridge the service invokes the function through a resource-based permission, which should name the source with aws:SourceArn. Either way, invocation authority is distinct from the execution role's permission to read storage. A broad execution role can expose unrelated data even if invocation is tightly restricted, and a narrow execution role does not stop an unapproved source from consuming concurrency. Review both directions, event validation and duplicate delivery.

The supplied reused-worker example also shows why request context must remain local. A function instance can process different invocations over time. Saving North’s token or tenant in mutable process-global state and reusing it when a later event omits a field can cross an invocation boundary. Reject missing required context and derive each invocation’s authority from its authenticated event contract; reuse immutable clients, not one caller’s mutable authorization state.

IDENTITIES (synthetic SPIFFE-style example)
spiffe://example.test/ns/prod/sa/reports
spiffe://example.test/ns/prod/sa/support
Policy: reports may ExportInvoice for tenant North; support may GetHealth only
Invoice I7 tenant=North; I8 tenant=South
Function F: approved queue Q may invoke; execution role may read north/* only

Start with a call inventory and grant only required source-to-destination operations. In a mesh, examine both authentication mode and authorization policy. A permissive migration mode may accept plaintext even when some traffic uses mTLS. Check bypass paths through direct listeners, host networking, excluded ports or workloads that do not participate in the mesh.

Protect policy administration and define namespace or trust-domain boundaries. Do not equate possession of any valid certificate with membership in the intended application. Test identity rotation, expired credentials and trust-bundle changes with long-lived connections. Operators need a recovery procedure that restores the control plane without permanently allowing unauthenticated service traffic.

The Istio policies below were tested with Istio 1.31.1 in sidecar mode on a kind cluster. In the default PERMISSIVE mode with no policy, reports, support and a pod without a sidecar could all export invoices. With the policies applied, reports could export but not read /health, support could read /health but not export, every denial was a 403 from the proxy, and the pod without a sidecar could no longer connect. The first version of the policy used /invoices/*/export, which Istio does not treat as a wildcard because * works only at the start or end of a path, so it denied reports too; the {*} template matches one segment, and /invoices/I7/extra/export stayed denied. Policy changes took several seconds to reach the proxies, so test after they propagate.

# Every workload in billing accepts only mutual TLS (no plaintext fallback)
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: billing
spec:
  mtls:
    mode: STRICT
---
# Allow only these calls to billing; anything not listed is denied with 403
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: billing-callers
  namespace: billing
spec:
  selector:
    matchLabels: {app: billing}
  action: ALLOW
  rules:
    - from:
        - source:
            principals: ["cluster.local/ns/reports/sa/reports"]
      to:
        - operation:
            methods: ["POST"]
            paths: ["/invoices/{*}/export"] # {*} is one path segment; * alone works only at either end
    - from:
        - source:
            principals: ["cluster.local/ns/support/sa/support"]
      to:
        - operation:
            methods: ["GET"]
            paths: ["/health"]

A function has at least two permission questions: who may invoke it, and what its execution identity may access. Review event-source permissions, resource policies and downstream roles separately. Validate event shape, source context and the requested business action; a well-formed queue message is not automatically an authorized instruction.

Use narrow execution roles, bounded concurrency and the platform's documented retry semantics. Do not retain one user's sensitive mutable state in a reused execution environment where later invocations could receive it. Make duplicate events idempotent, as the bounded background work guide at /domains/runtime/availability/bounded-background-work/ shows, and prevent recursive triggers from generating unbounded work. Identity federation reduces static credentials, but its trust conditions still need exact scoping; the cloud IAM trust guide at /domains/runtime/cloud-posture/cloud-iam-trust/ covers them.

The Terraform below scopes both directions for the export function: an SQS event source mapping with partial batch responses and a concurrency cap, an SNS permission limited to one topic and account, and an execution role that can read only its queue and the north prefix. Checkov rule CKV_AWS_364 passed it and failed the same permission without source_arn, which would let any SNS topic in any account invoke the function.

# Invocation: an SQS trigger is an event source mapping that Lambda polls with the function's
# execution role. SNS, S3 and EventBridge invoke through a resource-based permission instead.
resource "aws_lambda_event_source_mapping" "exports_queue" {
  event_source_arn        = "arn:aws:sqs:eu-west-1:111122223333:exports"
  function_name           = "export-north"
  batch_size              = 10
  function_response_types = ["ReportBatchItemFailures"]
  scaling_config {
    maximum_concurrency = 5
  }
}

resource "aws_lambda_permission" "alerts_topic" {
  statement_id   = "AllowAlertsTopic"
  action         = "lambda:InvokeFunction"
  function_name  = "export-north"
  principal      = "sns.amazonaws.com"
  source_arn     = "arn:aws:sns:eu-west-1:111122223333:export-alerts" # only this topic
  source_account = "111122223333"
}

# Execution: the role reads its own queue and one tenant prefix, nothing else
data "aws_iam_policy_document" "export_north_execution" {
  statement {
    actions   = ["sqs:ReceiveMessage", "sqs:DeleteMessage", "sqs:GetQueueAttributes"]
    resources = ["arn:aws:sqs:eu-west-1:111122223333:exports"]
  }
  statement {
    actions   = ["s3:GetObject"]
    resources = ["arn:aws:s3:::exports/north/*"]
  }
}

resource "aws_iam_role_policy" "export_north_execution" {
  role   = "export-north-execution"
  policy = data.aws_iam_policy_document.export_north_execution.json
}

The handler below reads the tenant from every message and refuses a message without one, instead of falling back to a value a previous invocation left in a global. It reuses the S3 client, which holds no caller authority, and returns rejected messages as batch item failures so they are retried and then reach the dead-letter queue. In its tests, a message without a tenant processed in the same warm process right after a North message was rejected and nothing was read, and an unknown tenant was rejected while a valid message in the same batch was still processed.

"""SQS-triggered export function. Each message must carry its own tenant: nothing is read from
an earlier invocation, because Lambda reuses the same process for many callers."""
import json

import boto3

s3 = boto3.client("s3")  # safe to reuse: a client holds no caller's authority
TENANTS = {"north", "south"}


def handler(event, context):
    failures = []
    for record in event["Records"]:
        try:
            message = json.loads(record["body"])
            tenant = message["tenant"]  # required; never fall back to a previous message
            if tenant not in TENANTS:
                raise ValueError(f"unknown tenant {tenant!r}")
            s3.get_object(Bucket="exports", Key=f"{tenant}/{message['report']}")
        except (KeyError, ValueError) as err:
            print(json.dumps({"rejected": record["messageId"], "reason": repr(err)}))
            failures.append({"itemIdentifier": record["messageId"]})  # retried, then dead-lettered
    return {"batchItemFailures": failures}  # needs ReportBatchItemFailures on the mapping

Try the intended caller, another valid workload identity and a caller outside the trust boundary. Observe both transport establishment and method result. Then rotate credentials, expire a test identity and replay a function event. Keep an authorized successful call in the suite. Record mesh/platform versions and policies so a control-plane or deployment change triggers revalidation rather than inheriting assumptions from an earlier topology.

The script below makes each caller try both billing operations and prints the status code, or a connection failure; it produced the results described above.

# Each caller tries the export and the health check on billing (status code or connection error)
for caller in reports support legacy; do
  for request in "POST /invoices/I7/export" "GET /health"; do
    method=${request% *}; path=${request#* }
    code=$(kubectl -n "$caller" exec client -c curl -- \
      curl -s -o /dev/null -w '%{http_code}' -X "$method" "http://billing.billing.svc.cluster.local$path" 2>/dev/null)
    echo "$caller $request -> ${code:-connection failed}"
  done
done

A mesh in permissive mode can accept both authenticated mTLS and plaintext traffic. A successful mTLS call does not prove plaintext is rejected. Test direct/excluded listeners and the actual destination policy; a sidecar existing in the pod is not itself an authorization rule.

Credential rotation tests should include already-open connections and newly established ones. A connection may retain a previously authenticated peer context according to the implementation. State the revocation and reauthentication contract rather than assuming certificate expiry instantly terminates every existing stream.

These outcomes belong to the worked example; the mesh probe reproduced the method rows and the handler tests the invocation rows, while the object-tenant row needs the application's own test.

Sources & further reading

The standards, specifications and documentation behind the guidance above. Each is the evidence for a claim on this page; consult it for detail beyond the defaults and versions listed here.

1. Istio security concepts
2. SPIFFE concepts: SPIFFE IDs, SVIDs and trust bundles
3. Istio: SPIRE integration (the Istio SPIFFE ID pattern)
4. AWS Lambda best practices
5. Istio: AuthorizationPolicy reference
6. Istio: PeerAuthentication reference
7. AWS Lambda: resource-based policies
8. AWS Lambda: configuring an SQS event source
9. AWS Lambda: partial batch responses for SQS

Reference information

Tags

Access controlWorkload identityTLSCloud securityKubernetes and containers

SDLC phase

ReleaseMaintenance

Role

Platform engineers and SREs

ASVS 5.0.0 requirements

v5.0.0-12.3.3 · v5.0.0-12.3.5 · v5.0.0-8.2.1

Download this guide as Markdown

Continue exploring