Turn runtime events into a tested investigation and containment path

A log archive shows that events were stored. A detection should identify a meaningful condition, provide enough context to investigate and reach someone able to act. Start with a concrete question: did a workload receive a new privileged role, did an export occur after session revocation, or did a deployment change disable an audit path?

Map the question to authoritative event sources. Application logs explain business decisions; cloud audit events explain control-plane operations; Kubernetes audit events explain API actions. Each has coverage, latency and configuration limits. Do not assume a provider's default history includes every data-access event or every resource you need to investigate. CloudTrail trails log management events by default but no data events, such as S3 object reads, until a selector enables them. A Kubernetes API server writes no audit log until it is started with an audit policy, and on Amazon EKS each control plane log type, including audit, is off until you enable it.

Audit logs record calls to the Kubernetes and cloud APIs, not what processes do inside a container. A shell spawned in a pod, a binary dropped and run, or a request to the metadata service needs a runtime sensor such as Falco, Tetragon or the provider's agent; the runtime threat detection guide at /domains/runtime/detection/runtime-threat-detection/ covers choosing, tuning and testing one.

The supplied rule alerts when a role gains ExportAll and that role performs an export within ten minutes. It describes a suspicious sequence, not a final judgment about intent. A planned migration could produce the same sequence. Investigation should first establish the authenticated actor, exact permission change, actual export outcome and whether the change had a matching authorization record.

Collection can reorder events. E2 arrives at 14:06 but occurred at 14:01; E3 arrives at 14:03 and occurred at 14:02. Sorting only by arrival would put the export before the privilege change and hide the rule’s sequence. Preserve both source event time and arrival time, source identifiers and known clock uncertainty. Where causal identifiers or source sequence numbers exist, use them to strengthen the ordering claim.

An export-start event proves an attempt or accepted operation, not a completed download. The application commit event below names completed object X9 and its tenant scope. A separate download audit might still be required to determine whether the object was retrieved. State these limits in the incident record; do not inflate a started operation into confirmed data theft.

Contain the authority path you can establish. Prevent further use of the affected role/session according to provider semantics, restrict the generated export object, and preserve the evidence. Reverting the permission alone may leave an already-created object or active session accessible. Coordinate interruption of legitimate exports with the incident owner, then test both the forbidden path and restored authorized service.

RULE: role gains ExportAll, then commits export within 10 minutes
E1 event=14:00 arrived=14:00 actor=ops-12 role=R7 change-request=CR8
E2 event=14:01 arrived=14:06 actor=ops-12 role=R7 grant=ExportAll
E3 event=14:02 arrived=14:03 role=R7 export-start operation=O9
E4 event=14:02:30 arrived=14:03 role=R7 export-commit operation=O9 object=X9 scope=all-tenants
CR8 authorizes North-only export; no ExportAll change

The script below implements the worked rule over JSON lines that carry both event_time and arrived. It keeps the first copy of each source ID, sorts by event time and alerts when a commit follows an ExportAll grant on the same role within ten minutes. Fed E1 to E4 in arrival order, with E2 delivered last and E4 delivered twice, it printed one alert: E2 granted ExportAll and E4 committed X9 for all tenants 90 seconds later. The same code sorting by arrival time printed nothing, because the commit arrived three minutes before the grant.

"""Alert when a role gains ExportAll and commits an export within 10 minutes, ordered by event time."""
import json
import sys
from datetime import datetime, timedelta

WINDOW = timedelta(minutes=10)
events = {}
with open(sys.argv[1], encoding="utf-8") as f:
    for line in f:
        event = json.loads(line)
        events.setdefault(event["source_id"], event)  # a redelivered event keeps its source ID


def at(event, field="event_time"):
    return datetime.fromisoformat(event[field])


grants = {}
for event in sorted(events.values(), key=at):  # when it happened, not when it arrived
    if event["action"] == "grant" and event.get("permission") == "ExportAll":
        grants[event["role"]] = event
    elif event["action"] == "export-commit" and event["role"] in grants:
        grant = grants[event["role"]]
        gap = at(event) - at(grant)
        if timedelta(0) <= gap <= WINDOW:
            print(f"ALERT role {event['role']}: {grant['source_id']} by {grant['actor']} granted ExportAll, "
                  f"{event['source_id']} committed {event['object']} ({event['scope']}) "
                  f"{gap.total_seconds():.0f}s later; grant arrived {at(grant, 'arrived'):%H:%M}, "
                  f"commit arrived {at(event, 'arrived'):%H:%M}")

Use stable actor, operation, target, result, timestamp and correlation fields, as the security event logging guide at /domains/runtime/detection/security-event-logging/ shows. Distinguish request attempts from completed effects. Protect transport, storage and access to the evidence, and record clock uncertainty or collection delays. Avoid raw tokens, confidential payloads and unnecessary personal data; a detection pipeline can become a second exposure if it copies the entire request.

Select audit detail intentionally. High-detail request or response capture may reveal secret contents and create an excessive storage load. Protect collection configuration and alert when sources stop arriving. Test the path with synthetic events whose expected identifiers are known so silent gaps become observable rather than inferred from an empty dashboard.

The API server applies the first matching rule to each request, so order rules from most to least specific. The policy below drops leader-election leases and events, records who touched Secrets and tokens without their contents, keeps full request and response bodies for RBAC changes, records exec, attach and port-forward, and drops the remaining reads. Pass it with --audit-policy-file and a log or webhook backend.

On a Kubernetes 1.37 test cluster, creating a ClusterRoleBinding to cluster-admin produced a RequestResponse event whose requestObject.roleRef.name was cluster-admin, a Secret read produced a Metadata event without the object, and kubectl exec produced a Metadata event for pods/exec. Before the RBAC rule was limited to write verbs, controller reads of roles and bindings were logged with full bodies, and lease renewals were the largest single source of entries.

apiVersion: audit.k8s.io/v1
kind: Policy
omitStages: ["RequestReceived"]
rules:
  # High-volume system traffic: leader-election leases and events
  - level: None
    resources:
      - group: "coordination.k8s.io"
        resources: ["leases"]
      - group: ""
        resources: ["events"]
  # Secret and token access: who and when, never the contents
  - level: Metadata
    resources:
      - group: ""
        resources: ["secrets", "serviceaccounts/token"]
      - group: "authentication.k8s.io"
        resources: ["tokenreviews"]
  # RBAC changes: keep the full request and response
  - level: RequestResponse
    verbs: ["create", "update", "patch", "delete", "deletecollection"]
    resources:
      - group: "rbac.authorization.k8s.io"
        resources: ["roles", "rolebindings", "clusterroles", "clusterrolebindings"]
  # Interactive access to running containers
  - level: Metadata
    resources:
      - group: ""
        resources: ["pods/exec", "pods/attach", "pods/portforward"]
  # Remaining reads and health checks
  - level: None
    verbs: ["get", "list", "watch"]
  - level: None
    nonResourceURLs: ["/healthz*", "/readyz*", "/livez*", "/version"]
  # Everything else: who changed what, without bodies
  - level: Metadata

The export in the worked example would be invisible in CloudTrail unless object-level events were enabled for its bucket. Apply the selectors below with aws cloudtrail put-event-selectors --trail-name org-trail --advanced-event-selectors file://selectors.json. The call replaces the trail's existing selectors, so the management-event selector must be listed again or management events stop. Scope data events to the buckets, functions or tables you will investigate, because they are charged per event. The AWS CLI 2.37 accepted this file and rejected a misspelled operator with a parameter validation error before sending anything.

[
  {
    "Name": "Management events",
    "FieldSelectors": [{"Field": "eventCategory", "Equals": ["Management"]}]
  },
  {
    "Name": "Object reads and writes in the export bucket",
    "FieldSelectors": [
      {"Field": "eventCategory", "Equals": ["Data"]},
      {"Field": "resources.type", "Equals": ["AWS::S3::Object"]},
      {"Field": "resources.ARN", "StartsWith": ["arn:aws:s3:::acme-exports/"]}
    ]
  }
]

Define the triggering condition, relevant exceptions, likely benign explanations and first evidence checks. A role change by a known deployment may differ from the same change by an unexpected interactive session. Preserve source event identifiers and original evidence while enriching the alert; enrichment failures should not erase the underlying facts.

Choose containment according to the compromised boundary: revoke a session or credential, restrict a workload, stop a deployment, or isolate a resource. Understand propagation delay and effects on legitimate service. Coordinate changes with the incident owner and preserve volatile evidence where necessary before replacing workloads. Avoid repeatedly restarting a suspect system before collecting the facts needed to explain it.

Sigma is a vendor-neutral YAML format for log detections. The rule below flags a RoleBinding or ClusterRoleBinding that grants cluster-admin, using the field names of the Kubernetes audit event, and maps to MITRE ATT&CK T1098.006, additional container cluster roles. sigma check validated it, and sigma convert produced verb IN ("create", "update", "patch") objectRef.resource IN ("clusterrolebindings", "rolebindings") requestObject.roleRef.name="cluster-admin" for Splunk and an equivalent ES|QL where clause for Elasticsearch. The same field paths matched the real event in the audit log above. In production, add the processing pipeline for your log mapping instead of --without-pipeline.

title: Kubernetes binding to cluster-admin created or changed
id: eeb628dd-ec5c-47ff-9178-59178cf3d324
status: experimental
description: A RoleBinding or ClusterRoleBinding now grants the cluster-admin ClusterRole.
references:
    - https://kubernetes.io/docs/reference/access-authn-authz/rbac/
author: Software Security
date: 2026-10-10
tags:
    - attack.persistence
    - attack.privilege-escalation
    - attack.t1098.006
logsource:
    product: kubernetes
    service: audit
detection:
    selection:
        verb:
            - create
            - update
            - patch
        objectRef.resource:
            - clusterrolebindings
            - rolebindings
        requestObject.roleRef.name: cluster-admin
    condition: selection
falsepositives:
    - Cluster bootstrap and approved break-glass changes
level: high

Inject a controlled privilege-change event and verify rule evaluation, notification, acknowledgment and an analyst's ability to reconstruct it. Include duplicate, delayed and missing events. Rehearse a reversible containment in a disposable environment, then confirm both the attack path is blocked and legitimate service is restored. Update the detection and application regression from the root cause; a closed alert does not establish that the vulnerable behavior was repaired.

An alert that cannot name the source evidence makes investigation depend on a fragile dashboard rendering. Retain event IDs and controlled references to original records. Enrichment should add context while preserving the raw event’s meaning; failure to resolve an actor name should not discard its authenticated identifier.

Automatic containment can cause an outage if a rule is noisy or the affected role serves many systems. Predefine authority, scope, rollback and escalation for the action. A dry-run match rate is useful, but a controlled end-to-end exercise is needed to establish notification delivery, analyst access and actual containment behavior.

These outcomes belong to the worked example; the script above reproduces the first and fourth rows, and the rest need your own pipeline and a controlled test event.

Sources & further reading

The standards, specifications and documentation behind the guidance above. Each is the evidence for a claim on this page; consult it for detail beyond the defaults and versions listed here.

1. Kubernetes auditing. Kubernetes v1.37.
2. AWS CloudTrail overview
3. AWS Security Pillar
4. AWS CloudTrail: logging data events
5. Amazon EKS: control plane logs
6. Sigma rule specification
7. sigma-cli
8. MITRE ATT&CK T1098.006: additional container cluster roles
9. OWASP Top 10:2025 A09 Security Logging and Alerting Failures

Reference information

Tags

Cloud securityKubernetes and containersLogging and detectionIncident response

SDLC phase

Maintenance

Role

Platform engineers and SREs

Download this guide as Markdown

Continue exploring