---
key: "runtime-detection-response"
title: "Turn runtime events into a tested investigation and containment path"
url: "https://softwaresecurity.com/domains/runtime/detection/runtime-detection-response/"
api: "https://softwaresecurity.com/api/v1/articles/runtime-detection-response.json"
domain: "runtime"
topic: "Runtime, cloud \u0026 APIs"
type: "Guide"
identifiers: ["CWE-778","A09:2025","V16"]
tags: ["Cloud security","Kubernetes and containers","Logging and detection","Incident response"]
stages: ["operate","respond"]
roles: ["platform-sre"]
license: "CC-BY-4.0"
license_url: "https://creativecommons.org/licenses/by/4.0/"
code_license: "MIT"
---

# Turn runtime events into a tested investigation and containment path

Collect evidence that answers a concrete security question and rehearse the response without destroying the facts needed to investigate.

Canonical URL: https://softwaresecurity.com/domains/runtime/detection/runtime-detection-response/

## In short

Start from a question, such as "did a role gain export rights and then export?", and make sure the events that answer it are collected: Kubernetes audit logging needs a policy, and CloudTrail records data events such as S3 object reads only when you enable them. Write the rule once, for example in Sigma, order events by when they happened rather than when they arrived, and deduplicate by source ID. Then rehearse containment and keep the original evidence.

- An alert is a hypothesis: confirm actor, action, outcome and authorization before calling it an incident.
- Kubernetes writes no audit log until the API server gets a policy, and managed services such as EKS leave audit logs off by default.
- CloudTrail trails log management events by default but no data events; S3 object reads need an explicit, separately charged selector.
- Order by source event time and deduplicate by source ID: in the worked example, arrival order hides the grant-then-export sequence.
- Sigma rules convert to Splunk, Elasticsearch and other query languages, so one reviewed rule can follow a change of SIEM.

## Recommended defaults

| Setting | Value | Note | Source |
| --- | --- | --- | --- |
| Secrets and tokens in the Kubernetes audit log | `Metadata` level, never request or response bodies | — | Kubernetes auditing |
| RBAC changes in the Kubernetes audit log | `RequestResponse` for create, update, patch and delete | — | Kubernetes auditing |
| Audit stage | `omitStages: ["RequestReceived"]` | Halves the volume; the `ResponseComplete` record carries the outcome. | Kubernetes auditing |
| CloudTrail data events | Advanced event selectors for the resources you investigate, such as `AWS::S3::Object` under one bucket ARN | Off by default and charged separately. | AWS CloudTrail: logging data events |
| Event ordering | Source event time plus source ID; keep arrival time as a separate field | — | Kubernetes auditing |

## Verify it

1. Create a test binding to `cluster-admin` and confirm a `RequestResponse` audit event with `requestObject.roleRef.name` set to `cluster-admin`.
2. Read a test Secret and confirm the audit event is at `Metadata` level with no response body.
3. Run `sigma check` on every rule and convert it with `sigma convert -t splunk` (or your backend) in CI.
4. Replay the E1 to E4 events with E2 delivered last and confirm exactly one alert.
5. After changing CloudTrail selectors, read an object in the export bucket and confirm a `GetObject` data event arrives.
6. Stop a log source in a test environment and confirm that a collection-gap alert fires.

## What you will learn

- Reconstruct event time without mistaking collector arrival for action order
- Distinguish a detection finding from proof of malicious intent
- Choose containment and verification for the compromised boundary

Audience: Detection engineers, SREs and platform engineers who own audit logging, alerts and the first response to them

Before you start:

- Read timestamped identity, process, network and application events.

Time: About 26 minutes: 8 to read and 18 for the exercise

## Collection and detection are different outcomes

A log archive shows that events were stored. A detection should identify a meaningful condition, provide enough context to investigate and reach someone able to act. Start with a concrete question: did a workload receive a new privileged role, did an export occur after session revocation, or did a deployment change disable an audit path?

Map the question to authoritative event sources. Application logs explain business decisions; cloud audit events explain control-plane operations; Kubernetes audit events explain API actions. Each has coverage, latency and configuration limits. Do not assume a provider's default history includes every data-access event or every resource you need to investigate. CloudTrail trails log management events by default but no data events, such as S3 object reads, until a selector enables them. A Kubernetes API server writes no audit log until it is started with an audit policy, and on Amazon EKS each control plane log type, including audit, is off until you enable it.

Audit logs record calls to the Kubernetes and cloud APIs, not what processes do inside a container. A shell spawned in a pod, a binary dropped and run, or a request to the metadata service needs a runtime sensor such as Falco, Tetragon or the provider's agent; the runtime threat detection guide at /domains/runtime/detection/runtime-threat-detection/ covers choosing, tuning and testing one.

## Worked investigation: privilege change followed by export

The supplied rule alerts when a role gains `ExportAll` and that role performs an export within ten minutes. It describes a suspicious sequence, not a final judgment about intent. A planned migration could produce the same sequence. Investigation should first establish the authenticated actor, exact permission change, actual export outcome and whether the change had a matching authorization record.

Collection can reorder events. E2 arrives at 14:06 but occurred at 14:01; E3 arrives at 14:03 and occurred at 14:02. Sorting only by arrival would put the export before the privilege change and hide the rule’s sequence. Preserve both source event time and arrival time, source identifiers and known clock uncertainty. Where causal identifiers or source sequence numbers exist, use them to strengthen the ordering claim.

An export-start event proves an attempt or accepted operation, not a completed download. The application commit event below names completed object X9 and its tenant scope. A separate download audit might still be required to determine whether the object was retrieved. State these limits in the incident record; do not inflate a started operation into confirmed data theft.

Contain the authority path you can establish. Prevent further use of the affected role/session according to provider semantics, restrict the generated export object, and preserve the evidence. Reverting the permission alone may leave an already-created object or active session accessible. Coordinate interruption of legitimate exports with the incident owner, then test both the forbidden path and restored authorized service.

```text
RULE: role gains ExportAll, then commits export within 10 minutes
E1 event=14:00 arrived=14:00 actor=ops-12 role=R7 change-request=CR8
E2 event=14:01 arrived=14:06 actor=ops-12 role=R7 grant=ExportAll
E3 event=14:02 arrived=14:03 role=R7 export-start operation=O9
E4 event=14:02:30 arrived=14:03 role=R7 export-commit operation=O9 object=X9 scope=all-tenants
CR8 authorizes North-only export; no ExportAll change
```

## Implement the rule on event time

The script below implements the worked rule over JSON lines that carry both `event_time` and `arrived`. It keeps the first copy of each source ID, sorts by event time and alerts when a commit follows an `ExportAll` grant on the same role within ten minutes. Fed E1 to E4 in arrival order, with E2 delivered last and E4 delivered twice, it printed one alert: E2 granted `ExportAll` and E4 committed X9 for all tenants 90 seconds later. The same code sorting by arrival time printed nothing, because the commit arrived three minutes before the grant.

```python
"""Alert when a role gains ExportAll and commits an export within 10 minutes, ordered by event time."""
import json
import sys
from datetime import datetime, timedelta

WINDOW = timedelta(minutes=10)
events = {}
with open(sys.argv[1], encoding="utf-8") as f:
    for line in f:
        event = json.loads(line)
        events.setdefault(event["source_id"], event)  # a redelivered event keeps its source ID


def at(event, field="event_time"):
    return datetime.fromisoformat(event[field])


grants = {}
for event in sorted(events.values(), key=at):  # when it happened, not when it arrived
    if event["action"] == "grant" and event.get("permission") == "ExportAll":
        grants[event["role"]] = event
    elif event["action"] == "export-commit" and event["role"] in grants:
        grant = grants[event["role"]]
        gap = at(event) - at(grant)
        if timedelta(0) <= gap <= WINDOW:
            print(f"ALERT role {event['role']}: {grant['source_id']} by {grant['actor']} granted ExportAll, "
                  f"{event['source_id']} committed {event['object']} ({event['scope']}) "
                  f"{gap.total_seconds():.0f}s later; grant arrived {at(grant, 'arrived'):%H:%M}, "
                  f"commit arrived {at(event, 'arrived'):%H:%M}")
```

## Collect enough context without copying secrets

Use stable actor, operation, target, result, timestamp and correlation fields, as the security event logging guide at /domains/runtime/detection/security-event-logging/ shows. Distinguish request attempts from completed effects. Protect transport, storage and access to the evidence, and record clock uncertainty or collection delays. Avoid raw tokens, confidential payloads and unnecessary personal data; a detection pipeline can become a second exposure if it copies the entire request.

Select audit detail intentionally. High-detail request or response capture may reveal secret contents and create an excessive storage load. Protect collection configuration and alert when sources stop arriving. Test the path with synthetic events whose expected identifiers are known so silent gaps become observable rather than inferred from an empty dashboard.

## Record Kubernetes API actions with an audit policy

The API server applies the first matching rule to each request, so order rules from most to least specific. The policy below drops leader-election leases and events, records who touched Secrets and tokens without their contents, keeps full request and response bodies for RBAC changes, records exec, attach and port-forward, and drops the remaining reads. Pass it with `--audit-policy-file` and a log or webhook backend.

On a Kubernetes 1.37 test cluster, creating a ClusterRoleBinding to `cluster-admin` produced a `RequestResponse` event whose `requestObject.roleRef.name` was `cluster-admin`, a Secret read produced a `Metadata` event without the object, and `kubectl exec` produced a `Metadata` event for `pods/exec`. Before the RBAC rule was limited to write verbs, controller reads of roles and bindings were logged with full bodies, and lease renewals were the largest single source of entries.

```yaml
apiVersion: audit.k8s.io/v1
kind: Policy
omitStages: ["RequestReceived"]
rules:
  # High-volume system traffic: leader-election leases and events
  - level: None
    resources:
      - group: "coordination.k8s.io"
        resources: ["leases"]
      - group: ""
        resources: ["events"]
  # Secret and token access: who and when, never the contents
  - level: Metadata
    resources:
      - group: ""
        resources: ["secrets", "serviceaccounts/token"]
      - group: "authentication.k8s.io"
        resources: ["tokenreviews"]
  # RBAC changes: keep the full request and response
  - level: RequestResponse
    verbs: ["create", "update", "patch", "delete", "deletecollection"]
    resources:
      - group: "rbac.authorization.k8s.io"
        resources: ["roles", "rolebindings", "clusterroles", "clusterrolebindings"]
  # Interactive access to running containers
  - level: Metadata
    resources:
      - group: ""
        resources: ["pods/exec", "pods/attach", "pods/portforward"]
  # Remaining reads and health checks
  - level: None
    verbs: ["get", "list", "watch"]
  - level: None
    nonResourceURLs: ["/healthz*", "/readyz*", "/livez*", "/version"]
  # Everything else: who changed what, without bodies
  - level: Metadata
```

## Turn on the CloudTrail data events you will need

The export in the worked example would be invisible in CloudTrail unless object-level events were enabled for its bucket. Apply the selectors below with `aws cloudtrail put-event-selectors --trail-name org-trail --advanced-event-selectors file://selectors.json`. The call replaces the trail's existing selectors, so the management-event selector must be listed again or management events stop. Scope data events to the buckets, functions or tables you will investigate, because they are charged per event. The AWS CLI 2.37 accepted this file and rejected a misspelled operator with a parameter validation error before sending anything.

```json
[
  {
    "Name": "Management events",
    "FieldSelectors": [{"Field": "eventCategory", "Equals": ["Management"]}]
  },
  {
    "Name": "Object reads and writes in the export bucket",
    "FieldSelectors": [
      {"Field": "eventCategory", "Equals": ["Data"]},
      {"Field": "resources.type", "Equals": ["AWS::S3::Object"]},
      {"Field": "resources.ARN", "StartsWith": ["arn:aws:s3:::acme-exports/"]}
    ]
  }
]
```

## Connect a rule to an investigation decision

Define the triggering condition, relevant exceptions, likely benign explanations and first evidence checks. A role change by a known deployment may differ from the same change by an unexpected interactive session. Preserve source event identifiers and original evidence while enriching the alert; enrichment failures should not erase the underlying facts.

Choose containment according to the compromised boundary: revoke a session or credential, restrict a workload, stop a deployment, or isolate a resource. Understand propagation delay and effects on legitimate service. Coordinate changes with the incident owner and preserve volatile evidence where necessary before replacing workloads. Avoid repeatedly restarting a suspect system before collecting the facts needed to explain it.

## Write the detection once in Sigma

Sigma is a vendor-neutral YAML format for log detections. The rule below flags a RoleBinding or ClusterRoleBinding that grants `cluster-admin`, using the field names of the Kubernetes audit event, and maps to MITRE ATT&amp;CK T1098.006, additional container cluster roles. `sigma check` validated it, and `sigma convert` produced `verb IN ("create", "update", "patch") objectRef.resource IN ("clusterrolebindings", "rolebindings") requestObject.roleRef.name="cluster-admin"` for Splunk and an equivalent ES|QL `where` clause for Elasticsearch. The same field paths matched the real event in the audit log above. In production, add the processing pipeline for your log mapping instead of `--without-pipeline`.

```yaml
title: Kubernetes binding to cluster-admin created or changed
id: eeb628dd-ec5c-47ff-9178-59178cf3d324
status: experimental
description: A RoleBinding or ClusterRoleBinding now grants the cluster-admin ClusterRole.
references:
    - https://kubernetes.io/docs/reference/access-authn-authz/rbac/
author: Software Security
date: 2026-10-10
tags:
    - attack.persistence
    - attack.privilege-escalation
    - attack.t1098.006
logsource:
    product: kubernetes
    service: audit
detection:
    selection:
        verb:
            - create
            - update
            - patch
        objectRef.resource:
            - clusterrolebindings
            - rolebindings
        requestObject.roleRef.name: cluster-admin
    condition: selection
falsepositives:
    - Cluster bootstrap and approved break-glass changes
level: high
```

## Exercise the full path to recovery

Inject a controlled privilege-change event and verify rule evaluation, notification, acknowledgment and an analyst's ability to reconstruct it. Include duplicate, delayed and missing events. Rehearse a reversible containment in a disposable environment, then confirm both the attack path is blocked and legitimate service is restored. Update the detection and application regression from the root cause; a closed alert does not establish that the vulnerable behavior was repaired.

## Diagnose incomplete fixes

An alert that cannot name the source evidence makes investigation depend on a fragile dashboard rendering. Retain event IDs and controlled references to original records. Enrichment should add context while preserving the raw event’s meaning; failure to resolve an actor name should not discard its authenticated identifier.

Automatic containment can cause an outage if a rule is noisy or the affected role serves many systems. Predefine authority, scope, rollback and escalation for the action. A dry-run match rate is useful, but a controlled end-to-end exercise is needed to establish notification delivery, analyst access and actual containment behavior.

## Check the result against concrete cases

These outcomes belong to the worked example; the script above reproduces the first and fourth rows, and the rest need your own pipeline and a controlled test event.

| Case | Expected result | What it establishes |
| --- | --- | --- |
| E2 then E4 by source event time | Alert matches supplied condition | Delayed arrival does not hide the sequence |
| E3 without E4 | Attempt observed; completion unproven | Event semantics remain precise |
| CR8 North-only approval | Does not authorize `ExportAll` | Exception matches the changed capability |
| Duplicate E4 | One logical commit in investigation | Stable source IDs prevent double counting |
| No source events during outage | Unknown coverage, not no incidents | Collection health is visible |

## Guided practice

Build a defensible incident note from E1–E4 and CR8. Separate observed facts, inferred risk and missing evidence.

### Supplied material

```text
Additional evidence: E4 is delivered twice with the same source ID. Session S7 performed O9. Object X9 is readable through a signed URL valid for 1 hour. Download audit is disabled. Collector reports a 5-minute delay for E2. Proposed response: remove ExportAll and close the alert immediately.
```

### 1. Reconstruct the sequence

Order the actions by source time and calculate the grant-to-commit interval.

Worked answer: `ExportAll` is granted at 14:01 and O9 commits at 14:02:30: 90 seconds later. E2’s late arrival does not reverse those actions. Duplicate E4 does not establish a second export.

### 2. Compare approval with action

Explain why CR8 does not dismiss the alert and what remains unknown.

Worked answer: CR8 covers North-only export, while the grant and completed object cover all tenants. That mismatch warrants investigation. The evidence does not yet establish whether ops-12 was malicious, compromised or mistaken.

### 3. Bound the exposure claim

State what can and cannot be concluded about X9.

Worked answer: X9 was created with all-tenant scope and had a one-hour access URL. Without download audit, retrieval is unknown; absence of download events is not evidence that nobody retrieved it.

### 4. Improve the response

Explain why removing `ExportAll` is incomplete and name a closure test.

Worked answer: Review S7’s remaining authority and the provider’s revocation behavior, restrict/invalidate X9 access through supported controls, and preserve evidence. Verify S7 cannot export again and X9 is no longer accessible through the affected path, while authorized North-only export still works. Keep unresolved retrieval uncertainty in the record.

Deliverable: An incident note that orders E1 to E4 by event time, separates facts from inferences, bounds the X9 exposure claim and proposes containment with a closure test.

Scope: The incident note uses only the supplied events; the audit policy, Sigma rule and detector on this page show how to produce and test the same evidence in a real pipeline.

Articles and teaching material: CC BY 4.0. Original executable code: MIT. Attribution: Software Security.

## Sources

- [Kubernetes auditing](https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/). Kubernetes v1.37.
- [AWS CloudTrail overview](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-user-guide.html)
- [AWS Security Pillar](https://docs.aws.amazon.com/wellarchitected/latest/security-pillar/welcome.html)
- [AWS CloudTrail: logging data events](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/logging-data-events-with-cloudtrail.html)
- [Amazon EKS: control plane logs](https://docs.aws.amazon.com/eks/latest/userguide/control-plane-logs.html)
- [Sigma rule specification](https://sigmahq.io/docs/basics/rules.html)
- [sigma-cli](https://github.com/SigmaHQ/sigma-cli)
- [MITRE ATT&amp;CK T1098.006: additional container cluster roles](https://attack.mitre.org/techniques/T1098/006/)
- [OWASP Top 10:2025 A09 Security Logging and Alerting Failures](https://top10.owasp.org/2025/A09_2025-Security_Logging_and_Alerting_Failures/)