--- key: "runtime-threat-detection" title: "Detect container and host threats at runtime with eBPF sensors and cloud services" url: "https://softwaresecurity.com/domains/runtime/detection/runtime-threat-detection/" api: "https://softwaresecurity.com/api/v1/articles/runtime-threat-detection.json" domain: "runtime" topic: "Runtime, cloud \u0026 APIs" type: "Guide" identifiers: ["CWE-778","A09:2025","V16"] tags: ["Cloud security","Kubernetes and containers","Logging and detection","Incident response"] stages: ["operate","respond"] roles: ["platform-sre"] license: "CC-BY-4.0" license_url: "https://creativecommons.org/licenses/by/4.0/" code_license: "MIT" --- # Detect container and host threats at runtime with eBPF sensors and cloud services Watch process, file and network activity inside containers with an eBPF sensor or a managed cloud agent, write and test rules for the attacks you expect, tune them with narrow exceptions and contain a compromised pod without destroying evidence. Canonical URL: https://softwaresecurity.com/domains/runtime/detection/runtime-threat-detection/ ## In short Run one kernel-level sensor on every node. Use Falco or Tracee to detect, Tetragon when you also want to block in the kernel, or your provider's agent: GuardDuty Runtime Monitoring on AWS, the Defender for Containers sensor on Azure, Security Command Center Container Threat Detection on Google Cloud. Keep the vendor's stable rules, add rules for the attacks you expect, such as a pod calling the metadata service, and trigger each rule in a test cluster before you trust it. Tune with exceptions keyed on image and namespace, never by turning a rule off. When an alert is real, isolate the pod with a NetworkPolicy and keep it for evidence before you replace it. - A runtime sensor sees what processes do: a shell, a binary written to `/tmp` and run, a connection to `169.254.169.254`. Image scans and audit logs see none of that. - Falco 0.45 uses the modern eBPF driver by default. In our test its upstream rules caught an attached shell and a dropped binary without any tuning. - Test every custom rule by triggering it. Our metadata rule keyed only on `fd.sip` missed a blocked connection, which the connect event still reported in `evt.arg.addr`. - Tune with exceptions on image repository and namespace. Falco reported a retagged alpine image as alpine, so an exception for the new name never matched it. - Managed agents need no rules but ship with detectors off. GKE's Added Binary Executed is disabled by default, and Defender's default drift rule ignores drift. - Contain before you delete: detach and isolate the pod, cordon the node, record the pod spec and logs, then roll out a clean image. ## Recommended defaults | Setting | Value | Note | Source | | --- | --- | --- | --- | | Falco driver | `modern_ebpf` | The default engine in Falco 0.45; the Helm chart's `driver.kind: auto` picks a driver per node. | https://github.com/falcosecurity/falco/blob/0.45.0/falco.yaml | | Falco rules | Upstream `falco_rules.yaml` plus your own files in `rules.d`; never edit the upstream file | Change an upstream rule with `override:` and `append` or `replace`. | https://falco.org/docs/concepts/rules/overriding/ | | Rule validation | `falco -V` on every rules file in CI, alongside the upstream rules | — | — | | Alert routing | JSON output to Falcosidekick or a log pipeline; page on WARNING and above for production namespaces | Team policy; tune the paging threshold after a week of data. | — | | Tetragon enforcement | `policy-mode: monitor` first, then `enforce` | — | https://tetragon.io/docs/concepts/tracing-policy/mode/ | | GuardDuty on EKS | Runtime Monitoring with automated agent configuration, plus EKS Protection for audit logs | Both feed the `AttackSequence:EKS/CompromisedCluster` finding. | https://docs.aws.amazon.com/guardduty/latest/ug/guardduty-extended-threat-detection.html | | Defender binary drift | Drift detection alert for production namespaces | The default rule's action is Ignore drift detection. | https://learn.microsoft.com/en-us/azure/defender-for-cloud/binary-drift-detection | | GKE Container Threat Detection | Enable Added Binary Executed and Added Library Loaded | Disabled by default; needs Security Command Center Premium or Enterprise. | https://docs.cloud.google.com/security-command-center/docs/concepts-container-threat-detection-overview | ## Verify it 1. Run `falco -V` on every rules file in CI, then confirm the Falco pod logs that it loaded your file at startup. 2. In a test namespace, trigger each rule: `kubectl exec` a shell, copy a binary to `/tmp` and run it, connect to `169.254.169.254`. Expect one alert each, with namespace, pod and image filled in. 3. Repeat each trigger from an excepted image and from an ordinary image: the first must stay quiet and the second must alert, which proves the exception is narrow. 4. Delete the sensor pod on one node and confirm you are told within your target time, from Falco's health endpoint or the provider's coverage view. 5. For a managed agent, generate the provider's sample findings and confirm they reach the on-call route. 6. Quarantine a test pod and confirm it can no longer reach the internet or the metadata address, while `kubectl logs` and `kubectl exec` still work for the investigation. ## What you will learn - Choose between an eBPF sensor you run and a provider-managed runtime agent. - Write a Falco rule for an attack you expect, test it, and add exceptions without disabling it. - Use a Tetragon policy to report or block an action in the kernel. - Contain a compromised pod while keeping the evidence an investigation needs. Audience: Platform, SRE and detection engineers who run Kubernetes clusters and own the alerts their workloads raise Before you start: - Read Kubernetes pod specs and Helm values. - Know which namespaces, images and service accounts your workloads use. Time: About 34 minutes: 9 to read and 25 for the exercise ## What a runtime sensor sees Scanners judge an image before it runs and audit logs record calls to the Kubernetes and cloud APIs. Neither sees what a process does inside a running container. A runtime sensor does: it attaches eBPF programs, or a kernel module, to system calls and kernel functions on each node, enriches each event with the container, pod and image it came from, and matches it against rules. That view catches the middle of an attack: a reverse shell after a deserialization bug, a cryptominer written to `/tmp` and executed, a read of a service account token, a request for cloud credentials from the instance metadata service, or an escape attempt through a writable host mount. These map to MITRE ATT&CK techniques such as T1609 container administration command, T1611 escape to host and T1552.005 cloud instance metadata API. The sensor runs as a privileged DaemonSet with access to the host kernel, so treat it as part of the node's trusted base: pin its image, restrict who can change its configuration, and alert when it stops reporting. An attacker with root on the node can blind it. ## Choose a sensor you run or one your provider runs Open-source sensors give you the rules and the data; managed agents give you detections written and maintained by the provider and findings in its console. Many teams run both, with the provider's agent for coverage and Falco or Tetragon rules for their own architecture. Check the details below against the current documentation before you buy or deploy. Google retired the Preview GKE threat detection feature in the GKE security posture dashboard on 31 March 2025. On Google Cloud, runtime detection now comes from Security Command Center: Container Threat Detection on nodes and Event Threat Detection on logs. The Enterprise tier of Security Command Center shuts down on 21 May 2027, and its customers move to Premium. | Option | How it collects | Detection content | Can it block? | | --- | --- | --- | --- | | Falco 0.45 (CNCF) | Modern eBPF driver or kernel module; container plugin adds container and pod fields | Upstream rules by maturity (stable, incubating, sandbox) plus your YAML rules | No; Falco Talon or your own automation responds | | Tetragon 1.7 (Cilium) | eBPF on kprobes, tracepoints and LSM hooks, filtered in the kernel | TracingPolicy resources you write; example library | Yes: Sigkill, Override and other in-kernel actions | | Tracee 0.24 (Aqua) | eBPF events and signatures | Built-in signatures plus Policy resources that pick events | No | | GuardDuty Runtime Monitoring | GuardDuty security agent: EKS add-on `aws-guardduty-agent`, a sidecar in each new ECS Fargate task, an EC2 agent | AWS finding types such as `Execution:Runtime/ReverseShell` | No; automate with EventBridge | | Defender for Containers | Defender sensor DaemonSet on AKS, EKS, GKE and Arc-enabled clusters | Microsoft analytics mapped to MITRE ATT&CK for Containers | Binary drift blocking and gated deployment | | SCC Container Threat Detection | Kernel-level instrumentation on Container-Optimized OS nodes | Google detectors such as Reverse Shell and Added Binary Executed | No | ## Install Falco with the upstream rules and your own Install the chart pinned to a version, keep the upstream rules file untouched, and put your rules in a separate file that loads after it. The chart's `customRules` value turns each key into a file under `/etc/falco/rules.d`. Setting `falcosidekick.enabled=true` also switches Falco to JSON output and deploys Falcosidekick, which forwards alerts to chat, a SIEM or a webhook. We rendered this chart, version 9.2.0 for Falco 0.45.0, with Helm 4.3 and confirmed the DaemonSet, the `modern_ebpf` engine setting, JSON output and the rules file in the generated configuration. Validate rules files in CI with `falco -V`: it loads them with the upstream rules and reports each load error with the file, rule and position, such as an undefined macro. ```bash helm repo add falcosecurity https://falcosecurity.github.io/charts helm repo update # custom-rules.yaml holds your rules under customRules: (one key per file). helm install falco falcosecurity/falco --version 9.2.0 \ --namespace falco --create-namespace \ --set driver.kind=modern_ebpf \ --set falcosidekick.enabled=true \ -f custom-rules.yaml # In CI: every rules file must load cleanly next to the upstream rules. docker run --rm -v "$PWD/rules:/rules:ro" falcosecurity/falco:0.45.0 \ falco -V /etc/falco/falco_rules.yaml -V /rules/rules-runtime.yaml ``` ## Write a rule for an attack you expect Start from your threat model, not the rule catalog. A pod that uses IRSA, EKS Pod Identity or Workload Identity Federation never needs the node's metadata service, so a connection to `169.254.169.254` from such a pod is worth an alert; it often follows a server-side request forgery bug of the kind the SSRF guide at /domains/appsec/secure-features/ssrf-egress-boundaries/ covers. The rule below uses the `container` macro from the upstream rules and an allowlist for the few images that legitimately call it. We ran this rule on Falco 0.45.0 with the modern eBPF driver, against a test server bound to `169.254.169.254`. A successful `wget` and an `nc` from ordinary containers both alerted, with `target=169.254.169.254:80 result=SUCCESS`. The first version matched only on `fd.sip`, the server address of an established connection. It missed a request that never completed: the connect event arrived with `res=ERESTARTSYS` and no `fd.sip`, but `evt.arg.addr` still held the destination. Blocked attempts matter, because a pod probing the metadata service is suspicious even when IMDSv2 or a hop limit stops it. Our test containers ran under Docker, so `k8s.ns.name` and `k8s.pod.name` printed ``. In a cluster the container plugin fills them from the container runtime. ```yaml # rules-runtime.yaml: loaded after falco_rules.yaml, so it can extend upstream rules. - list: imds_allowed_images items: [public.ecr.aws/aws-observability/aws-for-fluent-bit] - rule: Container contacts cloud metadata service desc: > A process in a container tried to connect to the instance metadata address. Pods with their own workload identity never need it; a connection here often follows SSRF or an interactive foothold. condition: > evt.type = connect and container and (fd.sip = "169.254.169.254" or evt.arg.addr startswith "169.254.169.254:") and not container.image.repository in (imds_allowed_images) output: > Container contacted the metadata service | process=%proc.name command=%proc.cmdline image=%container.image.repository ns=%k8s.ns.name pod=%k8s.pod.name target=%evt.arg.addr result=%evt.res priority: WARNING tags: [container, network, mitre_credential_access, T1552.005] ``` ## Tune with narrow exceptions, not by switching rules off Noise kills detection programs. Tune each noisy rule by naming the specific, expected behavior, and leave the rule on for everything else. Falco's `override` key appends exceptions or conditions to an upstream rule from your own file, so upgrades to the upstream rules still apply. We tested the exception below. A shell in a container from `registry.example.com/platform/debug-tools` stayed quiet, while the same shell in an alpine container still raised Terminal shell in container. A first attempt with a retagged copy of alpine kept alerting: Falco reported the repository of the underlying image, alpine, not the new name, so the exception never matched. Test exceptions with the images you actually deploy. The upstream rules also fired where we did not expect them. Redirect STDOUT/STDIN to Network Connection in Container triggered for busybox `ping` and for the test web server answering requests. Before writing an exception like that, check which process, image and namespace produced the alert, and whether the behavior could also be an attacker's. ```yaml # Add a narrow exception to an upstream rule without copying or editing it. - rule: Terminal shell in container exceptions: - name: approved_debug_images fields: [container.image.repository] comps: [=] values: - [registry.example.com/platform/debug-tools] override: exceptions: append ``` ## Report or block in the kernel with Tetragon Tetragon filters in the kernel and can act there: `Sigkill` stops the process before the call returns, and `Override` makes a hooked function return an error. That turns a detection into a control, so start every enforcing policy in monitor mode, where Tetragon reports each match but skips the action, and switch it to enforce only after it has run cleanly on real traffic. `tetra tracingpolicy list` shows each policy's mode. This namespaced policy adapts the network example in Tetragon's getting-started guide: it hooks `tcp_connect` and matches connections to the metadata address from pods in the payments namespace. We validated it against the Tetragon 1.7.1 TracingPolicyNamespaced schema but did not run it; Tetragon does not run under Docker Desktop on macOS. On AWS, also cover the IPv6 metadata address `fd00:ec2::254` if your instances enable it. ```yaml apiVersion: cilium.io/v1alpha1 kind: TracingPolicyNamespaced metadata: name: block-metadata-service namespace: payments spec: options: - name: policy-mode value: monitor # report only; change to enforce after a clean observation period kprobes: - call: tcp_connect syscall: false args: - index: 0 type: sock selectors: - matchArgs: - index: 0 operator: DAddr values: - 169.254.169.254 matchActions: - action: Sigkill ``` ## Use the provider's agent on EKS, AKS and GKE - GuardDuty Runtime Monitoring covers EKS, ECS on Fargate and EC2, but not EKS clusters on Fargate. With automated agent configuration GuardDuty installs and updates the `aws-guardduty-agent` add-on itself. Its findings include `Execution:Runtime/NewBinaryExecuted`, `Execution:Runtime/ReverseShell`, `PrivilegeEscalation:Runtime/ContainerMountsHostDirectory` and `UnauthorizedAccess:Runtime/MetadataDNSRebind`. - GuardDuty Extended Threat Detection correlates signals into `AttackSequence:EKS/CompromisedCluster` findings when EKS Protection, Runtime Monitoring with the EKS add-on, or both are on. It ignores archived findings, so broad suppression rules can hide attack sequences. - Defender for Containers runs a sensor DaemonSet and analyzes Kubernetes audit logs on AKS, EKS, GKE and Arc-enabled clusters. On AKS, EKS and GKE, binary drift detection and blocking flag or stop processes that did not come from the image, but the default drift rule ignores drift until you change it. - Security Command Center Container Threat Detection needs the Premium or Enterprise tier. Detectors such as Added Binary Executed, Added Library Loaded and Find Google Cloud Credentials are disabled by default. - Whichever agent you use, send its findings to the same queue as your own detections, with the cluster, namespace, pod and image attached. ## Contain the pod, keep the evidence, then replace it Deleting a compromised pod destroys its memory, its writable layer and often its logs, and the Deployment starts a fresh copy with the same weakness. Isolate it instead. Removing the label that its ReplicaSet and Services select on detaches it, so a clean replica takes its place. A quarantine NetworkPolicy then denies all of its traffic, provided no other policy selects it. Cordon the node, and record the pod spec and logs before anything changes. Then follow the authority the attacker could use: the pod's service account and its RBAC bindings, its cloud identity (IRSA, EKS Pod Identity, a managed identity or a Google service account) and any secrets mounted into it. The runtime detection and response guide at /domains/runtime/detection/runtime-detection-response/ covers collecting Kubernetes and CloudTrail audit evidence and ordering events by when they happened. Falco Talon, an incubating Falco project, can run actions such as labeling or terminating a pod from Falco alerts. Automate isolation only for high-confidence rules. ```bash # 1. Detach the pod: dropping the app label releases it from its ReplicaSet (a clean # replica starts), from Services, and from policies that select app=api. kubectl -n payments label pod api-7d9c4 app- security.example.com/quarantine=true # 2. Deny all traffic to and from quarantined pods. NetworkPolicies add up, so this # only isolates a pod that no other policy selects; step 1 makes sure of that. kubectl apply -f - <<'EOF' apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: quarantine namespace: payments spec: podSelector: matchLabels: security.example.com/quarantine: "true" policyTypes: [Ingress, Egress] EOF # 3. Keep the node for evidence and stop new work landing on it. NODE=$(kubectl -n payments get pod api-7d9c4 -o jsonpath='{.spec.nodeName}') kubectl cordon "$NODE" # 4. Record what you can see before anything is deleted. kubectl -n payments get pod api-7d9c4 -o yaml > evidence/pod.yaml kubectl -n payments logs api-7d9c4 --all-containers --timestamps > evidence/logs.txt ``` ## Pitfalls - Trusting a rule you never triggered. Keep a trigger for every rule and replay it after each upgrade; Falco's event-generator covers many upstream rules. - Disabling a noisy rule cluster-wide instead of excepting one image in one namespace. - Keying exceptions on mutable or ambiguous values: tags, pod name prefixes, or an image name that resolves to a shared image. - Running the sensor without a health check, so a crash or an eviction looks like a quiet week. - Letting every namespace exec into pods. A shell is normal during an incident and suspicious at any other time; restrict `pods/exec` in RBAC so the alert means something. - Enforcing a Tetragon policy straight away. A `Sigkill` on the wrong process is an outage. ## Guided practice Falco and the Kubernetes audit log produced the events below in a production cluster within four minutes. Decide which alerts are real, what to tune, and how to contain the pod without losing evidence. ### Supplied material ```text Falco alerts (JSON output, abridged) F1 10:02:11 Terminal shell in container ns=payments pod=api-7d9c4 image=ghcr.io/acme/api command=sh -c "id; cat /var/run/secrets/kubernetes.io/serviceaccount/token" F2 10:02:40 Drop and execute new binary ns=payments pod=api-7d9c4 proc_exe=/tmp/kworker F3 10:02:52 Container contacts metadata svc ns=payments pod=api-7d9c4 target=169.254.169.254:80 result=ERESTARTSYS F4 10:03:05 Terminal shell in container ns=platform pod=debug-5f6 image=registry.example.com/platform/debug-tools user=alice F5 10:04:30 Redirect STDOUT/STDIN to Network ns=monitoring pod=blackbox-exporter-2 process=ping Kubernetes audit log K1 10:02:09 create pods/exec user=system:serviceaccount:ci:deploy-bot ns=payments pod=api-7d9c4 K2 10:03:01 create pods/exec user=alice@example.com ns=platform pod=debug-5f6 (change ticket CHG-2231) Context - api-7d9c4 uses EKS Pod Identity; IMDSv2 is required with hop limit 1 on all nodes. - deploy-bot is the CI service account; it deploys with kubectl apply and has no documented need for exec. - debug-tools is the approved debugging image for the platform team. ``` ### 1. Correlate the alerts Group the events by pod and order them by time. Which alerts describe one intrusion? Worked answer: F1, F2 and F3 are one sequence in api-7d9c4: an exec by deploy-bot (K1) two seconds before F1, a service account token read, a binary dropped and run from `/tmp`, then a metadata request. F4 matches K2 and its ticket. F5 is a monitoring probe. ### 2. Judge F3 The metadata connection failed. Does F3 still matter? Worked answer: Yes. The pod uses EKS Pod Identity and has no reason to call the metadata service; the failed call shows intent and fits the sequence. IMDSv2 and the hop limit blocked it, which is why the rule watches attempts as well as completed connections. ### 3. Tune without blinding Decide what to do about F4 and F5. Worked answer: F4 is approved work. Add an exception for the debug-tools image repository in the platform namespace, not a broader one. F5: confirm the process is the exporter's ping, then except that process in that image and namespace. Neither rule is disabled. ### 4. Contain List the containment steps in order and say what you would not do first. Worked answer: Remove the app label from api-7d9c4 and apply the quarantine NetworkPolicy, cordon its node and record the pod spec and logs. Revoke or rotate the deploy-bot token and review its RBAC; it should not have `pods/exec`. Rotate secrets the pod could read. Do not delete the pod or restart the node first. ### 5. Close the detection gap Name one detection or control that would have caught this earlier or stopped it. Worked answer: Alert on `pods/exec` by service accounts in the audit log, remove exec from deploy-bot's role, and add a Tetragon or Defender drift policy that blocks binaries not in the image for the payments namespace, starting in monitor mode. Deliverable: An incident note that groups F1 to F5 by cause, separates the intrusion from approved and benign activity, lists the containment steps in order and proposes one exception, one RBAC change and one new detection. Scope: The alerts are realistic but supplied; the Falco rules and exception on this page were tested with Falco 0.45.0 under Docker, not in a production cluster. Articles and teaching material: CC BY 4.0. Original executable code: MIT. Attribution: Software Security. ## Sources - [Falco: rules basics](https://falco.org/docs/concepts/rules/basic-elements/) - [Falco: overriding rules](https://falco.org/docs/concepts/rules/overriding/) - [falcosecurity/rules: falco_rules.yaml (5.2.0)](https://github.com/falcosecurity/rules/blob/falco-rules-5.2.0/rules/falco_rules.yaml) - [Falco Helm chart 9.2.0 (Falco 0.45.0)](https://github.com/falcosecurity/charts/tree/falco-9.2.0/charts/falco) - [Tetragon: policy mode (monitor and enforce)](https://tetragon.io/docs/concepts/tracing-policy/mode/) - [Tetragon: policy enforcement](https://tetragon.io/docs/getting-started/enforcement/) - [Tetragon: selectors](https://tetragon.io/docs/concepts/tracing-policy/selectors/) - [Tracee: policies](https://aquasecurity.github.io/tracee/latest/docs/policies/) - [Amazon GuardDuty: Runtime Monitoring](https://docs.aws.amazon.com/guardduty/latest/ug/runtime-monitoring.html) - [Amazon GuardDuty: Runtime Monitoring finding types](https://docs.aws.amazon.com/guardduty/latest/ug/findings-runtime-monitoring.html) - [Microsoft Defender for Containers overview](https://learn.microsoft.com/en-us/azure/defender-for-cloud/defender-for-containers-introduction) - [Google Cloud: Container Threat Detection overview](https://docs.cloud.google.com/security-command-center/docs/concepts-container-threat-detection-overview) - [GKE: posture management deprecations](https://docs.cloud.google.com/kubernetes-engine/docs/deprecations/posture-management-deprecation) - [MITRE ATT&CK T1552.005: Cloud Instance Metadata API](https://attack.mitre.org/techniques/T1552/005/)