Detect container and host threats at runtime with eBPF sensors and cloud services

Scanners judge an image before it runs and audit logs record calls to the Kubernetes and cloud APIs. Neither sees what a process does inside a running container. A runtime sensor does: it attaches eBPF programs, or a kernel module, to system calls and kernel functions on each node, enriches each event with the container, pod and image it came from, and matches it against rules.

That view catches the middle of an attack: a reverse shell after a deserialization bug, a cryptominer written to /tmp and executed, a read of a service account token, a request for cloud credentials from the instance metadata service, or an escape attempt through a writable host mount. These map to MITRE ATT&CK techniques such as T1609 container administration command, T1611 escape to host and T1552.005 cloud instance metadata API.

The sensor runs as a privileged DaemonSet with access to the host kernel, so treat it as part of the node's trusted base: pin its image, restrict who can change its configuration, and alert when it stops reporting. An attacker with root on the node can blind it.

Open-source sensors give you the rules and the data; managed agents give you detections written and maintained by the provider and findings in its console. Many teams run both, with the provider's agent for coverage and Falco or Tetragon rules for their own architecture. Check the details below against the current documentation before you buy or deploy.

Google retired the Preview GKE threat detection feature in the GKE security posture dashboard on 31 March 2025. On Google Cloud, runtime detection now comes from Security Command Center: Container Threat Detection on nodes and Event Threat Detection on logs. The Enterprise tier of Security Command Center shuts down on 21 May 2027, and its customers move to Premium.

Install the chart pinned to a version, keep the upstream rules file untouched, and put your rules in a separate file that loads after it. The chart's customRules value turns each key into a file under /etc/falco/rules.d. Setting falcosidekick.enabled=true also switches Falco to JSON output and deploys Falcosidekick, which forwards alerts to chat, a SIEM or a webhook.

We rendered this chart, version 9.2.0 for Falco 0.45.0, with Helm 4.3 and confirmed the DaemonSet, the modern_ebpf engine setting, JSON output and the rules file in the generated configuration. Validate rules files in CI with falco -V: it loads them with the upstream rules and reports each load error with the file, rule and position, such as an undefined macro.

helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update

# custom-rules.yaml holds your rules under customRules: (one key per file).
helm install falco falcosecurity/falco --version 9.2.0 \
  --namespace falco --create-namespace \
  --set driver.kind=modern_ebpf \
  --set falcosidekick.enabled=true \
  -f custom-rules.yaml

# In CI: every rules file must load cleanly next to the upstream rules.
docker run --rm -v "$PWD/rules:/rules:ro" falcosecurity/falco:0.45.0 \
  falco -V /etc/falco/falco_rules.yaml -V /rules/rules-runtime.yaml

Start from your threat model, not the rule catalog. A pod that uses IRSA, EKS Pod Identity or Workload Identity Federation never needs the node's metadata service, so a connection to 169.254.169.254 from such a pod is worth an alert; it often follows a server-side request forgery bug of the kind the SSRF guide at /domains/appsec/secure-features/ssrf-egress-boundaries/ covers. The rule below uses the container macro from the upstream rules and an allowlist for the few images that legitimately call it.

We ran this rule on Falco 0.45.0 with the modern eBPF driver, against a test server bound to 169.254.169.254. A successful wget and an nc from ordinary containers both alerted, with target=169.254.169.254:80 result=SUCCESS. The first version matched only on fd.sip, the server address of an established connection. It missed a request that never completed: the connect event arrived with res=ERESTARTSYS and no fd.sip, but evt.arg.addr still held the destination. Blocked attempts matter, because a pod probing the metadata service is suspicious even when IMDSv2 or a hop limit stops it.

Our test containers ran under Docker, so k8s.ns.name and k8s.pod.name printed <NA>. In a cluster the container plugin fills them from the container runtime.

# rules-runtime.yaml: loaded after falco_rules.yaml, so it can extend upstream rules.
- list: imds_allowed_images
  items: [public.ecr.aws/aws-observability/aws-for-fluent-bit]

- rule: Container contacts cloud metadata service
  desc: >
    A process in a container tried to connect to the instance metadata address.
    Pods with their own workload identity never need it; a connection here
    often follows SSRF or an interactive foothold.
  condition: >
    evt.type = connect
    and container
    and (fd.sip = "169.254.169.254" or evt.arg.addr startswith "169.254.169.254:")
    and not container.image.repository in (imds_allowed_images)
  output: >
    Container contacted the metadata service | process=%proc.name
    command=%proc.cmdline image=%container.image.repository
    ns=%k8s.ns.name pod=%k8s.pod.name target=%evt.arg.addr result=%evt.res
  priority: WARNING
  tags: [container, network, mitre_credential_access, T1552.005]

Noise kills detection programs. Tune each noisy rule by naming the specific, expected behavior, and leave the rule on for everything else. Falco's override key appends exceptions or conditions to an upstream rule from your own file, so upgrades to the upstream rules still apply.

We tested the exception below. A shell in a container from registry.example.com/platform/debug-tools stayed quiet, while the same shell in an alpine container still raised Terminal shell in container. A first attempt with a retagged copy of alpine kept alerting: Falco reported the repository of the underlying image, alpine, not the new name, so the exception never matched. Test exceptions with the images you actually deploy.

The upstream rules also fired where we did not expect them. Redirect STDOUT/STDIN to Network Connection in Container triggered for busybox ping and for the test web server answering requests. Before writing an exception like that, check which process, image and namespace produced the alert, and whether the behavior could also be an attacker's.

# Add a narrow exception to an upstream rule without copying or editing it.
- rule: Terminal shell in container
  exceptions:
    - name: approved_debug_images
      fields: [container.image.repository]
      comps: [=]
      values:
        - [registry.example.com/platform/debug-tools]
  override:
    exceptions: append

Tetragon filters in the kernel and can act there: Sigkill stops the process before the call returns, and Override makes a hooked function return an error. That turns a detection into a control, so start every enforcing policy in monitor mode, where Tetragon reports each match but skips the action, and switch it to enforce only after it has run cleanly on real traffic. tetra tracingpolicy list shows each policy's mode.

This namespaced policy adapts the network example in Tetragon's getting-started guide: it hooks tcp_connect and matches connections to the metadata address from pods in the payments namespace. We validated it against the Tetragon 1.7.1 TracingPolicyNamespaced schema but did not run it; Tetragon does not run under Docker Desktop on macOS. On AWS, also cover the IPv6 metadata address fd00:ec2::254 if your instances enable it.

apiVersion: cilium.io/v1alpha1
kind: TracingPolicyNamespaced
metadata:
  name: block-metadata-service
  namespace: payments
spec:
  options:
    - name: policy-mode
      value: monitor # report only; change to enforce after a clean observation period
  kprobes:
    - call: tcp_connect
      syscall: false
      args:
        - index: 0
          type: sock
      selectors:
        - matchArgs:
            - index: 0
              operator: DAddr
              values:
                - 169.254.169.254
          matchActions:
            - action: Sigkill

• GuardDuty Runtime Monitoring covers EKS, ECS on Fargate and EC2, but not EKS clusters on Fargate. With automated agent configuration GuardDuty installs and updates the aws-guardduty-agent add-on itself. Its findings include Execution:Runtime/NewBinaryExecuted, Execution:Runtime/ReverseShell, PrivilegeEscalation:Runtime/ContainerMountsHostDirectory and UnauthorizedAccess:Runtime/MetadataDNSRebind.
• GuardDuty Extended Threat Detection correlates signals into AttackSequence:EKS/CompromisedCluster findings when EKS Protection, Runtime Monitoring with the EKS add-on, or both are on. It ignores archived findings, so broad suppression rules can hide attack sequences.
• Defender for Containers runs a sensor DaemonSet and analyzes Kubernetes audit logs on AKS, EKS, GKE and Arc-enabled clusters. On AKS, EKS and GKE, binary drift detection and blocking flag or stop processes that did not come from the image, but the default drift rule ignores drift until you change it.
• Security Command Center Container Threat Detection needs the Premium or Enterprise tier. Detectors such as Added Binary Executed, Added Library Loaded and Find Google Cloud Credentials are disabled by default.
• Whichever agent you use, send its findings to the same queue as your own detections, with the cluster, namespace, pod and image attached.

Deleting a compromised pod destroys its memory, its writable layer and often its logs, and the Deployment starts a fresh copy with the same weakness. Isolate it instead. Removing the label that its ReplicaSet and Services select on detaches it, so a clean replica takes its place. A quarantine NetworkPolicy then denies all of its traffic, provided no other policy selects it. Cordon the node, and record the pod spec and logs before anything changes.

Then follow the authority the attacker could use: the pod's service account and its RBAC bindings, its cloud identity (IRSA, EKS Pod Identity, a managed identity or a Google service account) and any secrets mounted into it. The runtime detection and response guide at /domains/runtime/detection/runtime-detection-response/ covers collecting Kubernetes and CloudTrail audit evidence and ordering events by when they happened. Falco Talon, an incubating Falco project, can run actions such as labeling or terminating a pod from Falco alerts. Automate isolation only for high-confidence rules.

# 1. Detach the pod: dropping the app label releases it from its ReplicaSet (a clean
#    replica starts), from Services, and from policies that select app=api.
kubectl -n payments label pod api-7d9c4 app- security.example.com/quarantine=true

# 2. Deny all traffic to and from quarantined pods. NetworkPolicies add up, so this
#    only isolates a pod that no other policy selects; step 1 makes sure of that.
kubectl apply -f - <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: quarantine
  namespace: payments
spec:
  podSelector:
    matchLabels:
      security.example.com/quarantine: "true"
  policyTypes: [Ingress, Egress]
EOF

# 3. Keep the node for evidence and stop new work landing on it.
NODE=$(kubectl -n payments get pod api-7d9c4 -o jsonpath='{.spec.nodeName}')
kubectl cordon "$NODE"

# 4. Record what you can see before anything is deleted.
kubectl -n payments get pod api-7d9c4 -o yaml > evidence/pod.yaml
kubectl -n payments logs api-7d9c4 --all-containers --timestamps > evidence/logs.txt

• Trusting a rule you never triggered. Keep a trigger for every rule and replay it after each upgrade; Falco's event-generator covers many upstream rules.
• Disabling a noisy rule cluster-wide instead of excepting one image in one namespace.
• Keying exceptions on mutable or ambiguous values: tags, pod name prefixes, or an image name that resolves to a shared image.
• Running the sensor without a health check, so a crash or an eviction looks like a quiet week.
• Letting every namespace exec into pods. A shell is normal during an incident and suspicious at any other time; restrict pods/exec in RBAC so the alert means something.
• Enforcing a Tetragon policy straight away. A Sigkill on the wrong process is an outage.

Sources & further reading

The standards, specifications and documentation behind the guidance above. Each is the evidence for a claim on this page; consult it for detail beyond the defaults and versions listed here.

1. Falco: rules basics
2. Falco: overriding rules
3. falcosecurity/rules: falco_rules.yaml (5.2.0)
4. Falco Helm chart 9.2.0 (Falco 0.45.0)
5. Tetragon: policy mode (monitor and enforce)
6. Tetragon: policy enforcement
7. Tetragon: selectors
8. Tracee: policies
9. Amazon GuardDuty: Runtime Monitoring
10. Amazon GuardDuty: Runtime Monitoring finding types
11. Microsoft Defender for Containers overview
12. Google Cloud: Container Threat Detection overview
13. GKE: posture management deprecations
14. MITRE ATT&CK T1552.005: Cloud Instance Metadata API

Reference information

Tags

Cloud securityKubernetes and containersLogging and detectionIncident response

SDLC phase

Maintenance

Role

Platform engineers and SREs

Download this guide as Markdown

Continue exploring