Contain cascading failures and rogue agents

The OWASP Top 10 for Agentic Applications 2026 names two failure modes that this guide contains. Cascading Failures (ASI08:2026) is a single fault, such as a hallucination, malicious input, corrupted tool or poisoned memory, propagating across autonomous agents and compounding into system-wide harm; each hop acts on it with its own authority, often faster than a person can follow. The entry covers propagation and amplification, not the original defect, which has its own item, such as ASI04 for the supply chain or ASI06 for memory poisoning. Rogue Agents (ASI10:2026) are malicious or compromised agents that deviate from their intended function or authorized scope; OWASP’s examples are goal drift and scheming, workflow hijacking, collusion and self-replication, and reward hacking. The first is about how far a fault travels; the second is about an agent that keeps acting once it should have stopped.

Neither is fixed by better prompting, because the agent that has failed is the component you can no longer trust to follow instructions. Containment has to live in components the agent cannot change: the tool gateway that executes its calls, a flag store it cannot write, the credentials it is issued and the data layer it writes to. Design those controls before the incident, test them on a schedule and give on-call engineers the authority and tooling to use them.

Prevention controls belong to other guides: the tool gateway in /domains/ai-emerging/agent-security/tool-authorization/, injection boundaries in /domains/ai-emerging/llm-security/prompt-injection-boundaries/ and inter-agent trust in /domains/ai-emerging/agent-security/mcp-multiagent-security/. This guide assumes one of them has failed.

A support agent may issue account credits of up to 50.00 per ticket. A billing timeout makes the agent believe its first credit failed, so it proposes the same issue_credit call again and again; a second agent that reconciles tickets reads each new credit as a fresh request and asks for more. Nothing in either prompt is wrong. The failure is that every proposal reaches billing with the agent’s full authority.

The code below holds the containment state the gateway checks. KillSwitch reads stop flags from a store the agent has no credential to write, at four scopes, and treats an unreadable store as a stop. RunBudget reserves each step against limits on steps, cost, time and records changed, and refuses the fourth call with the same tool and canonical arguments. CircuitBreaker opens after consecutive failures, so a failing dependency is not hammered by every agent at once. With these limits the loop stops after three credits, and the operation log names the two duplicates to reverse.

Keep the state in a shared store in production, keyed by run and agent identity, so every worker sees the same budget, breaker and flags. In-process state, as here, is enough for one worker and for tests.

import json
import time
from collections import Counter
from dataclasses import dataclass, field


class Contained(Exception):
    """Raised before any side effect when a containment control stops a step."""


class KillSwitch:
    """Stop flags in a control store that agents have no credential to write."""

    def __init__(self, read_flags):
        self.read_flags = read_flags  # e.g. a flag service the agent cannot write to

    def check(self, agent_id, tenant, tool):
        try:
            flags = self.read_flags()
        except Exception as error:
            raise Contained("kill switch unreadable: failing closed") from error
        scopes = ("global", f"agent:{agent_id}", f"tenant:{tenant}", f"tool:{tool}")
        for scope in scopes:
            if scope in flags:
                raise Contained(f"stopped by kill switch {scope}")


class CircuitBreaker:
    """Opens after `threshold` consecutive failures; one probe after `cooldown`."""

    def __init__(self, threshold=5, cooldown=60.0, clock=time.monotonic):
        self.threshold, self.cooldown, self.clock = threshold, cooldown, clock
        self.failures, self.opened_at = 0, None

    def before_call(self):
        if self.opened_at is not None:
            if self.clock() - self.opened_at < self.cooldown:
                raise Contained("circuit open")
            # Half-open: allow one probe; its failure re-opens the circuit at once.
            self.opened_at, self.failures = None, self.threshold - 1

    def record(self, ok):
        self.failures = 0 if ok else self.failures + 1
        if self.failures >= self.threshold:
            self.opened_at = self.clock()


@dataclass
class RunBudget:
    """Hard limits for one run. The gateway reserves against them before every step."""

    max_steps: int = 25
    max_cost_cents: int = 200
    max_seconds: float = 600.0
    max_records: int = 100  # blast radius: records one run may change
    max_repeats: int = 3  # identical calls allowed before the run counts as looping
    steps: int = 0
    cost_cents: int = 0
    records: int = 0
    started: float = field(default_factory=time.monotonic)
    calls: Counter = field(default_factory=Counter)

    def reserve(self, tool, arguments, cost_cents, records):
        key = (tool, json.dumps(arguments, sort_keys=True))
        if self.steps >= self.max_steps:
            raise Contained("step limit")
        if time.monotonic() - self.started > self.max_seconds:
            raise Contained("time limit")
        if self.cost_cents + cost_cents > self.max_cost_cents:
            raise Contained("cost limit")
        if self.records + records > self.max_records:
            raise Contained("blast-radius limit")
        if self.calls[key] >= self.max_repeats:
            raise Contained("repeated identical call")
        self.steps += 1
        self.cost_cents += cost_cents
        self.records += records
        self.calls[key] += 1

The gateway applies the checks in a fixed order on every proposal, whatever the model said: kill switch, rate limit per agent identity, circuit state, then the budget reservation with the cost and record count the tool’s contract estimates. A stop raises before the tool runs and is written to the audit log with its reason. The tool receives the tenant from the run, not from the model, as in the tool authorization guide.

The breaker records only real outcomes of the call, and a stop does not spend budget it never used. Retries belong here too: retry only idempotent calls, with backoff and a small fixed count, and reuse the same idempotency key so a retry cannot repeat an effect.

from containment import Contained


def run_step(run, proposal, audit):
    """Every check runs here, outside the model, before the tool is called."""
    tool, args = proposal["tool"], proposal.get("arguments", {})
    breaker = run.breakers[tool]
    try:
        run.kill_switch.check(run.agent_id, run.tenant, tool)
        run.rate_limit.take(run.agent_id)
        breaker.before_call()
        cost, records = run.tools.estimate(tool, args)  # from the tool's contract
        run.budget.reserve(tool, args, cost, records)
    except Contained as stop:
        audit.append({"run": run.id, "tool": tool, "decision": "stopped",
                      "reason": str(stop)})
        raise
    try:
        result = run.tools.call(tool, tenant=run.tenant, **args)
    except Exception:
        breaker.record(ok=False)
        audit.append({"run": run.id, "tool": tool, "decision": "failed"})
        raise
    breaker.record(ok=True)
    audit.append({"run": run.id, "tool": tool, "decision": "executed"})
    return result

• Enforce it outside the agent: the gateway and every queue worker check the flag before each step. An instruction to the model to stop is a request, not a control.
• Give it scopes: global, one agent identity, one tenant and one tool. Stopping only the misbehaving agent keeps the rest of the product running, so people will actually use it.
• Fail closed: if the flag store cannot be read, refuse the step. Cache flags for a few seconds at most, and alert when reads fail. Check the same for any external policy hook: the OWASP Agent Control Standard’s reference Guardian proceeds by default when it cannot decide, until you set ACS_ON_DECISION_FAILURE=deny.
• Back it with credential revocation: revoke or rotate the agent’s tokens and keys, so a call that bypasses the gateway still fails. Apply permission changes at once (ASVS v5.0.0-8.3.2).
• Stop in-flight work: cancel queued jobs and scheduled runs for the stopped scope, and hold any output the agent produced but has not yet delivered.
• Make it fast to use and hard to undo by accident: one command or button for on-call, an audit record of who set or cleared each flag, and a review before clearing.
• Test it: set and clear each scope in a scheduled drill and measure the time from decision to the last refused call.
• Use it as governance evidence: NIST AI RMF MANAGE 2.4 asks for mechanisms to disengage or deactivate AI systems that behave outside their intended use, and the EU AI Act asks high-risk systems for a stop procedure that halts them in a safe state (Article 14(4)(e)); see /domains/ai-emerging/governance/ai-governance-regulation/.

Limits turn an unbounded failure into a bounded one. Set them per run from the task, per agent identity and per tenant, and reserve against them before each step rather than counting afterwards, so concurrent steps cannot all pass the same check. A step limit catches plans that never converge, a wall-clock limit catches calls that hang, a cost limit caps model and tool spend, and a records-changed limit caps how much data one run can alter. The cost and memory guide at /domains/ai-emerging/agent-security/cost-memory-operations/ covers budget reservation and reconciliation in depth.

OWASP lists these limits under Unbounded Consumption (LLM06:2026) as agentic circuit breakers: step limits, recursion depth limits, time limits and per-run cost ceilings on every agent execution, with state hashing to detect recursive loops. Tool Misuse and Exploitation (ASI02:2026) names the same failure as loop amplification, a planner that keeps calling costly APIs.

Rate-limit each agent identity with a token bucket, separately from user-facing rate limits, so one agent cannot starve others of a shared dependency. Limit delegation too: a child agent receives a share of the parent’s remaining budget and a subset of its permissions, depth is capped, and the total across the delegation tree is enforced in one place. A helper that can spawn helpers, each with a fresh budget, is a cascading failure waiting to happen.

Detect loops directly. Count calls by tool and canonical arguments, and stop the run when the same call repeats past a small threshold; also stop when a run alternates between a few calls without progress, measured by a task-specific signal such as tickets closed. Provider-side spend limits on the model API are a backstop, not the control, because they stop every agent at once.

A circuit breaker stops a failing dependency from turning into a failing system. Put one in front of each downstream service and each peer agent an agent calls. While it is closed, calls pass and failures are counted; after a threshold it opens and calls fail at once without reaching the dependency; after a cool-down it lets one probe through and closes again only if the probe succeeds. ASVS v5.0.0-16.5.2 asks for this pattern, or graceful degradation, when external resources fail.

Decide what open means for each tool. Reads can fall back to a cached or degraded answer; writes and external effects should be refused, because an agent that cannot confirm a write is the agent most likely to repeat it. Trip breakers on more than errors: a peer agent whose outputs repeatedly fail schema validation, or whose answers your checks reject, is also a failing dependency, and stopping the flow there keeps one agent’s wrong output from becoming every other agent’s input.

• Give each agent its own identity and short-lived credentials scoped to its tools and tenant, so stopping or revoking one agent does not stop the others, and a compromised agent cannot act as its peers.
• Cap the effect of a single operation: records changed, amount moved, messages sent, recipients reached. Refuse bulk operations above the cap without a human approval.
• Allowlist destinations per agent for outbound requests, email and file shares; an agent that suddenly needs a new destination needs a review, not a retry.
• Isolate shared state: validate what one agent writes to shared memory or a queue before another agent reads it, and keep memory per tenant and per task where you can.
• Roll out changes to models, prompts, tools and agent code gradually, to a share of traffic or tenants, with the containment metrics below as the promotion gate.
• Prefer reversible operations: soft deletes, pending states, ledger entries that can be reversed, and an outbox that holds external messages briefly before they leave.

Give every agent a manifest: the tools it may propose, the destinations it may reach, its normal call rate and the data it handles. The gateway already writes each proposal, decision and effect to the audit log; ship that log to a store the agent cannot modify, outside the agent’s trust domain (ASVS v5.0.0-16.4.3), and evaluate it there. A monitor that runs inside the agent’s own context or with its credentials can be steered by the same input that steered the agent.

Start with deterministic rules before statistical anomaly detection, because they are explainable at 3 a.m. Denied and stopped proposals are the earliest signal: a hijacked agent usually tries something it is not allowed to do before it finds something it is. Then alert on tools and destinations outside the manifest, call rates far above baseline, sudden changes in the data volume an agent reads or writes, and a rising rate of failed validations. The function below flags one agent’s last minute of events; any flag should quarantine the identity automatically and page its owner, with /domains/runtime/detection/security-event-logging/ for the log design.

Add canaries where they are cheap: a honeytoken record or credential that no legitimate task touches, so any use of it is a reliable alert. Review the manifest and baselines whenever the agent, its model or its tools change.

from collections import Counter


def review(events, manifest, baseline_per_minute):
    """Flags one agent's last minute of gateway events against its declared manifest."""
    flags = []
    tools = Counter(event["tool"] for event in events)
    outside = sorted(set(tools) - manifest["tools"])
    if outside:
        flags.append(f"proposed tools outside its manifest: {outside}")
    hosts = {event["host"] for event in events if event.get("host")} - manifest["hosts"]
    if hosts:
        flags.append(f"new destinations: {sorted(hosts)}")
    if len(events) > 3 * baseline_per_minute:
        flags.append("call rate above three times its baseline")
    stopped = sum(event["decision"] in ("denied", "stopped") for event in events)
    if stopped >= 5:
        flags.append("repeated denied or stopped proposals")
    return flags  # any flag: quarantine the agent identity and page the owner

Human-in-the-loop gates belong where the cost of a wrong action is high and undoing it is hard: moving money, deleting data, changing access, sending data outside the organization, bulk operations above the blast-radius cap and any action outside the manifest. The gateway asks for the approval, binds it to the exact arguments and refuses the action if no decision arrives before a timeout. A gate that defaults to yes when the approver is busy is not a gate.

Too many approvals train people to click through them, and an agent can exploit that trust (Human-Agent Trust Exploitation, ASI09:2026). Keep gates for consequential actions, show the evidence and the exact effect rather than the agent’s summary of it, and route anomalies flagged by the monitor to a person with authority to set the kill switch.

Quarantine is a stop that preserves evidence, and OWASP’s ASI10 guidance pairs kill switches and credential revocation with quarantine for forensic review. Set the kill switch for the agent identity, revoke its credentials, snapshot its memory, conversation state and pending outputs, and mark everything it produced since the first anomaly as untrusted until reviewed, including memory entries and messages other agents may already have read. Freeze, do not delete: you will need the state to find the cause.

Rollback needs preparation. Write each effect to an operation log with the run, the agent identity, the tool, the arguments, the result and a compensating action: reverse a ledger entry, restore a soft-deleted record, revoke a share, send a correction. Then a bad run can be reversed by walking its log backwards. Effects that cannot be compensated, such as data sent outside the organization, are exactly the ones that need a human gate in advance.

Reintegrate deliberately, with a person approving the return, as ASI10 recommends. Fix the cause, add the incident as a regression case, issue fresh credentials, purge or restore poisoned memory, and re-enable the agent for a small share of traffic with the monitor watching before clearing the flag everywhere.

Agent incidents follow the usual incident life cycle, with three differences: the system keeps acting while you investigate, its evidence includes prompts, model and tool versions and memory, and the root cause may be content rather than code. Write the playbook before you need it, and rehearse it as a tabletop exercise; /domains/appsec/vulnerability-response/product-incident-response/ covers the general plan.

• Detect: a monitor flag, a stopped-proposal spike, a breaker that stays open, a customer report or a cost alert opens the incident.
• Contain: set the narrowest kill-switch scope that stops the harm, revoke the agent’s credentials, and widen the scope if the signal continues.
• Preserve: export the run’s audit and operation logs, prompts, retrieved content, tool results, memory snapshot, and the model, prompt and tool versions in use.
• Scope: list every effect from the operation log, and every agent or system that consumed the agent’s output, to find how far the failure cascaded.
• Recover: run the compensating actions, correct or purge poisoned memory and shared state, and notify affected users where the effects reached them.
• Learn: find the entry point (injection, poisoned data, model or tool change, bad limit), add it as a test, adjust the limits and manifest, and reintegrate gradually.

A system prompt that says “stop after five attempts” is a suggestion to the component that is failing. A hijacked or looping agent may ignore it, and a model update can change how it is read. Limits that matter are counted by the gateway.

An agent’s report of what it did is evidence about its output, not about its effects. Judge a run by the operation log and the state of the systems it touched, and do not let an agent’s summary close an incident. Likewise, a monitor or supervisor agent that reads the same untrusted content, runs in the same context or holds the same credentials can fail in the same way; keep at least one detector deterministic and outside the agent’s trust domain.

Each row is a case from the credit example and the result the containment controls must produce.

Sources & further reading

The standards, specifications and documentation behind the guidance above. Each is the evidence for a claim on this page; consult it for detail beyond the defaults and versions listed here.

1. OWASP Top 10 for Agentic Applications for 2026 (ASI08 Cascading Failures, ASI10 Rogue Agents). Version 2026 (December 2025).
2. OWASP AI Agent Security Cheat Sheet
3. OWASP Top 10 for LLM Applications 2026 (LLM06 Unbounded Consumption). 2026 edition (3 August 2026).
4. OWASP ASVS 5.0.0, V16 Security Logging and Error Handling. Version 5.0.0.
5. Google SRE book: Addressing Cascading Failures
6. Martin Fowler: CircuitBreaker. 6 March 2014.
7. OWASP Agent Control Standard: README (failure posture, specification v0.1.0)
8. NIST SP 800-61 Rev. 3: Incident Response Recommendations and Considerations for Cybersecurity Risk Management. April 2025.

Reference information

Tags

Incident responseAI agents and tools

SDLC phase

DesignDevelopmentMaintenance

Role

Platform engineers and SREs

ASVS 5.0.0 requirements

v5.0.0-2.4.1 · v5.0.0-8.3.2 · v5.0.0-16.4.3 · v5.0.0-16.5.2

Download this guide as Markdown

Continue exploring