This is a worked threat model for an LLM agent: a chat UI, an orchestrator, a hosted LLM, retrieval over company documents, tools (email, tickets, a code sandbox, web fetch), an MCP server, long-term memory and a human approval step. Model it by drawing the data flows, then enumerating the agent-specific threats: direct and indirect prompt injection through retrieved documents, web pages, emails and tool output; excessive agency and tool misuse; data exfiltration through tool calls and rendered links; confused-deputy use of the agent's privileges; cross-tenant retrieval leakage; memory and MCP-tool poisoning; unbounded cost; and unsafe output handling. The decisive mitigations enforce authority outside the model: a tool gateway with per-user authorization, least-privilege scoped credentials, allowlisted egress, human approval for consequential actions, and output encoding. The threats map to two OWASP lists: the Top 10 for LLM Applications 2026 and the Top 10 for Agentic Applications 2026.
- Every input the model reads is untrusted, including retrieved documents, fetched web pages, emails and the output of tools; indirect prompt injection is the central threat (LLM01:2026, ASI01:2026).
- The model cannot be the authorization boundary: enforce who-can-do-what in a tool gateway outside the model, with least-privilege scoped credentials (ASI02/ASI03:2026).
- Exfiltration happens through the agent's own tools and through rendered links and image URLs; allowlist egress and encode output (LLM02/LLM10:2026).
- A confused deputy acts with the agent's privileges on an attacker's behalf; bind actions to the requesting user's authority, not the agent's.
- Memory, retrieval and MCP tool descriptions are attack surface: poison them and you steer the agent later (ASI06:2026, ASI04:2026).
- Require human approval for consequential actions with the full parameters shown, bound cost and tools, and log every tool call for audit.
| Setting | Value | Note | Source |
|---|---|---|---|
| Trust of model inputs | All untrusted: user text, retrieved docs, web pages, emails, tool output | Indirect prompt injection rides in on any of these. | OWASP LLM01:2026; ASI01:2026 |
| Tool authorization | Enforced in a gateway outside the model, per requesting user | The model proposes; the gateway resolves the real actor, tool and scope and decides. | OWASP ASI02:2026, ASI03:2026 |
| Tool credentials | Least-privilege, scoped per tool, short-lived | A compromised or hijacked agent can only reach what its credentials allow. | OWASP ASI03:2026 |
| Egress | Allowlisted destinations for web fetch and outbound tools | Stops exfiltration to attacker hosts and SSRF to internal addresses. | LLM02:2026; ssrf-egress-boundaries |
| Consequential actions | Human approval with the exact parameters shown | Send email, move money, delete, publish, run code: a person confirms what will happen. | OWASP ASI02:2026; agent-tool-authorization |
| Output handling | Encoded for its sink; never executed; links and images sanitized | Model output rendered as HTML is XSS; a markdown image URL is an exfiltration channel. | OWASP LLM10:2026 Improper Output Handling |
| Retrieval and memory | Tenant-scoped retrieval; memory writes carry provenance | Prevents cross-tenant leakage and memory poisoning steering later turns. | OWASP LLM09:2026; ASI06:2026 |
| Cost and tools | Per-user budget and a bounded, vetted tool set | Caps unbounded consumption; MCP servers and tools are vetted before enabling. | OWASP LLM06:2026; ASI04:2026 |
- Walk each flow in the diagram and confirm that a malicious string in any model input (document, web page, email, tool result) cannot by itself cause a tool call to run without the gateway's authorization.
- Attempt an indirect injection: plant instructions in a retrieved document and a fetched page and confirm the agent does not exfiltrate data or call a tool outside its policy.
- Confirm every consequential tool call is authorized against the requesting user and requires human approval with the real parameters shown, not a blind approve/deny.
- Test exfiltration channels: a rendered link, a markdown image URL, and a tool call to an off-allowlist host, and confirm each is blocked or stripped.
- Cross-tenant test: ask the agent for another tenant's data and confirm retrieval returns nothing and no tool reaches it.
- Confirm per-user cost budgets, bounded tool use, provenance on memory writes, and an audit log of every tool call with its arguments and decision.
- Applies to
- OWASP Top 10 for LLM Applications 2026 edition
- OWASP Top 10 for Agentic Applications 2026 edition
Applications where an LLM plans and calls tools (RAG, email, tickets, code execution, web fetch, MCP) on behalf of a user. Maps to the OWASP Top 10 for LLM Applications 2026 and the OWASP Top 10 for Agentic Applications 2026.
- Part of
- Workflow: Review a design
Tested fixes: the vulnerable code, the fix, a plausible fix that still fails, and the regression tests that tell them apart.
Learning goals and prerequisites
What you will learn
- Draw the data flows of an LLM agent with tools, retrieval, memory and MCP.
- Enumerate the agent-specific threats, including indirect prompt injection and exfiltration.
- Choose mitigations that enforce authority outside the model.
- Map the threats to the OWASP Top 10 for LLM Applications 2026 and the OWASP Top 10 for Agentic Applications 2026.
Before you start
Engineers building LLM agents and reviewers doing a design review of an agentic feature.
- Read a data-flow diagram and know what a trust boundary is.
- Understand prompt injection and tool use in LLM applications.
Refresh first: Choose a threat modeling method and turn its findings into tests
Go to the guided practiceData flows of an LLM agent with tools
- User asksThe user sends a request through the chat UI to the orchestrator; this is untrusted input.
- Plan with contextThe orchestrator retrieves documents and memory and calls the LLM; retrieved content and prior tool output are also untrusted.
- Propose a tool callThe LLM proposes a tool call; the orchestrator sends it to the tool gateway as a proposal, never a command.
- Authorize and runThe gateway resolves the requesting user, checks policy and scope, requires human approval for consequential actions, then calls the tool or MCP server.
- Return and renderThe result returns to the model and the user; output is encoded for its sink and links are sanitized.
The system is an assistant that plans and acts. A chat UI sends the user's request to an orchestrator, which calls a hosted LLM, retrieves from a store of company documents (RAG), and can call tools: send email, file and update tickets, run code in a sandbox, and fetch web pages. It talks to external systems through an MCP server, keeps long-term memory across sessions, and has a human-approval step for some actions.
State the assumptions. Everything the model reads is untrusted: the user's text, retrieved documents, fetched web pages, incoming emails, tool output, memory and MCP tool descriptions can all carry instructions. The model has no reliable way to tell data from instructions, so it cannot be the place authority is decided. The hosted LLM and the retrieval store are inside the application's boundary but their inputs are not.
| Element | Data it handles | Trust |
|---|---|---|
| Chat UI | User request, rendered output | User input untrusted; output needs encoding |
| Orchestrator + LLM | Prompt, context, proposed tool calls | Proposals are untrusted |
| Retrieval store | Company documents, embeddings | Content untrusted; must be tenant-scoped |
| Memory | Facts kept across sessions | Writable by earlier turns; poisonable |
| Tool gateway | Authorization, scope, approval | The enforcement point |
| Tools / MCP | Email, tickets, code, web fetch | Each a capability with its own blast radius |
Walk the threats element by element. Unlike a classic app, the dangerous input often arrives indirectly, through content the agent reads while doing its job.
| Threat | Example | OWASP 2026 |
|---|---|---|
| Direct prompt injection | The user tells the agent to ignore its rules | LLM01:2026 |
| Indirect prompt injection | A retrieved document, web page or email contains instructions the agent follows | LLM01:2026, ASI01:2026 |
| Excessive agency / tool misuse | The agent calls a powerful tool it shouldn't for this request | LLM03:2026, ASI02:2026 |
| Privilege / identity abuse (confused deputy) | The agent uses its own broad credentials on an attacker's behalf | ASI03:2026 |
| Data exfiltration | Data leaves via a tool call, a rendered link or a markdown image URL | LLM02:2026 |
| Cross-tenant leakage | Retrieval returns another tenant's documents | LLM09:2026 |
| Memory poisoning | A planted memory steers later sessions | ASI06:2026 |
| Supply chain / MCP poisoning | A malicious tool description or server changes behavior | ASI04:2026 |
| Unbounded consumption | A loop or large context runs up cost | LLM06:2026 |
| Improper output handling | Model output is rendered as HTML or executed | LLM10:2026 |
You cannot reliably stop a model from being influenced by instructions hidden in the content it reads, so design as if injection will sometimes succeed and limit what a successful injection can achieve. Keep untrusted content clearly separated in the prompt, prefer retrieval over letting the agent fetch arbitrary URLs, and above all make sure a compromised plan still hits the gateway's authorization, egress allowlist and human-approval gates. The prompt-injection guide at /domains/ai-emerging/llm-security/prompt-injection-boundaries/ covers containment; the point for the threat model is that injection is a path to the real impact, not the impact itself.
Data leaves an agent in more ways than an obvious tool call. A fetched or rendered link with data in its query string, a markdown image whose URL the client loads, and a tool that posts to an attacker-chosen address are all exfiltration channels an injection can use. Allowlist egress for web fetch and outbound tools, and re-check a URL that came from model output against the allowlist (the SSRF overlap is covered at /domains/appsec/secure-features/ssrf-egress-boundaries/). Sanitize links and strip or proxy image URLs in rendered output, and encode output for its sink so it cannot execute (LLM10:2026).
Retrieval, memory and MCP are attack surface. Scope every retrieval query to the requesting tenant and user so one customer's question can't return another's documents (/domains/ai-emerging/llm-security/rag-tenant-isolation/; practice it in /labs/rag-isolation/). Treat memory writes as untrusted: record provenance, and don't let the agent silently persist instructions that steer future sessions. Vet MCP servers and their tool descriptions before enabling them, because a poisoned tool description is itself injected context; the MCP guide at /domains/ai-emerging/agent-security/mcp-multiagent-security/ covers vetting and inter-agent trust.
Be honest about what remains. Indirect prompt injection cannot be eliminated, so the model's protection is the authority, egress and approval gates around it; document that and make sure those gates have no bypass. Human approval degrades if people are asked to confirm too much, so reserve it for genuinely consequential actions and show the real parameters. Re-run the model when you add a tool, enable a new MCP server, change the retrieval corpus, or give the agent a new capability; map each threat to a control and a test, and log every tool call so an incident is reconstructable. /domains/ai-emerging/assurance/ai-red-teaming/ turns these threats into repeatable red-team cases mapped to OWASP and MITRE ATLAS IDs, scored by their effects and rerun on every model or provider change. EchoLeak (CVE-2025-32711, June 2025), an AI command-injection flaw in Microsoft 365 Copilot that allowed information disclosure over the network, and the 2025 demonstration of data exfiltration through a GitHub MCP server show these threats are real, not hypothetical.
Guided practice
You are reviewing a customer-support agent before launch. It reads the customer's tickets and a knowledge base, can email the customer, can issue account credits, and can fetch linked pages. The design notes below are from the team. For each, name the threat and OWASP 2026 id and give the fix.
Supplied scenario material
D1 The system prompt says "only issue credits under $50"; the agent decides whether to issue a credit.
D2 The agent fetches any URL found in a ticket to "get more context."
D3 Replies are rendered as HTML in the agent console, including any links and images the model produced.
D4 Retrieval searches the whole knowledge base, which includes other customers' tickets.
D5 The agent stores "useful facts" to long-term memory automatically from each conversation.
D6 A new MCP server was added from a public registry to let the agent look up shipping status.
D7 There is no per-conversation cost or tool-call limit.1. Move the credit decision out of the model
Classify D1 and give the fix.
Expected result and reasoning
Excessive agency (LLM03:2026) and tool misuse (ASI02:2026): a prompt rule is not an authorization boundary; an injection can talk the model past it. Enforce the credit limit and eligibility in the tool gateway against the requesting user, and require human approval above a threshold, with the amount and account shown.
2. Close the fetch channel
Classify D2 and give the fix.
Expected result and reasoning
Indirect prompt injection and SSRF/exfiltration (LLM01/LLM02:2026): a ticket URL is attacker-controlled; fetching it pulls in injected instructions and can exfiltrate or reach internal hosts. Allowlist egress, re-check model-supplied URLs, and prefer retrieval over arbitrary fetch.
3. Fix output handling
Classify D3 and give the fix.
Expected result and reasoning
Improper output handling (LLM10:2026): rendering model output as HTML is XSS, and a markdown image URL is an exfiltration channel. Encode output for the console, sanitize or strip links, and proxy or block image URLs.
4. Scope retrieval and memory
Classify D4 and D5 and give the fixes.
Expected result and reasoning
D4 is cross-tenant leakage (LLM09:2026): scope every retrieval to the requesting customer. D5 is memory poisoning (ASI06:2026): don't auto-persist arbitrary facts; record provenance and gate what can be written, so a planted instruction can't steer later sessions.
5. Vet the MCP server and bound cost
Classify D6 and D7 and give the fixes.
Expected result and reasoning
D6 is agentic supply chain (ASI04:2026): a public MCP server and its tool descriptions are untrusted context and a supply-chain risk; vet it, pin it, and scope its credential. D7 is unbounded consumption (LLM06:2026): set a per-conversation cost and tool-call budget.
The exercise reasons about design notes on paper. It does not run the agent or attempt real injections; a real review needs red-team testing of injection and exfiltration paths and tests that the gateway authorizes every consequential tool call.
Save a draft on this device to resume later. Automatic saving begins after you choose Save. Drafts are never sent by this application; use synthetic data.
No draft saved on this device yet.
Sources & further reading
The standards, specifications and documentation behind the guidance above. Each is the evidence for a claim on this page; consult it for detail beyond the defaults and versions listed here.
- OWASP Top 10 for LLM Applications 2026. 3 August 2026.
- OWASP Top 10 for Agentic Applications 2026
- CWE-1427: Improper Neutralization of Input Used for LLM Prompting. CWE 4.20.
- Microsoft MSRC: CVE-2025-32711, M365 Copilot information disclosure (EchoLeak). 11 June 2025.
- Invariant Labs: data exfiltration through the GitHub MCP server (2025)
- OWASP LLM10:2026 Improper Output Handling
- NIST AI Risk Management Framework (AI RMF 1.0). January 2023.
- Microsoft STRIDE threat model
Reference information
- SDLC phase
- Design
- Role
- AppSec engineers