--- key: "threat-model-llm-agent" title: "Worked threat model: an LLM agent with tools" url: "https://softwaresecurity.com/domains/appsec/threat-modeling/llm-agent-threat-model/" api: "https://softwaresecurity.com/api/v1/articles/threat-model-llm-agent.json" domain: "appsec" topic: "Application security" type: "Guide" identifiers: ["CWE-1427","CWE-862","CWE-918","CWE-79","LLM01:2026","LLM02:2026","LLM03:2026","LLM06:2026","LLM09:2026","LLM10:2026","ASI01:2026","ASI02:2026","ASI03:2026","ASI04:2026","ASI06:2026"] tags: ["LLM application security","Prompt injection","AI agents and tools","Threat modeling"] stages: ["design"] roles: ["appsec-engineer"] license: "CC-BY-4.0" license_url: "https://creativecommons.org/licenses/by/4.0/" code_license: "MIT" --- # Worked threat model: an LLM agent with tools A complete worked threat model for an LLM agent with tools: the data-flow diagram, the threats from prompt injection, excessive agency, data exfiltration, confused-deputy and memory poisoning, and the mitigations, mapped to the OWASP Top 10 for LLM Applications 2026 and the OWASP Top 10 for Agentic Applications 2026. Canonical URL: https://softwaresecurity.com/domains/appsec/threat-modeling/llm-agent-threat-model/ ## In short This is a worked threat model for an LLM agent: a chat UI, an orchestrator, a hosted LLM, retrieval over company documents, tools (email, tickets, a code sandbox, web fetch), an MCP server, long-term memory and a human approval step. Model it by drawing the data flows, then enumerating the agent-specific threats: direct and indirect prompt injection through retrieved documents, web pages, emails and tool output; excessive agency and tool misuse; data exfiltration through tool calls and rendered links; confused-deputy use of the agent's privileges; cross-tenant retrieval leakage; memory and MCP-tool poisoning; unbounded cost; and unsafe output handling. The decisive mitigations enforce authority outside the model: a tool gateway with per-user authorization, least-privilege scoped credentials, allowlisted egress, human approval for consequential actions, and output encoding. The threats map to two OWASP lists: the Top 10 for LLM Applications 2026 and the Top 10 for Agentic Applications 2026. - Every input the model reads is untrusted, including retrieved documents, fetched web pages, emails and the output of tools; indirect prompt injection is the central threat (LLM01:2026, ASI01:2026). - The model cannot be the authorization boundary: enforce who-can-do-what in a tool gateway outside the model, with least-privilege scoped credentials (ASI02/ASI03:2026). - Exfiltration happens through the agent's own tools and through rendered links and image URLs; allowlist egress and encode output (LLM02/LLM10:2026). - A confused deputy acts with the agent's privileges on an attacker's behalf; bind actions to the requesting user's authority, not the agent's. - Memory, retrieval and MCP tool descriptions are attack surface: poison them and you steer the agent later (ASI06:2026, ASI04:2026). - Require human approval for consequential actions with the full parameters shown, bound cost and tools, and log every tool call for audit. ## Recommended defaults | Setting | Value | Note | Source | | --- | --- | --- | --- | | Trust of model inputs | All untrusted: user text, retrieved docs, web pages, emails, tool output | Indirect prompt injection rides in on any of these. | OWASP LLM01:2026; ASI01:2026 | | Tool authorization | Enforced in a gateway outside the model, per requesting user | The model proposes; the gateway resolves the real actor, tool and scope and decides. | OWASP ASI02:2026, ASI03:2026 | | Tool credentials | Least-privilege, scoped per tool, short-lived | A compromised or hijacked agent can only reach what its credentials allow. | OWASP ASI03:2026 | | Egress | Allowlisted destinations for web fetch and outbound tools | Stops exfiltration to attacker hosts and SSRF to internal addresses. | LLM02:2026; ssrf-egress-boundaries | | Consequential actions | Human approval with the exact parameters shown | Send email, move money, delete, publish, run code: a person confirms what will happen. | OWASP ASI02:2026; agent-tool-authorization | | Output handling | Encoded for its sink; never executed; links and images sanitized | Model output rendered as HTML is XSS; a markdown image URL is an exfiltration channel. | OWASP LLM10:2026 Improper Output Handling | | Retrieval and memory | Tenant-scoped retrieval; memory writes carry provenance | Prevents cross-tenant leakage and memory poisoning steering later turns. | OWASP LLM09:2026; ASI06:2026 | | Cost and tools | Per-user budget and a bounded, vetted tool set | Caps unbounded consumption; MCP servers and tools are vetted before enabling. | OWASP LLM06:2026; ASI04:2026 | ## Verify it 1. Walk each flow in the diagram and confirm that a malicious string in any model input (document, web page, email, tool result) cannot by itself cause a tool call to run without the gateway's authorization. 2. Attempt an indirect injection: plant instructions in a retrieved document and a fetched page and confirm the agent does not exfiltrate data or call a tool outside its policy. 3. Confirm every consequential tool call is authorized against the requesting user and requires human approval with the real parameters shown, not a blind approve/deny. 4. Test exfiltration channels: a rendered link, a markdown image URL, and a tool call to an off-allowlist host, and confirm each is blocked or stripped. 5. Cross-tenant test: ask the agent for another tenant's data and confirm retrieval returns nothing and no tool reaches it. 6. Confirm per-user cost budgets, bounded tool use, provenance on memory writes, and an audit log of every tool call with its arguments and decision. ## What you will learn - Draw the data flows of an LLM agent with tools, retrieval, memory and MCP. - Enumerate the agent-specific threats, including indirect prompt injection and exfiltration. - Choose mitigations that enforce authority outside the model. - Map the threats to the OWASP Top 10 for LLM Applications 2026 and the OWASP Top 10 for Agentic Applications 2026. Audience: Engineers building LLM agents and reviewers doing a design review of an agentic feature. Before you start: - Read a data-flow diagram and know what a trust boundary is. - Understand prompt injection and tool use in LLM applications. Time: About 26 minutes: 6 to read and 20 for the exercise ## Describe the agent and its assumptions The system is an assistant that plans and acts. A chat UI sends the user's request to an orchestrator, which calls a hosted LLM, retrieves from a store of company documents (RAG), and can call tools: send email, file and update tickets, run code in a sandbox, and fetch web pages. It talks to external systems through an MCP server, keeps long-term memory across sessions, and has a human-approval step for some actions. State the assumptions. Everything the model reads is untrusted: the user's text, retrieved documents, fetched web pages, incoming emails, tool output, memory and MCP tool descriptions can all carry instructions. The model has no reliable way to tell data from instructions, so it cannot be the place authority is decided. The hosted LLM and the retrieval store are inside the application's boundary but their inputs are not. ## List the elements, data and trust boundaries | Element | Data it handles | Trust | | --- | --- | --- | | Chat UI | User request, rendered output | User input untrusted; output needs encoding | | Orchestrator + LLM | Prompt, context, proposed tool calls | Proposals are untrusted | | Retrieval store | Company documents, embeddings | Content untrusted; must be tenant-scoped | | Memory | Facts kept across sessions | Writable by earlier turns; poisonable | | Tool gateway | Authorization, scope, approval | The enforcement point | | Tools / MCP | Email, tickets, code, web fetch | Each a capability with its own blast radius | ## Enumerate the agent-specific threats Walk the threats element by element. Unlike a classic app, the dangerous input often arrives indirectly, through content the agent reads while doing its job. | Threat | Example | OWASP 2026 | | --- | --- | --- | | Direct prompt injection | The user tells the agent to ignore its rules | LLM01:2026 | | Indirect prompt injection | A retrieved document, web page or email contains instructions the agent follows | LLM01:2026, ASI01:2026 | | Excessive agency / tool misuse | The agent calls a powerful tool it shouldn't for this request | LLM03:2026, ASI02:2026 | | Privilege / identity abuse (confused deputy) | The agent uses its own broad credentials on an attacker's behalf | ASI03:2026 | | Data exfiltration | Data leaves via a tool call, a rendered link or a markdown image URL | LLM02:2026 | | Cross-tenant leakage | Retrieval returns another tenant's documents | LLM09:2026 | | Memory poisoning | A planted memory steers later sessions | ASI06:2026 | | Supply chain / MCP poisoning | A malicious tool description or server changes behavior | ASI04:2026 | | Unbounded consumption | A loop or large context runs up cost | LLM06:2026 | | Improper output handling | Model output is rendered as HTML or executed | LLM10:2026 | ## Enforce authority outside the model The single most important mitigation is that the model never decides what it is allowed to do. The model proposes a tool call; a gateway outside the model resolves the requesting user, checks that this user may use this tool with these exact arguments and scope, and decides. This defeats excessive agency and the confused-deputy problem, because the action runs with the requesting user's authority, not the agent's broad credentials. Give each tool a least-privilege, scoped, short-lived credential so a hijacked agent can reach only what that tool allows. The agent-tool-authorization guide at /domains/ai-emerging/agent-security/tool-authorization/ has the gateway pattern in code, and the lab at /labs/agent-tool-gateway/ has you bind an action to the approved arguments. ## Contain prompt injection; it cannot be fully prevented You cannot reliably stop a model from being influenced by instructions hidden in the content it reads, so design as if injection will sometimes succeed and limit what a successful injection can achieve. Keep untrusted content clearly separated in the prompt, prefer retrieval over letting the agent fetch arbitrary URLs, and above all make sure a compromised plan still hits the gateway's authorization, egress allowlist and human-approval gates. The prompt-injection guide at /domains/ai-emerging/llm-security/prompt-injection-boundaries/ covers containment; the point for the threat model is that injection is a path to the real impact, not the impact itself. ## Close the exfiltration channels Data leaves an agent in more ways than an obvious tool call. A fetched or rendered link with data in its query string, a markdown image whose URL the client loads, and a tool that posts to an attacker-chosen address are all exfiltration channels an injection can use. Allowlist egress for web fetch and outbound tools, and re-check a URL that came from model output against the allowlist (the SSRF overlap is covered at /domains/appsec/secure-features/ssrf-egress-boundaries/). Sanitize links and strip or proxy image URLs in rendered output, and encode output for its sink so it cannot execute (LLM10:2026). ## Scope retrieval and vet memory and MCP Retrieval, memory and MCP are attack surface. Scope every retrieval query to the requesting tenant and user so one customer's question can't return another's documents (/domains/ai-emerging/llm-security/rag-tenant-isolation/; practice it in /labs/rag-isolation/). Treat memory writes as untrusted: record provenance, and don't let the agent silently persist instructions that steer future sessions. Vet MCP servers and their tool descriptions before enabling them, because a poisoned tool description is itself injected context; the MCP guide at /domains/ai-emerging/agent-security/mcp-multiagent-security/ covers vetting and inter-agent trust. ## Record residual risk and keep the model current Be honest about what remains. Indirect prompt injection cannot be eliminated, so the model's protection is the authority, egress and approval gates around it; document that and make sure those gates have no bypass. Human approval degrades if people are asked to confirm too much, so reserve it for genuinely consequential actions and show the real parameters. Re-run the model when you add a tool, enable a new MCP server, change the retrieval corpus, or give the agent a new capability; map each threat to a control and a test, and log every tool call so an incident is reconstructable. /domains/ai-emerging/assurance/ai-red-teaming/ turns these threats into repeatable red-team cases mapped to OWASP and MITRE ATLAS IDs, scored by their effects and rerun on every model or provider change. EchoLeak (CVE-2025-32711, June 2025), an AI command-injection flaw in Microsoft 365 Copilot that allowed information disclosure over the network, and the 2025 demonstration of data exfiltration through a GitHub MCP server show these threats are real, not hypothetical. ## Guided practice You are reviewing a customer-support agent before launch. It reads the customer's tickets and a knowledge base, can email the customer, can issue account credits, and can fetch linked pages. The design notes below are from the team. For each, name the threat and OWASP 2026 id and give the fix. ### Supplied material ```text D1 The system prompt says "only issue credits under $50"; the agent decides whether to issue a credit. D2 The agent fetches any URL found in a ticket to "get more context." D3 Replies are rendered as HTML in the agent console, including any links and images the model produced. D4 Retrieval searches the whole knowledge base, which includes other customers' tickets. D5 The agent stores "useful facts" to long-term memory automatically from each conversation. D6 A new MCP server was added from a public registry to let the agent look up shipping status. D7 There is no per-conversation cost or tool-call limit. ``` ### 1. Move the credit decision out of the model Classify D1 and give the fix. Worked answer: Excessive agency (LLM03:2026) and tool misuse (ASI02:2026): a prompt rule is not an authorization boundary; an injection can talk the model past it. Enforce the credit limit and eligibility in the tool gateway against the requesting user, and require human approval above a threshold, with the amount and account shown. ### 2. Close the fetch channel Classify D2 and give the fix. Worked answer: Indirect prompt injection and SSRF/exfiltration (LLM01/LLM02:2026): a ticket URL is attacker-controlled; fetching it pulls in injected instructions and can exfiltrate or reach internal hosts. Allowlist egress, re-check model-supplied URLs, and prefer retrieval over arbitrary fetch. ### 3. Fix output handling Classify D3 and give the fix. Worked answer: Improper output handling (LLM10:2026): rendering model output as HTML is XSS, and a markdown image URL is an exfiltration channel. Encode output for the console, sanitize or strip links, and proxy or block image URLs. ### 4. Scope retrieval and memory Classify D4 and D5 and give the fixes. Worked answer: D4 is cross-tenant leakage (LLM09:2026): scope every retrieval to the requesting customer. D5 is memory poisoning (ASI06:2026): don't auto-persist arbitrary facts; record provenance and gate what can be written, so a planted instruction can't steer later sessions. ### 5. Vet the MCP server and bound cost Classify D6 and D7 and give the fixes. Worked answer: D6 is agentic supply chain (ASI04:2026): a public MCP server and its tool descriptions are untrusted context and a supply-chain risk; vet it, pin it, and scope its credential. D7 is unbounded consumption (LLM06:2026): set a per-conversation cost and tool-call budget. Deliverable: A threat table for D1-D7 with the threat, OWASP 2026 id and fix, plus the residual risks (notably that indirect injection can't be eliminated) and the tests that would confirm the gateway, egress and approval gates hold. Scope: The exercise reasons about design notes on paper. It does not run the agent or attempt real injections; a real review needs red-team testing of injection and exfiltration paths and tests that the gateway authorizes every consequential tool call. Articles and teaching material: CC BY 4.0. Original executable code: MIT. Attribution: Software Security. ## Sources - [OWASP Top 10 for LLM Applications 2026](https://genai.owasp.org/resource/owasp-genai-llm-top-10-2026/). 3 August 2026. - [OWASP Top 10 for Agentic Applications 2026](https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/) - [CWE-1427: Improper Neutralization of Input Used for LLM Prompting](https://cwe.mitre.org/data/definitions/1427.html). CWE 4.20. - [Microsoft MSRC: CVE-2025-32711, M365 Copilot information disclosure (EchoLeak)](https://msrc.microsoft.com/update-guide/vulnerability/CVE-2025-32711). 11 June 2025. - [Invariant Labs: data exfiltration through the GitHub MCP server (2025)](https://invariantlabs.ai/blog/mcp-github-vulnerability) - [OWASP LLM10:2026 Improper Output Handling](https://genai.owasp.org/resource/owasp-genai-llm-top-10-2026/) - [NIST AI Risk Management Framework (AI RMF 1.0)](https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-1.pdf). January 2023. - [Microsoft STRIDE threat model](https://learn.microsoft.com/en-us/azure/security/develop/threat-modeling-tool-threats)