Contain LLM misinformation (OWASP LLM07:2026)

OWASP defines LLM07:2026 Misinformation (/owasp-llm-top-10/2026/llm07/) as output that is incorrect, incomplete, unsupported or misleading, yet credible enough to influence a human decision, an automated workflow or an agent action. The 2026 edition renumbered it from LLM09:2025. The core risk is not that the model is sometimes wrong; it is that the wrong output is trusted and acted upon. In modern systems, model output drives tool calls, generates code, infers system state and coordinates across agents, so a false representation becomes a bad action with financial, safety or security consequences.

Misinformation arises from hallucination, stale or incomplete context, weak grounding, ambiguous prompts, corrupted data, misleading summaries or unvalidated tool output, and attackers can induce it deliberately. OWASP keeps the root causes with injected or poisoned content separate: where the cause is prompt injection or data poisoning, those are their own categories; where the harm is unsafe execution of generated code, that is LLM10:2026 Improper Output Handling; and where a hallucinated package name is registered by an attacker, that is LLM04:2026 Supply Chain. This guide is about the resulting failure mode: a false claim that drives a harmful decision.

OWASP calls overreliance a key factor: people and systems treat fluent, confident, well-structured output as authoritative, and in agent architectures that overreliance is often baked into the design, where one component simply consumes another's output as fact. The controls in this guide exist to break that assumption, in the model's output, in the interface and in the wiring between components.

• Unsupported decision support: wrong information shapes a business, legal, healthcare, financial or operational decision.
• Incorrect state inference: the model concludes a condition is met when it is not, and a workflow acts on it.
• Fabricated code and dependencies: the model recommends wrong code or a non-existent package.
• Misleading summaries: a summary drops a constraint, exception, timestamp or risk.
• Cross-agent propagation: one agent's wrong output becomes another agent's trusted input.
• Forged or misattributed evidence: fabricated content is presented as an authoritative citation.

OWASP's first prevention is to require outputs to be grounded in authoritative and current sources. Retrieval-augmented generation that pulls the relevant policy, document or record into context, and a prompt that instructs the model to answer only from that context and to say when the answer is not present, cuts the rate of free-floating invention. Pair it with retrieval you trust: keep stale documents out, and isolate each tenant's sources so one customer's answer can't be grounded in another's data (see /domains/ai-emerging/llm-security/rag-tenant-isolation/).

Grounding reduces hallucination but does not prove a specific answer is supported. The model can still summarize the retrieved text wrongly, or cite a source that does not actually say what the claim says. That is why the next step verifies citations rather than trusting that grounding happened.

Ask the model to return claims with the source id and the exact quoted span it relies on, then check, in code, that the source exists and literally contains that span. A claim whose quote is not in its cited source is unsupported and should be withheld or flagged, which also catches the forged or misattributed evidence case. This tested verifier rejected a claim whose quote was absent and one that cited an unknown source.

from dataclasses import dataclass


@dataclass(frozen=True)
class Source:
    id: str
    text: str


@dataclass(frozen=True)
class Claim:
    text: str
    source_id: str
    quote: str  # the span the model says supports the claim


class UngroundedClaim(Exception):
    pass


def verify_answer(claims, sources):
    """Every claim must cite a source that exists and contains its quoted span."""
    by_id = {s.id: s for s in sources}
    for c in claims:
        src = by_id.get(c.source_id)
        if src is None:
            raise UngroundedClaim(f"claim cites unknown source {c.source_id!r}")
        if c.quote not in src.text:
            raise UngroundedClaim(f"quote not found in {c.source_id}: {c.quote!r}")
    return True

OWASP's second and third preventions are to separate generation from execution and to validate tool calls before running them. Don't let the model's assertion be the action. Treat what the model produces as a proposed claim or a proposed tool call, check it against authoritative data and the current system state, and only then act. For a refund agent, the claim is 'this order qualifies for a refund'; the check queries the order service for the real purchase date and policy; the act issues the refund only if the check passes. This structure directly answers OWASP's incorrect-state-inference and cross-agent-trust scenarios, where one component trusts another's say-so.

def handle_refund(agent_claim, orders, policy):
    # 1. Claim: the model proposes an action with its reasoning.
    order_id = agent_claim["order_id"]

    # 2. Check: verify against authoritative state, not the model's assertion.
    order = orders.get(order_id)
    if order is None:
        return {"status": "rejected", "reason": "unknown order"}
    if not policy.refund_allowed(order):   # real dates, amounts, prior refunds
        return {"status": "rejected", "reason": "policy does not allow a refund"}

    # 3. Act: only after the check, and only within verified bounds.
    return orders.issue_refund(order_id, amount=order.amount)

OWASP's fifth prevention is runtime verification for high-impact actions through approval workflows. Decide which actions are consequential (moving money, changing access, publishing, deleting, anything safety-relevant) and require a person to approve them with the evidence shown: the claim, the sources, and what the system will do. The agent-tool-authorization guide at /domains/ai-emerging/agent-security/tool-authorization/ covers enforcing these gates outside the model, where the model cannot talk its way past them. Keep humans effective by not flooding them with low-stakes approvals, so the consequential ones get real attention.

A frequent misinformation failure is a confident but wrong code suggestion, including a reference to a package that does not exist. Spracklen et al., in the USENIX Security 2025 paper 'We Have a Package for You!', generated 576,000 code samples from 16 models and found that on average at least 5.2% of the packages commercial models referenced, and 21.7% for open-source models, did not exist, with 205,474 unique hallucinated names; 43% of hallucinated packages reappeared in all ten repeats of the prompt that produced them, which is what makes them predictable enough to pre-register (slopsquatting). Treat a suggested dependency as a claim: check it exists in the registry, has a plausible history and is the package you meant, before adding it. The AI-assisted development guide at /domains/ai-emerging/agent-security/ai-assisted-development/ covers the slopsquatting gate in a build pipeline; the execution of unsafe generated code is LLM10:2026.

OWASP is explicit that confidence is not a verification signal and that you should test for misinformation with adversarial scenarios. Build an evaluation set from your own domain: real questions with known-good answers, questions whose answer is deliberately absent from the sources (to test abstention), and questions whose context omits a critical constraint (to test the misleading-summary case). Score groundedness, the rate of unsupported citations, and abstention when appropriate, and run it on every model or prompt change. The AI security evaluation guide at /domains/ai-emerging/assurance/ai-security-evaluation/ covers building and gating on these suites.

A simple groundedness metric reuses the citation verifier: the fraction of answers whose every claim passes.

def groundedness(records, verify):
    """Fraction of answers whose every claim is grounded (verify raises on failure)."""
    passed = 0
    for r in records:
        try:
            verify(r["claims"], r["sources"])
            passed += 1
        except Exception:
            pass
    return passed / len(records)

The last line of defense is the human reading the output, so design for a skeptical reader. Show the sources next to the claims so they can be checked, surface uncertainty and abstention plainly instead of always answering, and distinguish verified facts from the model's assumptions (OWASP's calibrate-trust prevention). For agent output that other components consume, carry the provenance and verification status along with the data so a downstream component can refuse to act on an unverified claim rather than treating it as fact.

Sources & further reading

The standards, specifications and documentation behind the guidance above. Each is the evidence for a claim on this page; consult it for detail beyond the defaults and versions listed here.

1. OWASP Top 10 for LLM Applications 2026 (LLM07 Misinformation; LLM10 Improper Output Handling). 2026 edition (3 August 2026).
2. OWASP: LLM09:2025 Misinformation (the 2025 entry, renumbered to LLM07:2026). 2025 edition.
3. CWE-1426: Improper Validation of Generative AI Output. CWE 4.20.
4. Spracklen et al., 'We Have a Package for You!' USENIX Security 2025. USENIX Security 2025.
5. Spracklen et al., package hallucination analysis (arXiv 2406.10279). arXiv, June 2024.
6. Moffatt v. Air Canada, 2024 BCCRT 149 (Civil Resolution Tribunal of British Columbia). Decision of 14 February 2024.
7. Mata v. Avianca, Inc., No. 22-cv-1461 (PKC) (S.D.N.Y.), opinion and order on sanctions, 678 F. Supp. 3d 443. 22 June 2023.
8. NIST AI Risk Management Framework (AI RMF 1.0). AI RMF 1.0 (January 2023).

Reference information

Tags

LLM application securityAI evaluation and governance

SDLC phase

DesignDevelopmentTestingSecurity program

Role

AppSec engineers

Download this guide as Markdown

Continue exploring