AI security
Apply security boundaries to models, agents, retrieved content, and generated output.
22 guides · 3 labs · 3 local projects · 1 learning path
Start here
LLM applications
Contain prompt injection, keep retrieval in bounds, treat output as untrusted and assume hidden context leaks.
Contain prompt injection at the application boundary
Assume retrieved content can influence model output; enforce data access and consequential actions in components the model cannot authorize for itself.
Keep retrieval inside the user’s data boundary
Carry document permissions through ingestion, retrieval, caching and citations so a model never receives another tenant’s records.
Keep generated output out of executable contexts
Choose safe rendering and narrow operations so model text cannot become browser code, database syntax or an unrestricted command.
Contain LLM misinformation (OWASP LLM07:2026)
Stop plausible but wrong model output from driving decisions and actions: ground answers in sources, verify every citation, separate generation from execution with a claim-check-act pattern, gate consequential actions on a human, and evaluate factuality on your own domain.
Call a hosted LLM API safely
A code-first checklist for wiring up an OpenAI or Anthropic API call from a backend: keep keys server-side, redact PII before prompts and logs, choose provider retention settings deliberately, validate structured output and still treat it as untrusted, and set timeouts, retries and per-user cost caps.
Keep secrets and controls out of hidden context (OWASP LLM08:2026)
OWASP LLM08:2026 Hidden Context Exposure, which replaced System Prompt Leakage: why system prompts, tool schemas and retrieved policy text get extracted, what must never be in them, and how to enforce rules where a leak changes nothing.
Case study: Moffatt v. Air Canada, a chatbot that misstated the refund policy (2022-2024)
How an airline's website chatbot told a grieving customer they could claim a bereavement fare after travel, what the British Columbia Civil Resolution Tribunal decided in February 2024, and the grounding, testing and correction practices that keep an automated answer consistent with the policy it describes.
Agents, tools and MCP
Authorize tool calls outside the model, sandbox the code agents run, protect their memory and bound what they spend.
Authorize agent tools outside the model
Bind each proposed action to the user, resource and approved arguments before allowing a tool to cause a side effect.
Bound agent spending, memory and recovery
Enforce budgets and stop conditions outside the model, and make retained context and side effects recoverable.
Constrain MCP and multi-agent trust boundaries
Validate each connection and delegated action without treating a token for another service, a tool description, another agent or a state handle as authority.
Contain cascading failures and rogue agents
Stop a looping, compromised or drifting agent from spreading harm: kill switches, budget and rate limits, circuit breakers, behavior monitoring, blast-radius limits, quarantine, rollback and incident response, all enforced outside the agent.
Secure AI-assisted development with coding assistants and agents
Treat generated code, suggested packages, repository text and MCP servers as untrusted inputs, and confine coding agents with OS-level isolation and short-lived, narrowly scoped credentials.
Sandbox model-generated code and computer-use agents
Run the code an agent writes, and agents that drive a browser or desktop, inside a boundary built for hostile code: a disposable runtime with no secrets and no default network, hard resource limits, and a person in front of every side effect.
Defend agent memory and context against poisoning (OWASP ASI06)
OWASP ASI06 Memory & Context Poisoning: how an instruction or false fact saved once in agent memory, conversation summaries or a retrieval store steers later sessions, users and agents, and the write, scope, provenance, expiry and recovery controls that contain it.
Case study: the Nx s1ngularity compromise and its AI-assisted payload (2025)
How a pull-request-title injection in a GitHub Actions workflow on an outdated branch gave an attacker the Nx npm token in August 2025, how the malicious versions tried to turn developers' own AI command-line tools into file-search agents, and the workflow, publishing and agent controls that stop the same chain.
Case study: the Amazon Q Developer extension compromise (2025)
How an over-scoped GitHub token, extracted from an AWS CodeBuild pull request build, let a threat actor commit code to the Amazon Q Developer extension for VS Code that shipped in version 1.84.0 in July 2025, why the injected code aimed at the Q Developer CLI failed, and the CI and AI-agent controls the incident argues for.
Models and training data
Model artifacts as supply-chain inputs, data poisoning and privacy attacks.
Treat model releases as executable supply-chain inputs
Verify provenance, constrain loading and preserve the relationship between model weights, configuration and runtime dependencies.
Protect training and retrieval data from targeted poisoning
Track who can influence datasets, keep evaluation independent and investigate targeted behavior changes without relying on aggregate accuracy.
Assess extraction, membership and inference risks
Define what an adversary can observe and measure privacy leakage without confusing a model’s output with proof about its training data.
Evaluation and red teaming
Evaluate AI features against prohibited outcomes, and red-team them as model and provider change.
Build evaluations around prohibited outcomes
Test the application’s actual security boundaries, preserve legitimate tasks and distinguish deterministic enforcement from variable model behavior.
Red-team LLM applications and agents by their effects
A red-teaming method for LLM features and agents: scope the whole system, map attacks to OWASP and MITRE ATLAS identifiers, score trials from observed effects, report rates with confidence intervals and retest on every model or provider change.
Governance and regulation
Map AI features to the NIST AI RMF, ISO/IEC 42001 and the EU AI Act.
RAG access control lab: Stop a retriever from returning another tenant’s documents
A search helper sends every matching document to a model and asks the model to respect permissions. Repair filtering before any content reaches the model.
Excessive agency lab: Bind an agent’s action to the approved arguments
An assistant receives approval to export one report, then substitutes a different report and destination. Repair the tool gateway that authorizes the final action.
XSS lab: Render a model’s answer as text instead of active markup
A report page shows generated answers with a link to the cited source. The renderer inserts both as HTML, so an answer containing an image handler becomes active markup and a javascript: source becomes a clickable script. Repair the text serializer and the DOM renderer.
Enforce retrieval and tool boundaries across crashes
Repair a two-tenant retrieval application and SQLite approval/outbox workflow. Prevent cross-tenant context, bind exact approvals, revoke queued work and recover a committed side effect without duplication.
Gate model and provider changes on a versioned red-team corpus
Run a versioned adversarial corpus many times per case, judge each trial by the secrets, tool calls, egress and spend it produced rather than the reply text, and fail a model change on per-case regressions with confidence intervals.
Run model-generated Python in a locked-down container
Repair an agent's run-Python tool until it refuses to fall back to the host, shares no folder with it, runs each job offline with memory, CPU and process limits, kills the container at the deadline and checks the result envelope the code could have forged.