Protect training and retrieval data from targeted poisoning

Poisoning introduces manipulated examples, labels, documents or model updates into a trusted pipeline. It can degrade overall quality or plant a narrow behavior that appears only under a chosen trigger. A small targeted change may leave aggregate validation accuracy almost unchanged while creating a consequential failure for one class of inputs. The OWASP Top 10 for LLM Applications 2026 lists this as Data and Model Poisoning (LLM05:2026, /owasp-llm-top-10/2026/llm05/), and MITRE ATLAS tracks it as AML.T0020 Training Data Poisoning and AML.T0070 RAG Poisoning.

The amount of poison needed is small. In the largest pre-training experiments published so far, Souly et al. (2025) found that 250 malicious documents planted a backdoor in models from 600 million to 13 billion parameters, even though the largest trained on more than 20 times more clean data; the same held for fine-tuning. Defenses that rely on poison being a meaningful share of the corpus do not hold.

Distinguish training-time changes from retrieval-time document contamination. They persist differently: removing a retrieved document can stop future context exposure, while a trained model keeps the effect until it is replaced or retrained. An anomalous output alone does not identify which path was responsible.

Two models are evaluated on the same protected 10,000-case set. The approved model gets 9,500 correct and the candidate gets 9,490 correct: 95.0% versus 94.9%. That small aggregate change looks modest. But the 100 cases in the business-critical trigger slice fall from 90 correct to 10 correct, an 80-percentage-point drop that the much larger ordinary set hides.

The other 9,900 cases improve from 9,410 correct to 9,480 correct, so the candidate’s ordinary accuracy rises while its targeted behavior collapses. The gate below compares each protected slice against a limit its owner sets, and fails this candidate on the trigger slice while the aggregate passes. Set limits with the slice size in mind: 100 cases cannot resolve a one-point change, so either enlarge small slices or accept a wider limit knowingly. The synthetic trigger is a predefined test category, not a recipe for attacking an uncontrolled pipeline.

Lineage shows the candidate added batch B7, containing 400 near-duplicate examples with reversed labels in that slice. That is a plausible causal lead, not proof. A controlled ablation removes B7 and rebuilds with the supported, reproducible process while holding the evaluation set and other conditions fixed. If slice performance recovers, it strengthens the hypothesis that B7 contributed. Training variation and other changes still need examination; one correlation does not prove the contributor’s intent.

Protect the holdout set during the investigation. Copying its answers into training could restore the reported score while contaminating the measurement. Quarantine the suspect source and version, identify derived models and indexes, and preserve the original batch with its admission history. Removing a document from a live index does not erase a trained model’s learned behavior; recover at each affected layer.

def accuracy(counts):
    correct, total = counts
    return correct / total


def slice_gate(approved, candidate, max_drop):
    """Compare each protected slice, not only the total.

    approved/candidate: {slice: (correct, total)} on the same frozen evaluation set.
    max_drop: {slice: largest acceptable fall in accuracy}, set by the slice owner.
    """
    failures = []
    for name, limit in max_drop.items():
        before, after = accuracy(approved[name]), accuracy(candidate[name])
        if before - after > limit:
            failures.append(f"{name}: {before:.1%} -> {after:.1%}")
    return failures


def overall(results):
    return accuracy(tuple(map(sum, zip(*results.values()))))

Inventory data sources, contributors, ingestion credentials, labeling workflows and transformations. Record immutable dataset versions or reproducible snapshots, including exclusions and deduplication rules. Separate the accounts or automation that add examples from those that approve a dataset for training, so one compromised contributor cannot both insert and release data. NIST SP 800-218A task PS.1.2 sets the same baseline: protect training, testing, fine-tuning and alignment data from unauthorized access and modification, and monitor its integrity.

Validate schemas and label ranges, but do not confuse clean formatting with trustworthy meaning. Review source concentration, sudden distribution shifts, repeated near-duplicates and unusual label changes. Decide which contributions need human or independent review based on their authority and potential effect, rather than using one threshold for every source.

Protect held-out evaluation data from contamination and leakage. If training and test sets share manipulated or duplicated examples, the reported score can reward the same defect you are trying to find. Evaluate relevant subgroups and targeted trigger conditions alongside ordinary performance; /domains/ai-emerging/assurance/ai-security-evaluation/ covers designing those cases.

Compare candidate and approved model/data versions under the same evaluation protocol. Investigate material changes with lineage information and controlled ablations: remove the suspect batch, rebuild through the supported process and compare the affected behavior. Preserve legitimate examples so the mitigation does not simply erase a useful class of functionality.

When evidence supports poisoning, quarantine the implicated source or version, stop further promotion and identify derived models, indexes and caches. Roll back to a pinned known-good release (/domains/ai-emerging/ml-security/model-artifact-security/) where appropriate and retain the evidence needed for analysis. A dataset checksum tells you which bytes were used, not whether their labels were honest; the recovery decision needs both provenance and measured behavior.

Schema validation can reject malformed labels but cannot tell that a valid label was intentionally reversed. Review contributor authority, source changes and near-duplicate concentration with subject-matter context. A valid checksum likewise identifies the poisoned bytes without making their meaning trustworthy.

Do not delete the whole targeted class simply to improve the attack metric. Keep legitimate examples in the acceptance set and measure utility for that class. Otherwise a model that refuses every relevant task can appear protected while eliminating the feature.

Each row is a measurement from the M1 and M2 comparison and what it shows about the suspected poisoning of batch B7.

Sources & further reading

The standards, specifications and documentation behind the guidance above. Each is the evidence for a claim on this page; consult it for detail beyond the defaults and versions listed here.

1. OWASP Top 10 for LLM Applications 2026 (LLM05 Data and Model Poisoning). 3 August 2026.
2. OWASP LLM04:2025 Data and Model Poisoning (2025 list; LLM05 in 2026)
3. Souly et al. (2025): Poisoning attacks on LLMs require a near-constant number of poison samples
4. NIST AI 100-2 E2025: Adversarial Machine Learning. March 2025.
5. OWASP Secure AI Model Ops Cheat Sheet
6. MITRE ATLAS data (AML.T0020 Training Data Poisoning, AML.T0070 RAG Poisoning). Release 2026.09 (15 September 2026).
7. NIST SP 800-218A: Secure software development practices for generative AI and dual-use foundation models. July 2024.

Reference information

Tags

Supply chain attacksModels and training dataAI evaluation and governance

SDLC phase

DevelopmentTestingReleaseSecurity program

Role

DevelopersAppSec engineersPlatform engineers and SREs

Download this guide as Markdown

Continue exploring