Assess extraction, membership and inference risks

Membership inference asks whether an example was used in training. Model inversion seeks sensitive attributes or representative inputs from model behavior. Extraction attempts to reproduce a model or recover protected information through access to outputs or parameters. These overlap in practice but require different attacker capabilities and evidence. The OWASP Top 10 for LLM Applications 2026 groups their effects as Sensitive Information Disclosure (LLM02:2026, /owasp-llm-top-10/2026/llm02/), and MITRE ATLAS lists Infer Training Data Membership (AML.T0024.000) and Invert AI Model (AML.T0024.001).

A model mentioning a person is not proof that the person’s record was in training; the information may be public, retrieved or inferred. Conversely, a model never printing an exact training row does not establish absence of membership or attribute leakage. State which disclosure would harm the people or organization involved.

The attack rule predicts “member” when a supplied score is at least 0.80. On four known members (scores 0.95, 0.85, 0.81 and 0.40) it flags three, so the true-positive rate is 3/4=75%. On four known nonmembers (0.90, 0.60, 0.30 and 0.10) it flags one, so the false-positive rate is 1/4=25%. The balanced test-set accuracy is 6/8=75%, but that single number hides how often the rule labels a nonmember as a member.

Now apply those rates hypothetically to 1,000 people when only 10 are members. Expected true positives are 10×0.75=7.5; expected false positives are 990×0.25=247.5. The expected precision is 7.5/(7.5+247.5), about 2.94%. Fractional counts are expectations across repeated comparable populations, not literal half-people. This is the base-rate problem; eight synthetic examples are far too few to establish stable real-world rates.

Carlini et al. (2022) argue that attacks should be compared by their true-positive rate when false positives are rare, because an attack that confidently identifies a few members is a real privacy failure even if its average accuracy is poor. The tpr_at_fpr function below picks the strictest threshold allowed for a target false-positive rate. Here, flagging no nonmember at all leaves only M1 above N1’s 0.90, a true-positive rate of 25%; and with four nonmembers no rate between 0% and 25% can even be observed. A 0.1% target needs thousands of nonmembers.

A membership experiment also needs comparable member and nonmember distributions. If members are recent high-quality records and nonmembers are old malformed records, the score may detect those features rather than training membership. Duan et al. (2024) found that published attacks on LLM pre-training data barely beat random guessing once members and nonmembers came from the same period. Match sampling conditions, report uncertainty, and never label a person’s record as a training member from one favorable guess.

Canary disclosure asks a different question. If a unique synthetic marker appears in an answer, inspect its path. The trace in the exercise includes the marker in retrieved document D7 immediately before generation; that supports a retrieval exposure path and does not establish that the model memorized the marker during training. Repair the boundary shown by the evidence (for retrieval, /domains/ai-emerging/llm-security/rag-tenant-isolation/), then isolate other paths in a separately designed experiment.

def confusion(scores, is_member, threshold):
    flagged = [s >= threshold for s in scores]
    tp = sum(f and m for f, m in zip(flagged, is_member))
    fp = sum(f and not m for f, m in zip(flagged, is_member))
    fn = sum(not f and m for f, m in zip(flagged, is_member))
    tn = sum(not f and not m for f, m in zip(flagged, is_member))
    return tp, fp, fn, tn


def rates(tp, fp, fn, tn):
    return {
        "tpr": tp / (tp + fn),
        "fpr": fp / (fp + tn),
        "accuracy": (tp + tn) / (tp + fp + fn + tn),
    }


def expected_precision(tpr, fpr, members, nonmembers):
    """Share of flagged people who really are members, at a given prevalence."""
    true_positives, false_positives = tpr * members, fpr * nonmembers
    return true_positives / (true_positives + false_positives)


def tpr_at_fpr(member_scores, nonmember_scores, max_fpr):
    """True-positive rate at the lowest threshold that flags at most max_fpr of
    the nonmembers.

    Pick the threshold on development data, then report the rate on separate
    held-out data. With n nonmembers, the smallest nonzero false-positive rate
    you can observe is 1/n.
    """
    ranked = sorted(nonmember_scores, reverse=True)
    allowed = int(
        max_fpr * len(ranked)
    )  # nonmembers that may score above the threshold
    threshold = ranked[allowed] if allowed < len(ranked) else float("-inf")
    flagged = sum(s > threshold for s in member_scores)
    return threshold, flagged / len(member_scores)

List training and fine-tuning data, retrieval sources, logs, embeddings, output detail and query access. Ask whether sensitive information needs to be present in the first place. Minimize unnecessary collection (/domains/identity-crypto/privacy/data-minimization-retention/) and restrict raw data and model artifact access. Treat a hosted model’s retention and reuse terms as dependencies to verify for the chosen service and account settings, including whether prompts and outputs are logged or used for training.

Limit outputs and queries to the intended product task. Full confidence scores, unlimited repeated queries or downloadable parameters enable attacks that a label-only interface does not, and model extraction (Tramèr et al. 2016) needs nothing more than many paid queries. Rate limits and monitoring raise the cost of some attacks but do not create a formal privacy guarantee.

Build an authorized experiment with synthetic or appropriately approved records. Separate known members from nonmembers drawn from the same distribution and control obvious confounders such as collection date and data quality. Specify attacker knowledge, query budget, target metric and false-positive rate before interpreting success. High accuracy on an artificially easy split can exaggerate practical leakage.

For generative systems, include synthetic canaries and trace where a disclosed marker originated: training, retrieval, prompt history or logs. A canary is evidence about one exposure path under the test conditions, not a universal privacy score. Evaluate rare and duplicated examples separately: Kandpal et al. (2022) showed that most of the extraction risk in language models comes from text duplicated in the training data.

If leakage originates in retrieval permissions, repair that boundary; changing training privacy parameters will not fix it. If the concern is training memorization or inference, start with data minimization and deduplication, then consider differential privacy. DP-SGD, available for PyTorch in the Opacus library, bounds what any single training record can change, but only for the epsilon and delta it reports, and usually at some cost in accuracy. Arbitrary noise without an accounting of that budget is not differential privacy.

Re-evaluate task utility and the intended privacy claim after every change, using the same attack and the same false-positive rate as before. Report the residual true-positive rate rather than a pass or fail label.

Selecting a threshold after inspecting the test labels can overfit the evaluation. Choose the operating threshold on a separate development set, then measure it on a held-out set with the relevant population and query access. Report both the number of examples and the actual false positives; a rounded rate of zero can conceal an undersized experiment.

Avoid testing with unapproved sensitive records simply to measure whether they leak. Use synthetic or properly authorized data with controlled destinations and retention. The test logs, prompts and exported score files are themselves data flows that need an appropriate access policy.

Each row is a case from the eight synthetic scores or the canary trace and what it does and does not show. The first four follow from confusion and rates above.

Sources & further reading

The standards, specifications and documentation behind the guidance above. Each is the evidence for a claim on this page; consult it for detail beyond the defaults and versions listed here.

1. NIST AI 100-2 E2025: Adversarial Machine Learning. March 2025.
2. OWASP Top 10 for LLM Applications 2026 (LLM02 Sensitive Information Disclosure). 3 August 2026.
3. OWASP LLM02:2025 Sensitive Information Disclosure (2025 list; same ID in 2026)
4. Carlini et al. (2022): Membership Inference Attacks From First Principles
5. Duan et al. (2024): Do Membership Inference Attacks Work on Large Language Models?
6. Nasr et al. (2023): Scalable Extraction of Training Data from (Production) Language Models
7. Kandpal et al. (2022): Deduplicating Training Data Mitigates Privacy Risks in Language Models
8. Tramèr et al. (2016): Stealing Machine Learning Models via Prediction APIs
9. Abadi et al. (2016): Deep Learning with Differential Privacy
10. Opacus FAQ: what epsilon and delta mean and how to choose delta
11. MITRE ATLAS data (AML.T0024.000 Infer Training Data Membership, AML.T0024.001 Invert AI Model). Release 2026.09 (15 September 2026).

Reference information

Tags

Privacy engineeringModels and training data

SDLC phase

DesignDevelopmentRelease

Role

AppSec engineers

Download this guide as Markdown

Continue exploring