Data minimization and retention: collect less and prove that deletion works

Data minimization starts with the operation the product must perform. A delivery service needs a delivery address while fulfilling an order; a conversion-count dashboard may need only aggregate events. Collecting the full address for both creates extra copies, access paths, and retention obligations without improving the dashboard’s purpose.

Retention is a lifecycle rule, not a single database TTL. A record can move from the transactional store to search, caches, logs, analytics, exports, and backups. Deleting the primary row does not delete these descendants. The engineering task is to define which copies remain useful for which purpose and when each must expire.

Distinguish deletion, access restriction, aggregation, and pseudonymization. They change different properties. A hidden row still exists; a tokenized row may remain linkable (/domains/identity-crypto/privacy/tokenization-privacy-engineering/ compares the transformations); an aggregate can still reveal information about a small group. Name the actual transformation instead of reporting every change as deletion.

For each dataset, record source fields, purpose, recipients, storage location, access policy, retention trigger, maximum duration, deletion mechanism, and backup behavior. Include identifiers used to find all related copies. A deletion job cannot reliably remove one person’s data from a warehouse if the ingestion pipeline discarded every join key before a deletion strategy was designed.

Use event-based triggers where appropriate: account closure, order completion, or resolution of a support case. Define time zones and exact boundary semantics. A rule saying thirty days is incomplete if one system counts from creation, another from last update, and a third restarts the clock on every restore.

Separate documented exceptions such as a required preservation hold from normal retention. Record the authority, scope, expiration or review point, and access restrictions for the exception. The technical guide does not determine legal obligations; teams must translate their established requirements into explicit executable rules.

Keep operational metadata proportional. A deletion ledger may need a record identifier, request time, scope, and completion evidence. Copying the full deleted payload into that ledger recreates the dataset under another name.

Minimize at ingress: reject unnecessary fields, avoid logging raw request bodies, and configure analytics events to carry the least detail needed. Once sensitive fields enter a general log stream, removing them from all downstream vendors and archives is harder than preventing collection.

Issue an idempotent deletion or expiry event to every dependent store. A consumer should safely process the event more than once and report a meaningful outcome such as removed, already absent, held, or retry required. Keep failures in a retry path with an owner; a successful primary-store deletion must not hide a permanently failed search deletion.

Backups often cannot support selective in-place deletion without undermining recovery design. Set a bounded backup retention policy and restrict restored access. On restore, replay deletion tombstones or otherwise reconcile current deletion state before the restored service becomes available. A snapshot should not resurrect an account just because it predates the deletion.

Derived data needs a separate assessment. A monthly total may be retained if it no longer permits the original subject to be singled out under the intended use; a per-user feature vector still identifies or profiles that person even without their name. Have the data owner document the decision with the actual fields and recipients.

Retention jobs can delete too much as well as too little. Preview selection counts, stage risky changes, and use precise predicates. A timezone or unit mistake can shift a cutoff by hours or orders of magnitude. Deletion is one place where a reversible dry run and representative recovery test materially improve safety.

Backups are hard to edit, so the usual answer is a bounded backup lifetime plus replaying deletions after a restore. Crypto-shredding adds a stronger option: encrypt each person’s records with their own key and keep those keys in a separate store with a short backup lifetime. Deleting the key makes every copy of the records unreadable, including copies inside data backups. NIST SP 800-88 Rev. 2 (September 2025) calls this cryptographic erase. The envelope encryption guide at /domains/identity-crypto/key-management-pki/envelope-encryption-rotation/ covers key deletion safeguards.

In this example, the data store holds only ciphertext bound to the person and record, and the key store holds one 256-bit key per person. It was tested with cryptography 50 on Python 3.14: after forget_subject, a data backup taken earlier still contains the ciphertext but cannot be decrypted, other people’s records still decrypt, and, as a negative test, restoring an old key backup brings the data back. Hold the keys in a KMS-protected store in production, and treat the key store’s backup lifetime as your real erasure deadline.

import json
import os
import sqlite3

from cryptography.hazmat.primitives.ciphers.aead import AESGCM

# Two stores: `keys` (one data key per person, short backup retention) and `data` (ciphertext only).
# Backups of `data` never contain a key, so deleting the key erases every copy of that person's records.


def _aad(subject_id: str, record_id: str) -> bytes:
    return json.dumps([subject_id, record_id]).encode()


def save_record(keys: sqlite3.Connection, data: sqlite3.Connection, subject_id: str, record_id: str, value: bytes):
    with keys:
        keys.execute("INSERT OR IGNORE INTO subject_keys VALUES (?, ?)", (subject_id, AESGCM.generate_key(bit_length=256)))
        key = keys.execute("SELECT key FROM subject_keys WHERE subject_id = ?", (subject_id,)).fetchone()[0]
    nonce = os.urandom(12)
    with data:
        data.execute("INSERT INTO records VALUES (?, ?, ?, ?)",
                     (record_id, subject_id, nonce, AESGCM(key).encrypt(nonce, value, _aad(subject_id, record_id))))


def read_record(keys: sqlite3.Connection, data: sqlite3.Connection, record_id: str) -> bytes | None:
    row = data.execute("SELECT subject_id, nonce, ciphertext FROM records WHERE record_id = ?", (record_id,)).fetchone()
    if row is None:
        return None
    key = keys.execute("SELECT key FROM subject_keys WHERE subject_id = ?", (row[0],)).fetchone()
    if key is None:
        return None  # shredded: the ciphertext may still exist, but nothing can decrypt it
    return AESGCM(key[0]).decrypt(row[1], row[2], _aad(row[0], record_id))


def forget_subject(keys: sqlite3.Connection, data: sqlite3.Connection, subject_id: str):
    with keys:
        keys.execute("DELETE FROM subject_keys WHERE subject_id = ?", (subject_id,))
    with data:  # tidy the live store; backups become unreadable once the key and its backups expire
        data.execute("DELETE FROM records WHERE subject_id = ?", (subject_id,))

Seed a clearly synthetic record with a unique marker, send it through the normal ingestion paths, and exercise the retention trigger. Verify absence or the documented restricted state in the primary store, search, cache, analytics, export queue, and support tooling. Observe consumer acknowledgments and retry handling.

At the boundary, a record just before expiry remains, while a record at or after the defined expiry is selected. Test a held record separately. Restore a pre-deletion backup into an isolated environment and show that reconciliation prevents the synthetic subject from reappearing in served results.

A completed job log establishes that a process ran. It does not establish that every copy was found or inaccessible. Report the covered stores, exceptions, oldest unresolved deletion, and remaining backup lifetime so the evidence has a precise scope.

Sources & further reading

The standards, specifications and documentation behind the guidance above. Each is the evidence for a claim on this page; consult it for detail beyond the defaults and versions listed here.

1. NIST Privacy Framework (CSWP 10). Version 1.0 (January 2020).
2. NIST SP 800-188: De-identification techniques and governance. September 2023.
3. OWASP User Privacy Protection
4. NIST SP 800-88 Rev. 2: Guidelines for Media Sanitization. Revision 2 (September 2025).
5. OWASP ASVS 5.0: V14 Data Protection. Version 5.0.0 (May 2025).
6. Regulation (EU) 2016/679 (GDPR), Article 5: data minimization and storage limitation

Reference information

Tags

EncryptionPrivacy engineering

SDLC phase

PlanningDesignSecurity program

Role

DevelopersAppSec engineers

ASVS 5.0.0 requirements

v5.0.0-14.1.2 · v5.0.0-14.2.6 · v5.0.0-14.2.7 · v5.0.0-11.7.2

Download this guide as Markdown

Continue exploring