---
key: "tokenization-privacy-engineering"
title: "Tokenization, pseudonymization and privacy transformations: choose deliberately"
url: "https://softwaresecurity.com/domains/identity-crypto/privacy/tokenization-privacy-engineering/"
api: "https://softwaresecurity.com/api/v1/articles/tokenization-privacy-engineering.json"
domain: "identity-crypto"
topic: "Identity \u0026 cryptography"
type: "Guide"
identifiers: ["CWE-359","CWE-328","V14.1.1","V14.2.4","V14.2.6"]
tags: ["Encryption","Privacy engineering"]
stages: ["design","govern"]
roles: ["developer"]
license: "CC-BY-4.0"
license_url: "https://creativecommons.org/licenses/by/4.0/"
code_license: "MIT"
---

# Tokenization, pseudonymization and privacy transformations: choose deliberately

Choose between vault tokens, keyed pseudonyms, format-preserving encryption, aggregation and differential privacy by the joins, reversibility and leakage each allows, and test re-identification on the data you release.

Canonical URL: https://softwaresecurity.com/domains/identity-crypto/privacy/tokenization-privacy-engineering/

## In short

Use random tokens from a vault when only a few services may ever recover the original value, and keyed HMAC pseudonyms with a separate key per purpose when analytics needs stable joins; a plain hash of an email or phone number can be reversed by guessing. Format-preserving encryption is still encryption. None of these makes data anonymous: test what the released fields reveal, aggregate where you can, and use differential privacy for published statistics.

- A plain hash of a low-entropy identifier is reversible by guessing; use a keyed HMAC.
- Scope keys to one purpose and version so the same person does not get one global join key.
- Random vault tokens reveal nothing without the vault; whoever can detokenize holds the real data.
- Format-preserving encryption is reversible encryption; NIST’s draft revision keeps only FF1.
- Pseudonymized data is still personal data; uniqueness of the remaining fields decides re-identification risk.
- Differential privacy needs a defined mechanism and privacy budget; adding random noise is not enough.

## Recommended defaults

| Setting | Value | Note | Source |
| --- | --- | --- | --- |
| Keyed pseudonym | HMAC-SHA256 with a key per purpose and version; at least 128 bits of output | Derive purpose keys from a KMS-held master key, for example with HKDF. | RFC 2104; RFC 5869 |
| Vault token | At least 128 random bits, unrelated to the original value | — | ASVS 5.0 V11.5.1 |
| Format-preserving encryption | FF1 only, with a domain of at least 1,000,000 values | Required by the SP 800-38G Rev. 1 second public draft (February 2025), which removes FF3 and FF3-1. | NIST SP 800-38G Rev. 1 (draft) |
| PCI DSS scope | Out of scope only where the PAN cannot be retrieved | The token vault and every system that can detokenize stay in scope. | PCI SSC tokenization guidelines |
| Differential privacy | A documented mechanism, privacy unit and total budget across releases | — | NIST SP 800-226 (March 2025) |

## Verify it

1. Hash a list of known emails and confirm none of them matches a stored pseudonym.
2. Compute the pseudonym for one person under two purposes and confirm the values differ.
3. Ask each service holding tokens to detokenize one, and confirm only the approved services can.
4. Count how many released rows are unique on the quasi-identifiers a recipient could know.
5. Rotate the pseudonym key version in a test dataset and confirm old and new values do not join without the mapping step.

## What you will learn

- Distinguish random tokens, hashes, keyed pseudonyms, and encryption.
- Evaluate residual linkage beyond direct identifiers.
- Explain utility and key-rotation tradeoffs.

Audience: Data and backend engineers who share, analyze or store personal data, and privacy reviewers who assess re-identification risk.

Before you start:

- Distinguish identifiers, linkable records, reversible tokens and statistical disclosure.

Time: About 20 minutes: 6 to read and 14 for the exercise

## Changing identifiers does not necessarily remove identity

Tokenization replaces a sensitive value with a substitute. A vault-backed random token can refer to a separately protected mapping; a deterministic keyed transformation can produce the same token for the same input. Both can preserve linkage, which may be useful for analytics and also useful to an attacker.

Encryption is reversible with a key. Hashing is a one-way computation but does not hide low-entropy inputs from guessing: an attacker can hash a list of likely email addresses or phone numbers and compare results. A keyed HMAC changes that database-only guessing threat, but compromise or query access to the key-bearing service changes the analysis again.

Pseudonymization reduces direct exposure while retaining some ability to connect records. Anonymity is a stronger outcome that depends on other fields, external information, population size, and release context. Removing a name column does not establish it. An unusual combination of age, postcode, and event time can single out a person.

## Pick the transformation that supports only necessary operations

For customer support that must retrieve a payment reference, a random token and tightly controlled vault may be appropriate. The support system sees the token; a separate service decides which callers may resolve it. A token that any caller can detokenize provides little isolation from the original value.

For analytics that only needs counts by product, remove person-level identifiers entirely and aggregate early. If cross-event linkage is necessary, scope pseudonyms to that purpose or dataset. Using the same global token in marketing, support, and research creates a universal join key that defeats the intended separation.

A keyed deterministic token needs unambiguous input encoding and domain separation. Conceptually, `token=HMAC(K, encode(purpose, canonicalIdentifier))`. Canonicalization must match the application’s actual identifier semantics; casually lowercasing every address can merge values that the identity system treats differently. Domain separation ensures the same raw identifier does not automatically produce a cross-purpose join.

Rotating a tokenization key changes deterministic outputs and therefore joins. A migration may need versioned tokens or a controlled mapping period. That transition can temporarily increase linkage, so document who can access both versions. Deleting a vault entry can remove one resolution path, but cannot erase copies of plaintext previously exported by authorized clients; /domains/identity-crypto/privacy/data-minimization-retention/ covers deleting those copies.

## Create keyed, purpose-scoped pseudonyms in Python

This function derives one key per purpose and version from a master key with HKDF (see the cryptography quick reference at /domains/identity-crypto/encryption/crypto-quick-reference/), then computes an HMAC over the canonical identifier. The same person gets the same pseudonym within a purpose, different pseudonyms across purposes, and a new set after a key-version change. Keep the master key in a KMS. It was tested with cryptography 50 on Python 3.14, including Unicode normalization, case preservation, and a dictionary of plain hashes that fails to match.

```python
import hashlib
import hmac
import unicodedata

from cryptography.hazmat.primitives import hashes
from cryptography.hazmat.primitives.kdf.hkdf import HKDF


def purpose_key(master_key: bytes, purpose: str, version: int) -> bytes:
    """A separate key per purpose and version, so pseudonyms from two purposes never join."""
    info = f"pseudonym/{purpose}/v{version}".encode()
    return HKDF(algorithm=hashes.SHA256(), length=32, salt=None, info=info).derive(master_key)


def pseudonym(master_key: bytes, purpose: str, version: int, identifier: str) -> str:
    # Canonicalize exactly as the identity system compares identifiers; do not lowercase by habit.
    canonical = unicodedata.normalize("NFC", identifier.strip())
    tag = hmac.new(purpose_key(master_key, purpose, version), canonical.encode(), hashlib.sha256)
    return f"{purpose}.v{version}.{tag.hexdigest()[:32]}"  # 128 bits, versioned for rotation

```

## Where do format-preserving encryption and PCI tokenization fit?

Format-preserving encryption (FPE) encrypts a value into the same format, such as 16 digits, so legacy schemas and validators keep working. It is deterministic, reversible with the key, and weak over small domains. NIST SP 800-38G specifies the FF1 mode; the second public draft of its first revision (February 2025) removes FF3 and FF3-1 after published attacks and requires at least one million possible inputs. Use FPE only when the format constraint is real, and protect its key like any data-encryption key.

For payment card data, the PCI Security Standards Council’s tokenization guidance says tokenization can reduce the number of systems in PCI DSS scope, but only for systems from which the card number cannot be retrieved. The vault, the tokenization service and every system that can request detokenization remain in scope, and tokens produced by reversible encryption need the same key protection.

For published statistics, NIST SP 800-226 (March 2025) explains how to evaluate a differential privacy guarantee: what counts as one person’s contribution, how the privacy budget adds up across releases, and the implementation hazards, such as floating-point noise, that can undo it.

## Inspect the residual fields and the available side information

Suppose a released table contains a pseudonym, age band, district, and clinic visit day. A recipient who knows a person’s district and visit day may locate a unique row, even if the pseudonym is random. The privacy risk comes from the combination, not just from whether the token is cryptographically strong.

Generalization and suppression can reduce uniqueness but also reduce analytical value. A k-anonymity check on selected quasi-identifiers asks whether each released combination occurs at least k times. It does not prove that sensitive values within those groups are diverse or that all external linkage attacks are prevented.

Differential privacy addresses disclosure through a defined randomized mechanism and privacy accounting. Adding arbitrary noise to a table is not automatically differential privacy. The neighboring-dataset definition, contribution limits, mechanism, parameters, and composition across releases are part of the design.

Choose the release model as well as the transformation: internal restricted analysis, controlled query access, and public downloadable data have different risks. A transformation safe enough for a vetted analyst under access controls may be inadequate when the same rows are published indefinitely.

## Test intended utility and unwanted linkage

Verify that authorized operations still work: the correct service resolves an allowed token or computes the intended aggregate. The wrong service or purpose must not obtain the original value. Test identical inputs, different normalization forms, empty values, and key/version transitions.

For data releases, calculate uniqueness on the actual released fields and attempt documented linkage with plausible external knowledge. Record what was tested and what remains uncertain. A clean uniqueness check on one field subset is evidence about that subset, not a universal guarantee of anonymity.

Retain a decision record describing purpose, recipients, fields, transformation, key/vault access, permitted joins, retention, and residual risks. This makes the privacy choice reviewable when a new team asks to reuse the data for a different purpose.

## Guided practice

A team removes names and claims its small released dataset is anonymous.

### Supplied material

```text
Released rows:
T1 | age 30–39 | district North | visit Monday | condition X
T2 | age 30–39 | district North | visit Tuesday | condition X
T3 | age 40–49 | district South | visit Monday | condition Y
T4 | age 40–49 | district South | visit Monday | condition Y
External knowledge: Alice is 35, lives in North, and visited Monday.
Tokens T1–T4 are random; no vault access is available.
```

### 1. Attempt linkage

Find rows matching Alice’s supplied age band, district, and visit day.

Worked answer: Only T1 matches, so the recipient can single out the row under the supplied knowledge. Random token strength does not prevent that linkage.

### 2. Test one proposed generalization

Remove visit day and group by age band plus district.

Worked answer: Each group has two rows, so this subset meets k=2 for those quasi-identifiers. Both North rows still have condition X, illustrating that group size alone does not hide that attribute.

### 3. Choose purpose-limited output

If the task needs only total visits, choose an output and state the utility tradeoff.

Worked answer: Release a total of 4 with an appropriately reviewed aggregation/release policy, rather than person-level rows. This supports the stated total but cannot support individual follow-up or detailed segmentation.

Deliverable: A linkage calculation, a two-group table, and a transformation decision tied to the actual analytical purpose.

Scope: The dataset and external knowledge are synthetic. The exercise is not a proof of anonymity or a differential-privacy mechanism.

Articles and teaching material: CC BY 4.0. Original executable code: MIT. Attribution: Software Security.

## Sources

- [NIST SP 800-188: De-identification techniques and governance](https://csrc.nist.gov/pubs/sp/800/188/final). September 2023.
- [NIST Privacy Framework (CSWP 10)](https://csrc.nist.gov/pubs/cswp/10/nist-privacy-framework-version-10/final). Version 1.0 (January 2020).
- [RFC 2104: HMAC](https://www.rfc-editor.org/rfc/rfc2104.html). February 1997.
- [NIST SP 800-38G Rev. 1 (second public draft): format-preserving encryption](https://csrc.nist.gov/pubs/sp/800/38/g/r1/2pd). Second public draft, 3 February 2025.
- [NIST SP 800-226: Guidelines for Evaluating Differential Privacy Guarantees](https://csrc.nist.gov/pubs/sp/800/226/final). March 2025.
- [PCI SSC: PCI DSS tokenization guidelines](https://listings.pcisecuritystandards.org/documents/Tokenization_Guidelines_Info_Supplement.pdf). Version 2.0 (August 2011).
- [RFC 5869: HKDF](https://www.rfc-editor.org/rfc/rfc5869.html). May 2010.