Preserve business invariants under concurrent requests

A race is a failure of the combined schedule, even when each request looks correct alone. With 100 credits, requests T1 and T2 each read 100 and decide an 80-credit purchase is affordable. Both insert an accepted purchase and write their computed balance of 20. The final balance is nonnegative, but the ledger accepted 160 credits of spending. Testing only balance >= 0 misses the loss of 60 credits. Define the invariant across all relevant representations: accepted spending plus remaining balance must equal the original balance, subject to explicitly modeled credits or refunds.

Wrapping those statements in transactions does not automatically solve the problem. Isolation and locking determine what concurrent transactions may observe and commit. A process-local mutex covers only one process, while two service replicas or a background worker can still race. Integer minor units avoid rounding ambiguity but do not serialize decisions. The repair must put the affordability condition where all competing writers meet.

Attackers look for exactly this window. Limit-overrun races let one gift card, coupon or withdrawal be redeemed several times, and PortSwigger's 2023 research showed that sending 20 to 30 HTTP/2 requests in a single TCP packet, the single-packet attack, makes them arrive within about a millisecond of each other, so a window of a few milliseconds is enough. /domains/appsec/business-logic/business-logic-flaws/ covers the related limit and workflow abuses.

For a single-row balance, an UPDATE that subtracts only when balance >= amount combines the check and write. Require amount to be a positive permitted integer and inspect the affected-row count. One changed row means the debit happened; zero means the account was absent or insufficient under the query's scope. Insert the ledger row in the same transaction, and roll back if any later local requirement fails. Do not grant the purchase based solely on an earlier SELECT.

For invariants spanning several rows, use a reviewed locking or serializable-transaction design. Lock rows in a consistent order to reduce deadlocks and keep transactions short. PostgreSQL serialization failures require retrying the whole transaction with fresh state, not merely its final UPDATE. Retries need a bound and backoff so contention does not become an unbounded request storm. A unique constraint can express one reservation per operation and belongs in the database even when the application also checks first.

The Python program below runs both versions against PostgreSQL with psycopg 3. The unsafe version reads the balance, waits at a barrier until the other request has read it too, then writes its computed balance: both purchases are accepted, the ledger records 160 credits and the balance is 20. The repaired purchase makes the balance condition part of the UPDATE and records the operation id under a primary key in the same transaction: exactly one of two concurrent purchases succeeds, a replayed operation id raises a unique violation that rolls its debit back, the exact-balance purchase succeeds and zero or negative amounts are refused. A second test holds the first transaction open after its UPDATE and confirms the competing UPDATE waits for the row lock and then matches no row. Both ran against PostgreSQL 18.6, with each of the two requests winning about half of 30 races; set DSN to your database and run python3 checkout_race.py. Pass purchase an autocommit connection: inside an already open transaction, psycopg turns conn.transaction() into a savepoint, and nothing commits until the outer transaction does.

"""checkout_race.py: one 80-credit purchase from 100 credits, under real concurrency.

Needs PostgreSQL and psycopg 3: DSN=postgresql://user:pass@host/db python3 checkout_race.py
"""
import os
import threading

import psycopg

DSN = os.environ.get("DSN", "postgresql://postgres:postgres@127.0.0.1:55432/postgres")


def purchase(conn, account, operation_id, amount):
    """Debit and record in one transaction; the WHERE clause is the affordability check."""
    if not isinstance(amount, int) or amount <= 0:
        raise ValueError("amount must be a positive integer")
    with conn.transaction():  # a top-level transaction on an autocommit connection
        debited = conn.execute(
            "UPDATE accounts SET balance = balance - %s WHERE id = %s AND balance >= %s",
            (amount, account, amount),
        ).rowcount
        if debited != 1:
            return False  # insufficient funds or no such account: nothing changed
        conn.execute(  # operation_id is UNIQUE: a replayed operation rolls the debit back
            "INSERT INTO ledger (operation_id, account_id, amount) VALUES (%s, %s, %s)",
            (operation_id, account, amount),
        )
        return True


def unsafe_purchase(conn, account, operation_id, amount, barrier):
    """The vulnerable shape: read, decide in the application, then write."""
    with conn.transaction():
        balance = conn.execute("SELECT balance FROM accounts WHERE id = %s", (account,)).fetchone()[0]
        barrier.wait()  # both requests have read 100 before either writes
        if balance < amount:
            return False
        conn.execute("UPDATE accounts SET balance = %s WHERE id = %s", (balance - amount, account))
        conn.execute("INSERT INTO ledger VALUES (%s, %s, %s)", (operation_id, account, amount))
        return True


def reset():
    with psycopg.connect(DSN, autocommit=True) as c:
        c.execute("DROP TABLE IF EXISTS ledger, accounts")
        c.execute("CREATE TABLE accounts (id int PRIMARY KEY, balance int NOT NULL)")
        c.execute("CREATE TABLE ledger (operation_id text PRIMARY KEY, account_id int NOT NULL,"
                  " amount int NOT NULL CHECK (amount > 0))")
        c.execute("INSERT INTO accounts VALUES (1, 100)")


def state():
    with psycopg.connect(DSN) as c:
        balance = c.execute("SELECT balance FROM accounts WHERE id = 1").fetchone()[0]
        spent = c.execute("SELECT coalesce(sum(amount), 0) FROM ledger").fetchone()[0]
        return balance, spent


def race(fn, *extra):
    """Runs T1 and T2 on separate connections at the same time."""
    results = {}
    def run(op):
        with psycopg.connect(DSN, autocommit=True) as conn:
            results[op] = fn(conn, 1, op, 80, *extra)
    threads = [threading.Thread(target=run, args=(op,)) for op in ("K1", "K2")]
    for t in threads: t.start()
    for t in threads: t.join()
    return sorted(results.values())


if __name__ == "__main__":
    reset()
    assert race(unsafe_purchase, threading.Barrier(2)) == [True, True]
    assert state() == (20, 160)  # both accepted: 160 credits spent from 100

    reset()
    assert race(purchase) == [False, True]
    assert state() == (20, 80)  # exactly one purchase, whichever wins

    with psycopg.connect(DSN, autocommit=True) as conn:
        winner = conn.execute("SELECT operation_id FROM ledger").fetchone()[0]
        try:
            purchase(conn, 1, winner, 20)  # replay of the committed operation id
            raise AssertionError("duplicate operation accepted")
        except psycopg.errors.UniqueViolation:
            pass
        assert state() == (20, 80)  # the duplicate's debit rolled back
        assert purchase(conn, 1, "K3", 20) is True and state() == (0, 100)  # exact balance
        for bad in (0, -1):
            try:
                purchase(conn, 1, "K4", bad)
                raise AssertionError(bad)
            except ValueError:
                pass
    print("PASS: unsafe version spent 160 of 100; conditional update allowed one purchase")

A transfer changes two balances, and a seat booking may check a whole row of seats, so one conditional UPDATE is not enough. SELECT ... FOR UPDATE locks the rows the decision reads until the transaction ends, so a competing transaction waits and then reads the committed result. Lock every row involved in the same order, here by id, so two transfers in opposite directions cannot deadlock. In a test that ran 20 rounds of opposite transfers between the same two accounts at the same moment, every round completed with both transfers applied and the total unchanged.

Serializable isolation is the alternative when the set of rows is hard to lock in advance. PostgreSQL then aborts one of two conflicting transactions with SQLSTATE 40001, and the application must rerun the whole transaction with fresh reads, not just its last statement. SQLite serializes writers with a database-level lock, and MySQL's InnoDB handles locking reads and gap locks differently, so test the pattern on the production engine.

-- Move :amount from :src to :dst; both rows stay locked until COMMIT.
BEGIN;
-- Lock both accounts in id order, so two opposite transfers cannot deadlock.
SELECT id, balance FROM accounts WHERE id IN (:src, :dst) ORDER BY id FOR UPDATE;
-- The application checks the locked balance and limits, then:
UPDATE accounts SET balance = balance - :amount WHERE id = :src AND balance >= :amount;
-- Require exactly 1 affected row; otherwise ROLLBACK and decline.
UPDATE accounts SET balance = balance + :amount WHERE id = :dst;
INSERT INTO ledger (operation_id, account_id, amount) VALUES (:operation_id, :src, :amount);
COMMIT;

A database can roll back a ledger insert, but cannot normally roll back a shipment already requested over HTTP. Store the accepted purchase and an outbox entry atomically, then let a worker deliver that intent. The worker may crash after the provider accepts the request but before recording completion. The queue then redelivers, so the receiver must support idempotency or the worker must reconcile with provider state before attempting another effect.

Scope an idempotency key to the actor and operation and bind it to material inputs such as amount, currency and recipient. Reusing a key with different inputs should conflict, not silently replay a different purchase. A key only in volatile process memory disappears on restart. A short deduplication expiry can also reopen a duplicate effect when a delayed retry arrives later. Define retention from the maximum supported retry and replay behavior rather than a convenient cache TTL.

Use a supplied interleaving or a deterministic harness that makes both requests reach the decision point together. Assert accepted count, ledger sum, final balance and external effect count. For the conditional update, either request may win; the invariant is one acceptance, ledger 80 and balance 20. Tests that assert T1 must always win are accidentally testing scheduling rather than safety.

Include exact-balance spending, zero or negative amounts, duplicate operations and failure after durable intent. Test unrelated accounts progressing in parallel; a global lock may preserve correctness while destroying useful capacity. Operationally, distinguish genuine insufficient funds from a transient lock or serialization error. Monitor retries and contention by operation type, with bounded diagnostics that do not expose account balances unnecessarily. The browser lab at /labs/checkout-race/ and the checkout-sqlite project below provide runnable related practice, while the exercise below is complete as a paper schedule. SQLite and PostgreSQL lock differently, so confirm the behavior of your production database.

To test a deployed endpoint rather than the database layer, make the requests arrive together. Burp Suite Repeater can send a group of requests in parallel, using the single-packet attack over HTTP/2 and last-byte synchronization over HTTP/1, and Turbo Intruder scripts larger attacks. Run these only against systems you are authorized to test, and assert stored state afterwards: a 200 response for every request tells you nothing about the ledger.

Race conditions are not only an application concern. CVE-2016-5195 (Dirty COW) was a race condition (CWE-362) in the Linux kernel's copy-on-write handling that let a local user write to a read-only memory mapping, exploited in the wild to gain privileges. The same root cause, two operations on shared state interleaving without atomicity, drives the double-spend and balance races this guide prevents.

An idempotency record is an authorization-sensitive resource. If keys are unique only globally and any caller can retrieve the stored response by guessing a key, one user can obtain another user's order result. Scope lookup to actor and operation, or enforce equivalent ownership before returning the replay. Bind the key to canonical material inputs; if amount=80 and amount=080 are accepted as the same integer, compare their validated meaning rather than arbitrary textual formatting.

The point where a key is reserved also affects recovery. Reserving it permanently before a transaction that later rolls back can strand an operation as in-progress. Recording it only after an external effect can permit duplicate effects during a crash. Model states such as pending, committed and reconciled with ownership and timeout rules, and make transitions durable. A timeout should not allow two workers to independently take over the same still-running operation.

The concurrency test must therefore cover more than two simultaneous debits. Add failure after key reservation, after ledger commit, after provider acceptance and before acknowledgement. Under the stated contract, all successful retries of K1 with the original inputs converge on one 80-credit ledger entry and one provider effect. A failed local transaction leaves no debit; a committed transaction never becomes eligible for a second debit merely because a response or worker acknowledgement was lost.

Repair the code

Race condition lab: Repair a balance check that races

Business logic lab: Price an order without trusting the client

Sources & further reading

The standards, specifications and documentation behind the guidance above. Each is the evidence for a claim on this page; consult it for detail beyond the defaults and versions listed here.

1. PostgreSQL transaction isolation. PostgreSQL 18.
2. Amazon SQS duplicate-delivery behavior
3. PostgreSQL: explicit locking. PostgreSQL 18.
4. PortSwigger Web Security Academy: race conditions
5. James Kettle: Smashing the state machine (PortSwigger Research, 2023). 9 August 2023.
6. MITRE CWE-362: race condition. CWE 4.20.
7. psycopg 3: transactions management
8. NVD: CVE-2016-5195, Linux kernel copy-on-write race condition (Dirty COW)

Reference information

Tags

Business logic and race conditions

SDLC phase

DesignTesting

Role

DevelopersAppSec engineers

ASVS 5.0.0 requirements

v5.0.0-2.3.3 · v5.0.0-2.3.4 · v5.0.0-15.4.2

CAPEC attack patterns

• CAPEC-26 at MITRE
• CAPEC-29 at MITRE

Download this guide as Markdown

Continue exploring