---
key: "transaction-races"
title: "Preserve business invariants under concurrent requests"
url: "https://softwaresecurity.com/domains/appsec/business-logic/transaction-races/"
api: "https://softwaresecurity.com/api/v1/articles/transaction-races.json"
domain: "appsec"
topic: "Application security"
type: "Guide"
identifiers: ["CWE-362","CWE-367","CWE-837","A06:2025","V2.3.3","V2.3.4","V15.4.2","CAPEC-26","CAPEC-29"]
tags: ["Business logic and race conditions"]
stages: ["design","test"]
roles: ["developer","appsec-engineer"]
license: "CC-BY-4.0"
license_url: "https://creativecommons.org/licenses/by/4.0/"
code_license: "MIT"
---

# Preserve business invariants under concurrent requests

Make a balance, reservation or approval rule hold when requests overlap, retry and fail halfway through.

Canonical URL: https://softwaresecurity.com/domains/appsec/business-logic/transaction-races/

## In short

Do not check a limit in application code and write afterwards; another request can pass the same check in between. Put the check inside the write, for example `UPDATE accounts SET balance = balance - :amount WHERE id = :id AND balance >= :amount`, and require one affected row. Lock rows with `SELECT ... FOR UPDATE`, in a fixed order, when a rule spans several rows. Give each operation a unique id, and test with truly simultaneous requests.

- A race breaks the combined schedule: two requests that each read 100 can both spend 80.
- Transactions alone do not prevent it; the check and the change must happen atomically where all writers meet, usually the database.
- Use a conditional `UPDATE` for one row, `SELECT ... FOR UPDATE` in a consistent order or serializable transactions with whole-transaction retry for several.
- A unique operation id in the same transaction makes a replayed request roll back instead of debiting twice.
- Bind idempotency keys to the actor and the material inputs, and keep them for the full retry window.

## Recommended defaults

| Setting | Value | Note | Source |
| --- | --- | --- | --- |
| Single-row limit | `UPDATE ... SET balance = balance - :amount WHERE id = :id AND balance >= :amount`, then require 1 affected row | — | PostgreSQL: transaction isolation (Read Committed re-checks the WHERE clause on the updated row) |
| Multi-row invariant | `SELECT ... FOR UPDATE` on every row involved, ordered by key, or `SERIALIZABLE` isolation | Retry the whole transaction on SQLSTATE `40001` serialization failures and `40P01` deadlocks, with a bound. | PostgreSQL: explicit locking; transaction isolation |
| Operation identity | A `UNIQUE` or primary-key column for the operation id in the same transaction as the change | — | — |
| Balance floor | `CHECK (balance >= 0)` as a database backstop | Defense in depth; the conditional update is still the decision. | — |
| Idempotency record retention | At least the longest client and queue retry window | — | — |

## Verify it

1. Run two requests against the real database with a barrier or held lock so both pass the read before either writes; assert one acceptance.
2. Assert the invariant on stored state: ledger sum plus balance equals the starting balance, and the external effect count is one.
3. Replay a committed operation id and confirm the balance and ledger do not change.
4. Send 20 to 30 identical requests together with Burp Repeater's send group in parallel, or Turbo Intruder, against a test environment.
5. Retry a serialization failure in a test and confirm the whole transaction reruns with fresh reads.

## What you will learn

- Calculate invariants across concurrent schedules.
- Combine a condition and update at the shared database boundary.
- Separate atomic local state from retried external effects.

Audience: Backend developers who write balance, inventory, coupon or approval logic, and testers who probe it for race conditions.

Before you start:

- Understand a read-modify-write operation, transaction boundaries and retries.

Time: About 29 minutes: 8 to read, 4 on the code and 17 for the exercise

## Compute the invariant across an interleaving

A race is a failure of the combined schedule, even when each request looks correct alone. With 100 credits, requests T1 and T2 each read 100 and decide an 80-credit purchase is affordable. Both insert an accepted purchase and write their computed balance of 20. The final balance is nonnegative, but the ledger accepted 160 credits of spending. Testing only `balance >= 0` misses the loss of 60 credits. Define the invariant across all relevant representations: accepted spending plus remaining balance must equal the original balance, subject to explicitly modeled credits or refunds.

Wrapping those statements in transactions does not automatically solve the problem. Isolation and locking determine what concurrent transactions may observe and commit. A process-local mutex covers only one process, while two service replicas or a background worker can still race. Integer minor units avoid rounding ambiguity but do not serialize decisions. The repair must put the affordability condition where all competing writers meet.

Attackers look for exactly this window. Limit-overrun races let one gift card, coupon or withdrawal be redeemed several times, and PortSwigger's 2023 research showed that sending 20 to 30 HTTP/2 requests in a single TCP packet, the single-packet attack, makes them arrive within about a millisecond of each other, so a window of a few milliseconds is enough. /domains/appsec/business-logic/business-logic-flaws/ covers the related limit and workflow abuses.

## Make the condition part of the atomic database change

For a single-row balance, an `UPDATE` that subtracts only when `balance >= amount` combines the check and write. Require `amount` to be a positive permitted integer and inspect the affected-row count. One changed row means the debit happened; zero means the account was absent or insufficient under the query's scope. Insert the ledger row in the same transaction, and roll back if any later local requirement fails. Do not grant the purchase based solely on an earlier `SELECT`.

For invariants spanning several rows, use a reviewed locking or serializable-transaction design. Lock rows in a consistent order to reduce deadlocks and keep transactions short. PostgreSQL serialization failures require retrying the whole transaction with fresh state, not merely its final `UPDATE`. Retries need a bound and backoff so contention does not become an unbounded request storm. A unique constraint can express one reservation per operation and belongs in the database even when the application also checks first.

The Python program below runs both versions against PostgreSQL with psycopg 3. The unsafe version reads the balance, waits at a barrier until the other request has read it too, then writes its computed balance: both purchases are accepted, the ledger records 160 credits and the balance is 20. The repaired `purchase` makes the balance condition part of the `UPDATE` and records the operation id under a primary key in the same transaction: exactly one of two concurrent purchases succeeds, a replayed operation id raises a unique violation that rolls its debit back, the exact-balance purchase succeeds and zero or negative amounts are refused. A second test holds the first transaction open after its `UPDATE` and confirms the competing `UPDATE` waits for the row lock and then matches no row. Both ran against PostgreSQL 18.6, with each of the two requests winning about half of 30 races; set `DSN` to your database and run `python3 checkout_race.py`. Pass `purchase` an autocommit connection: inside an already open transaction, psycopg turns `conn.transaction()` into a savepoint, and nothing commits until the outer transaction does.

```python
"""checkout_race.py: one 80-credit purchase from 100 credits, under real concurrency.

Needs PostgreSQL and psycopg 3: DSN=postgresql://user:pass@host/db python3 checkout_race.py
"""
import os
import threading

import psycopg

DSN = os.environ.get("DSN", "postgresql://postgres:postgres@127.0.0.1:55432/postgres")


def purchase(conn, account, operation_id, amount):
    """Debit and record in one transaction; the WHERE clause is the affordability check."""
    if not isinstance(amount, int) or amount <= 0:
        raise ValueError("amount must be a positive integer")
    with conn.transaction():  # a top-level transaction on an autocommit connection
        debited = conn.execute(
            "UPDATE accounts SET balance = balance - %s WHERE id = %s AND balance >= %s",
            (amount, account, amount),
        ).rowcount
        if debited != 1:
            return False  # insufficient funds or no such account: nothing changed
        conn.execute(  # operation_id is UNIQUE: a replayed operation rolls the debit back
            "INSERT INTO ledger (operation_id, account_id, amount) VALUES (%s, %s, %s)",
            (operation_id, account, amount),
        )
        return True


def unsafe_purchase(conn, account, operation_id, amount, barrier):
    """The vulnerable shape: read, decide in the application, then write."""
    with conn.transaction():
        balance = conn.execute("SELECT balance FROM accounts WHERE id = %s", (account,)).fetchone()[0]
        barrier.wait()  # both requests have read 100 before either writes
        if balance < amount:
            return False
        conn.execute("UPDATE accounts SET balance = %s WHERE id = %s", (balance - amount, account))
        conn.execute("INSERT INTO ledger VALUES (%s, %s, %s)", (operation_id, account, amount))
        return True


def reset():
    with psycopg.connect(DSN, autocommit=True) as c:
        c.execute("DROP TABLE IF EXISTS ledger, accounts")
        c.execute("CREATE TABLE accounts (id int PRIMARY KEY, balance int NOT NULL)")
        c.execute("CREATE TABLE ledger (operation_id text PRIMARY KEY, account_id int NOT NULL,"
                  " amount int NOT NULL CHECK (amount > 0))")
        c.execute("INSERT INTO accounts VALUES (1, 100)")


def state():
    with psycopg.connect(DSN) as c:
        balance = c.execute("SELECT balance FROM accounts WHERE id = 1").fetchone()[0]
        spent = c.execute("SELECT coalesce(sum(amount), 0) FROM ledger").fetchone()[0]
        return balance, spent


def race(fn, *extra):
    """Runs T1 and T2 on separate connections at the same time."""
    results = {}
    def run(op):
        with psycopg.connect(DSN, autocommit=True) as conn:
            results[op] = fn(conn, 1, op, 80, *extra)
    threads = [threading.Thread(target=run, args=(op,)) for op in ("K1", "K2")]
    for t in threads: t.start()
    for t in threads: t.join()
    return sorted(results.values())


if __name__ == "__main__":
    reset()
    assert race(unsafe_purchase, threading.Barrier(2)) == [True, True]
    assert state() == (20, 160)  # both accepted: 160 credits spent from 100

    reset()
    assert race(purchase) == [False, True]
    assert state() == (20, 80)  # exactly one purchase, whichever wins

    with psycopg.connect(DSN, autocommit=True) as conn:
        winner = conn.execute("SELECT operation_id FROM ledger").fetchone()[0]
        try:
            purchase(conn, 1, winner, 20)  # replay of the committed operation id
            raise AssertionError("duplicate operation accepted")
        except psycopg.errors.UniqueViolation:
            pass
        assert state() == (20, 80)  # the duplicate's debit rolled back
        assert purchase(conn, 1, "K3", 20) is True and state() == (0, 100)  # exact balance
        for bad in (0, -1):
            try:
                purchase(conn, 1, "K4", bad)
                raise AssertionError(bad)
            except ValueError:
                pass
    print("PASS: unsafe version spent 160 of 100; conditional update allowed one purchase")

```

## Lock rows when the rule spans several records

A transfer changes two balances, and a seat booking may check a whole row of seats, so one conditional `UPDATE` is not enough. `SELECT ... FOR UPDATE` locks the rows the decision reads until the transaction ends, so a competing transaction waits and then reads the committed result. Lock every row involved in the same order, here by id, so two transfers in opposite directions cannot deadlock. In a test that ran 20 rounds of opposite transfers between the same two accounts at the same moment, every round completed with both transfers applied and the total unchanged.

Serializable isolation is the alternative when the set of rows is hard to lock in advance. PostgreSQL then aborts one of two conflicting transactions with SQLSTATE `40001`, and the application must rerun the whole transaction with fresh reads, not just its last statement. SQLite serializes writers with a database-level lock, and MySQL's InnoDB handles locking reads and gap locks differently, so test the pattern on the production engine.

```sql
-- Move :amount from :src to :dst; both rows stay locked until COMMIT.
BEGIN;
-- Lock both accounts in id order, so two opposite transfers cannot deadlock.
SELECT id, balance FROM accounts WHERE id IN (:src, :dst) ORDER BY id FOR UPDATE;
-- The application checks the locked balance and limits, then:
UPDATE accounts SET balance = balance - :amount WHERE id = :src AND balance >= :amount;
-- Require exactly 1 affected row; otherwise ROLLBACK and decline.
UPDATE accounts SET balance = balance + :amount WHERE id = :dst;
INSERT INTO ledger (operation_id, account_id, amount) VALUES (:operation_id, :src, :amount);
COMMIT;

```

## Carry one operation identity into external effects

A database can roll back a ledger insert, but cannot normally roll back a shipment already requested over HTTP. Store the accepted purchase and an outbox entry atomically, then let a worker deliver that intent. The worker may crash after the provider accepts the request but before recording completion. The queue then redelivers, so the receiver must support idempotency or the worker must reconcile with provider state before attempting another effect.

Scope an idempotency key to the actor and operation and bind it to material inputs such as amount, currency and recipient. Reusing a key with different inputs should conflict, not silently replay a different purchase. A key only in volatile process memory disappears on restart. A short deduplication expiry can also reopen a duplicate effect when a delayed retry arrives later. Define retention from the maximum supported retry and replay behavior rather than a convenient cache TTL.

## Verify results and contention deliberately

Use a supplied interleaving or a deterministic harness that makes both requests reach the decision point together. Assert accepted count, ledger sum, final balance and external effect count. For the conditional update, either request may win; the invariant is one acceptance, ledger 80 and balance 20. Tests that assert T1 must always win are accidentally testing scheduling rather than safety.

Include exact-balance spending, zero or negative amounts, duplicate operations and failure after durable intent. Test unrelated accounts progressing in parallel; a global lock may preserve correctness while destroying useful capacity. Operationally, distinguish genuine insufficient funds from a transient lock or serialization error. Monitor retries and contention by operation type, with bounded diagnostics that do not expose account balances unnecessarily. The browser lab at /labs/checkout-race/ and the checkout-sqlite project below provide runnable related practice, while the exercise below is complete as a paper schedule. SQLite and PostgreSQL lock differently, so confirm the behavior of your production database.

To test a deployed endpoint rather than the database layer, make the requests arrive together. Burp Suite Repeater can send a group of requests in parallel, using the single-packet attack over HTTP/2 and last-byte synchronization over HTTP/1, and Turbo Intruder scripts larger attacks. Run these only against systems you are authorized to test, and assert stored state afterwards: a 200 response for every request tells you nothing about the ledger.

## A real-world failure

Race conditions are not only an application concern. CVE-2016-5195 (Dirty COW) was a race condition (CWE-362) in the Linux kernel's copy-on-write handling that let a local user write to a read-only memory mapping, exploited in the wild to gain privileges. The same root cause, two operations on shared state interleaving without atomicity, drives the double-spend and balance races this guide prevents.

## Choose what an idempotent replay is allowed to reveal

An idempotency record is an authorization-sensitive resource. If keys are unique only globally and any caller can retrieve the stored response by guessing a key, one user can obtain another user's order result. Scope lookup to actor and operation, or enforce equivalent ownership before returning the replay. Bind the key to canonical material inputs; if `amount=80` and `amount=080` are accepted as the same integer, compare their validated meaning rather than arbitrary textual formatting.

The point where a key is reserved also affects recovery. Reserving it permanently before a transaction that later rolls back can strand an operation as in-progress. Recording it only after an external effect can permit duplicate effects during a crash. Model states such as pending, committed and reconciled with ownership and timeout rules, and make transitions durable. A timeout should not allow two workers to independently take over the same still-running operation.

The concurrency test must therefore cover more than two simultaneous debits. Add failure after key reservation, after ledger commit, after provider acceptance and before acknowledgement. Under the stated contract, all successful retries of K1 with the original inputs converge on one 80-credit ledger entry and one provider effect. A failed local transaction leaves no debit; a committed transaction never becomes eligible for a second debit merely because a response or worker acknowledgement was lost.

## Guided practice

Work through both an unsafe and repaired schedule for a 100-credit account.

### Supplied material

```text
Initial balance=100; ledger=[]; T1 amount=80 key=K1; T2 amount=80 key=K2.
Unsafe: T1 reads 100; T2 reads 100; T1 accepts and writes 20; T2 accepts and writes 20; both insert 80.
Repaired atomic rule: if current balance>=positive amount, subtract and add ledger in one transaction; otherwise no changes.
Exact-boundary independent case: balance=80, amount=80.
Invalid independent cases: amount=0; amount=-1.
Delivery: outbox K1 sent; provider records effect K1; worker crashes before ack; retry K1 arrives. Provider deduplicates the same key; different amount with same key conflicts.
```

### 1. Calculate the hidden loss

Compute final stored balance, ledger sum and invariant total for the unsafe schedule.

Worked answer: Stored balance=20 and ledger sum=160. Their total=180 rather than the initial 100, so 80 credits of accepted spending are unaccounted for in the balance reduction. Accepted spending exceeds the initial funds by 60; these are different useful measures.

### 2. Apply atomic decisions

Choose T1 first, execute both repaired operations, then reverse the order.

Worked answer: T1 first leaves 20 and `ledger[K1:80]`; T2 declines. Reversing yields `ledger[K2:80]` and the same 20 balance. Exactly one succeeds whichever wins.

### 3. Test the amount boundaries

Evaluate `balance=80` with `amount=80`, `amount=0` and `amount=-1` independently.

Worked answer: The exact-balance purchase succeeds and leaves 0. Zero and negative amounts are rejected before mutation; a negative subtraction would otherwise increase the balance and violate the purchase contract.

### 4. Trace uncertain external delivery

Apply the provider contract after the worker crash and then consider K1 reused for 90.

Worked answer: Retry K1 with the original 80 returns the existing effect and creates no second shipment. K1/90 conflicts because the key is bound to the original inputs. Durable intent alone cannot force an external provider to deduplicate without that contract.

Deliverable: A computed invariant table and one-operation retry trace covering local and external state.

Scope: Paper exercise: the schedule and provider contract are supplied. `checkout_race.py` above runs the unsafe and repaired schedules against a real PostgreSQL database.

## Executable practice

- [Race condition lab: Repair a balance check that races](https://softwaresecurity.com/labs/checkout-race/)
- [Business logic lab: Price an order without trusting the client](https://softwaresecurity.com/labs/coupon-limits/)
- [Make a checkout transaction atomic: project ZIP](https://softwaresecurity.com/downloads/projects/checkout-sqlite-0f06facdf0c0bb1b.zip) — Python 3.11+ · standard library

Articles and teaching material: CC BY 4.0. Original executable code: MIT. Attribution: Software Security.

## Sources

- [PostgreSQL transaction isolation](https://www.postgresql.org/docs/current/transaction-iso.html). PostgreSQL 18.
- [Amazon SQS duplicate-delivery behavior](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/standard-queues-at-least-once-delivery.html)
- [PostgreSQL: explicit locking](https://www.postgresql.org/docs/current/explicit-locking.html). PostgreSQL 18.
- [PortSwigger Web Security Academy: race conditions](https://portswigger.net/web-security/race-conditions)
- [James Kettle: Smashing the state machine (PortSwigger Research, 2023)](https://portswigger.net/research/smashing-the-state-machine). 9 August 2023.
- [MITRE CWE-362: race condition](https://cwe.mitre.org/data/definitions/362.html). CWE 4.20.
- [psycopg 3: transactions management](https://www.psycopg.org/psycopg3/docs/basic/transactions.html)
- [NVD: CVE-2016-5195, Linux kernel copy-on-write race condition (Dirty COW)](https://nvd.nist.gov/vuln/detail/cve-2016-5195)