# Software Security: all guides and fix cards > 213 guides and 188 fix cards from Software Security (https://softwaresecurity.com/). Licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/); original code is MIT licensed. The index is https://softwaresecurity.com/llms.txt. # Fix SQL injection in Python sqlite3 Bind query values separately from SQL syntax, then prove the fix preserves legitimate behavior and blocks an injection probe. Canonical URL: https://softwaresecurity.com/domains/appsec/secure-coding/sql-injection-python-sqlite3/ ## In short Never build SQL text from input. Keep the statement constant and pass values separately: `con.execute("SELECT id FROM accounts WHERE owner = ?", (owner,))`. The driver sends the value as data, so a quote or `OR 1=1` cannot change the query. Binding covers values only: map sort columns and other identifiers through an allowlist, and still check that the caller may read the rows the query selects. - Use `?` or `:name` placeholders with a separate parameter sequence; `(owner,)` needs its trailing comma. - Do not quote the placeholder or pre-escape quotes: `O'Connor` is valid data that binding preserves. - Identifiers and keywords cannot be bound: map a sort choice such as `newest` to a fixed fragment such as `id DESC`. - For an `IN` list, generate one placeholder per element, bind every element and cap the list length. - Binding stops injection, not unauthorized reads: keep the owner or tenant condition in the query. ## Verify it 1. Run the program and confirm it prints `PASS: injection blocked; legitimate lookup preserved`. 2. Assert that the probe `' OR 1=1 --` returns `[]` from the fixed lookup while `alice` returns `[(1,)]`. 3. Insert an `O'Connor` row and assert the fixed lookup returns it unchanged. 4. Search for `execute(` calls whose SQL is built with `+`, f-strings, `%` or `.format()` and replace each with placeholders. 5. Check that every user-selected column, direction or table name passes through an allowlist map before it reaches SQL. ## What you will learn - Explain how concatenation changes SQL grammar. - Use sqlite3 binding without corrupting valid values. - Distinguish value binding, identifier selection and object authorization. Audience: Python developers fixing a SQL injection finding, and reviewers checking that the fix keeps legitimate lookups working. Before you start: - Read a Python function, a SELECT query, and a bound parameter tuple. Time: About 27 minutes: 7 to read, 3 on the code and 17 for the exercise ## Understand why a value becomes SQL syntax The failure starts before SQLite looks up an account. The vulnerable function builds one string containing both the developer's query and the supplied owner. For `alice`, the resulting condition is `owner = 'alice'`. For the supplied probe, the first quote closes the string literal, `OR` introduces another condition, `1=1` makes it true for every row, and the two hyphens comment out the trailing quote. The database faithfully executes the resulting program. It cannot infer which characters the programmer intended as data. This exercise uses Python's standard-library sqlite3 module, an in-memory database and two fictional accounts. No server, credentials or third-party package is required. Its narrow security property is that a supplied owner cannot change the query's grammar. It does not establish that the caller is entitled to select that owner. Alice supplying the ordinary text `bob` is an authorization problem even after SQL injection is fixed. Keep those two assertions separate when reviewing a patch. ## Run the vulnerable and repaired lookup Copy the entire block into a file named `sql_binding_demo.py` and run `python3 sql_binding_demo.py`. The vulnerable helper is a local demonstration only. The repaired helper sends fixed SQL text and a separate sequence of values through the database driver's binding interface. SQLite treats the probe as one text value, so the `OR` and comment characters have no syntactic role. There is no account whose literal owner equals that complete string, so the answer is an empty list. The lab at /labs/sql-injection-sqlite/ has you make the same repair, plus a literal `LIKE` search, in the browser. The trailing comma in `(owner,)` forms a one-element tuple. A parenthesized string is still a string and may be interpreted as a sequence of parameters. Do not quote the placeholder: `WHERE owner = '?'` compares against the literal question mark rather than binding the supplied owner. Do not pre-escape quotes either; that changes legitimate stored values and recreates a second, inconsistent encoding scheme. ```python import sqlite3 con = sqlite3.connect(":memory:") try: con.execute("CREATE TABLE accounts (id INTEGER, owner TEXT)") con.executemany("INSERT INTO accounts VALUES (?, ?)", [(1, "alice"), (2, "bob")]) def vulnerable(owner): # Demonstration only; do not deploy. sql = "SELECT id FROM accounts WHERE owner = '" + owner + "'" return con.execute(sql).fetchall() def fixed(owner): return con.execute( "SELECT id FROM accounts WHERE owner = ?", (owner,) ).fetchall() probe = "' OR 1=1 --" assert len(vulnerable(probe)) == 2 assert fixed(probe) == [] assert fixed("alice") == [(1,)] assert fixed("nobody") == [] assert fixed("O'Connor") == [] print("PASS: injection blocked; legitimate lookup preserved") finally: con.close() ``` ## Use the same binding in other drivers Every mainstream database API has a prepared-statement or parameter interface; only the placeholder syntax changes. The rule is identical in each: the SQL text is a constant written by the developer, and each value is passed through the driver as a separate argument. Never format the values into the string yourself, even with a driver that uses `%s`: psycopg replaces `%s` safely only when the values go in the second argument of `execute`. /fix/sql-injection/ shows the same repair for each common framework. The Java program below is the JDBC version of the same lookup, run against an in-memory SQLite database through the `org.xerial:sqlite-jdbc` driver. `PreparedStatement` sends the statement and the owner separately, so the probe matches nothing and `O'Connor` keeps its quote. ORM escape hatches need the same care: Django `raw()` and `RawSQL`, SQLAlchemy `text()` and Hibernate query strings are safe with bound parameters and injectable when built by concatenation. A stored procedure that assembles dynamic SQL with string concatenation inside the database reintroduces the flaw even when the call itself is parameterized. ```java import java.sql.*; import java.util.ArrayList; import java.util.List; public class AccountLookup { // The SQL text is a constant; the owner travels separately as a bound value. static List idsForOwner(Connection con, String owner) throws SQLException { String sql = "SELECT id FROM accounts WHERE owner = ? ORDER BY id"; try (PreparedStatement ps = con.prepareStatement(sql)) { ps.setString(1, owner); try (ResultSet rs = ps.executeQuery()) { List ids = new ArrayList<>(); while (rs.next()) ids.add(rs.getInt("id")); return ids; } } } public static void main(String[] args) throws SQLException { // Needs a JDBC driver on the classpath, for example org.xerial:sqlite-jdbc. try (Connection con = DriverManager.getConnection("jdbc:sqlite::memory:"); Statement st = con.createStatement()) { st.execute("CREATE TABLE accounts (id INTEGER, owner TEXT)"); st.execute("INSERT INTO accounts VALUES (1, 'alice'), (2, 'bob'), (3, 'O''Connor')"); check(idsForOwner(con, "' OR 1=1 --").isEmpty(), "probe is data"); check(idsForOwner(con, "alice").equals(List.of(1)), "alice"); check(idsForOwner(con, "O'Connor").equals(List.of(3)), "quote preserved"); System.out.println("PASS: PreparedStatement binds the owner as data"); } } static void check(boolean ok, String what) { if (!ok) throw new AssertionError(what); } } ``` | Stack | Placeholder | Bound call | | --- | --- | --- | | Python sqlite3 | `?` or `:owner` | `con.execute(sql, (owner,))` | | Python psycopg (PostgreSQL) | `%s` or `%(owner)s` | `cur.execute(sql, (owner,))` | | SQLAlchemy | `:owner` | `conn.execute(text(sql), {"owner": owner})` | | Java JDBC | `?` | `ps.setString(1, owner)` on a `PreparedStatement` | | .NET SqlClient | `@owner` | `cmd.Parameters.Add("@owner", SqlDbType.NVarChar, 100).Value = owner` | | Node.js pg | `$1` | `client.query(sql, [owner])` | | Go database/sql | `$1` (PostgreSQL) or `?` (MySQL, SQLite) | `db.QueryContext(ctx, sql, owner)` | | PHP PDO | `:owner` | `$stmt->execute(['owner' => $owner])` | | Ruby on Rails | `?` or a hash condition | `Account.where(owner: owner)` | ## Interpret each result and strengthen the regression The first assertion demonstrates impact: the probe obtains two rows from a lookup that should match an owner. The fixed assertions then test the unwanted behavior and ordinary behavior independently. Seeing no exception is insufficient; a filter that rejects every request would also stop this probe. Alice must still produce `[(1,)]`, while `nobody` and the literal probe produce `[]`. `O'Connor` is a useful boundary because a quote is valid data, not evidence that the person is attacking the application. To distinguish quote handling from a missing-record result, insert an `O'Connor` account and assert that the fixed lookup retrieves it. Also test the empty string deliberately: an application may reject it as invalid input, while SQL binding itself correctly compares it as data. If a test fails, inspect the statement shape and parameter count before adding a character blacklist. In a real service, record a request identifier and database error category without logging passwords, full query values or an unrestricted stack trace. | Input in the original fixture | Vulnerable result | Fixed result | | --- | --- | --- | | `alice` | `[(1,)]` | `[(1,)]` | | `' OR 1=1 --` | `[(1,), (2,)]` | `[]` | | `O'Connor` | SQL syntax error | `[]` | | `nobody` | `[]` | `[]` | ## Handle identifiers, authorization and later reuse Binding applies to values, not identifiers or SQL keywords. A user-selected sort field needs a mapping such as `newest` → `created_at DESC` and `owner` → `owner ASC`, with only developer-authored fragments inserted into the statement. Binding `ORDER BY ?` would sort by a constant value rather than select a column. Reject an unknown sort choice before executing the query. For an `IN` list, generate the required number of placeholders and bind every element; define empty-list behavior explicitly rather than constructing invalid SQL. Stored text stays untrusted when reused. A safely inserted display name can become a second-order injection if an export later concatenates it into a new query. Follow the value through each execution call. Finally, derive tenant and caller constraints from authenticated server context and include them in the lookup. Use a database identity with only necessary permissions; this limits damage if another route is vulnerable but cannot repair a broken lookup. The runnable example proves its local binding property; other routes and queries need their own tests. ## A real-world failure In May and June 2023 the Cl0p group mass-exploited CVE-2023-34362, an unauthenticated SQL injection in Progress MOVEit Transfer, to steal data from thousands of organizations. The missing control was exactly what this recipe enforces: passing values as parameters instead of building SQL from strings. ## Extend the binding rule to variable-length filters A common follow-up is selecting several ids. If the approved id list is `[1, 2]`, the statement shape can be `WHERE id IN (?, ?)` with parameters `(1, 2)`. The program constructs only the count of placeholders, never the submitted values. If the list is empty, return an empty result or deliberately skip the query according to the API contract; do not interpolate an empty string and hope every database interprets `IN ()` identically. Cap the list length as a separate resource rule. Binding ten thousand values may preserve grammar while still consuming unacceptable work. `LIKE` introduces another distinction. Binding the value `%` is safe from SQL injection, but `%` retains its documented wildcard meaning inside a `LIKE` expression. If the feature promises a literal substring search, the application must apply the database's supported wildcard-escaping convention and specify the matching `ESCAPE` behavior. If wildcard search is intentional, retain it and limit result count. A bound value's inability to rewrite SQL does not mean that every value has the same business meaning. Finally, distinguish `SELECT` permissions from write permissions. An `UPDATE accounts SET owner=? WHERE id=?` is bound correctly but can still transfer ownership if `id` is caller-controlled and the actor is unauthorized. Require the permitted owner or tenant constraint, then inspect the affected-row result. A regression for grammar safety and a regression for ownership safety should fail for different reasons so later maintainers can see which property a change breaks. ## Guided practice Run the complete sqlite3 program above, then reason about one extension. ### Supplied material ```text Existing rows: (1, alice), (2, bob). Additional row for step 3: (3, O'Connor). Authenticated caller: alice. Requested sort choices (three separate inputs): S1=newest S2=owner S3=owner; DROP TABLE accounts Allowed map: newest => id DESC; owner => owner ASC. ``` ### 1. Predict the SQL grammar Expand the vulnerable query for the supplied probe before running the file. Mark the string close, new predicate and comment. Worked answer: The condition becomes `owner = '' OR 1=1`, with the final quote commented out. Both ids qualify. The issue does not require a second SQL statement. ### 2. Execute the regression Save the complete code block as `sql_binding_demo.py` and run `python3 sql_binding_demo.py`. Compare the printed result with the table. Worked answer: The program prints `PASS: injection blocked; legitimate lookup preserved`. A failing assertion identifies a behavior change; a syntax error indicates the pasted program or local interpreter needs correction. ### 3. Prove quotes survive Before the print line, add `con.execute("INSERT INTO accounts VALUES (?, ?)", (3, "O'Connor"))` and assert `fixed("O'Connor") == [(3,)]`. Run the file again. Worked answer: The same PASS message appears. A quote is preserved as data in both `INSERT` and `SELECT`; removing or doubling it in application input would corrupt this behavior. ### 4. Separate two remaining decisions Evaluate the three supplied sort strings against the map, then decide whether Alice may request `fixed("bob")`. Worked answer: `newest` selects `id DESC` and `owner` selects `owner ASC`. The entire third string is unknown and must be rejected. `fixed("bob")` correctly returns `[(2,)]` at the database level but the application must deny Alice if the rule permits only her own account. Deliverable: A passing local program and four explicit explanations: vulnerable grammar, binding, legitimate quote handling, and the separate authorization rule. Scope: Requires Python 3 with its standard-library sqlite3 module. The program creates only an in-memory database and two fictional accounts. ## Executable practice - [SQL injection lab: Separate SQL structure from data with sqlite3](https://softwaresecurity.com/labs/sql-injection-sqlite/) Articles and teaching material: CC BY 4.0. Original executable code: MIT. Attribution: Software Security. ## Sources - [Python sqlite3: parameter substitution](https://docs.python.org/3/library/sqlite3.html). Python 3.15. - [OWASP SQL Injection Prevention](https://cheatsheetseries.owasp.org/cheatsheets/SQL_Injection_Prevention_Cheat_Sheet.html) - [MITRE CWE-89: SQL injection](https://cwe.mitre.org/data/definitions/89.html). CWE 4.20. - [OWASP Query Parameterization Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/Query_Parameterization_Cheat_Sheet.html) - [Java SE 21: java.sql.PreparedStatement](https://docs.oracle.com/en/java/javase/21/docs/api/java.sql/java/sql/PreparedStatement.html). Java SE 21. - [psycopg 3: passing parameters to SQL queries](https://www.psycopg.org/psycopg3/docs/basic/params.html) - [NVD: CVE-2023-34362, SQL injection in MOVEit Transfer](https://nvd.nist.gov/vuln/detail/cve-2023-34362) --- # Enforce authorization for every object action Authorize the authenticated subject, requested operation, and specific resource together; a valid login or unpredictable identifier is insufficient. Canonical URL: https://softwaresecurity.com/domains/appsec/authorization/object-level-authorization/ ## In short Check permission for the specific object on every request, on the server, after loading it and before releasing anything. Take the caller's id, tenant and roles from the verified session, never from the request. Encode the whole rule, for example active AND same tenant AND (owner OR billing admin), and apply it to lists, exports, attachments and cached responses too. Random ids make guessing harder but are not authorization. - Authentication says who is calling; object authorization says whether this caller may perform this action on this record now. - Keep role exceptions inside the tenant boundary: `active && sameTenant && (owner || billingAdmin)`. - Scope the query to permitted records, or check the loaded record before serializing, logging or caching it. - Return the same 404 for missing and forbidden records when the contract hides existence. - Apply the rule to lists, counts, search, exports, attachments, background jobs and caches, not only the detail route. ## Verify it 1. Run a matrix that changes one fact at a time: owner, tenant, role, active flag and a missing id. 2. Call the API directly with another user's object id and assert that no fields, totals or links appear in the response. 3. Revoke membership or transfer the record between two requests and confirm the second request is denied, including from cache. 4. Request a bulk export containing one forbidden id and confirm the documented atomic or partial result. 5. Check every route that loads the record and confirm the decision runs before serialization, logging or caching. ## What you will learn - Translate a business rule into a complete predicate. - Verify tenant and ownership exceptions independently. - Find authorization bypasses across routes and caches. Audience: API and backend developers adding record-level access checks, and reviewers who test them. Before you start: - Distinguish the authenticated actor, active tenant, record owner, and requested action. Time: About 24 minutes: 8 to read and 16 for the exercise ## Turn a role label into a decision rule Authentication answers who is making the request. Object authorization answers whether that identity may perform this action on this particular resource now. A route such as `GET /invoices/502` contains an attacker-controlled selector, even when the interface never displays invoice 502. Random identifiers reduce guessing but cannot make a leaked link safe. The server must evaluate permission after establishing identity and before releasing the selected data. Use this concrete rule: an active member may read an invoice only when its tenant equals the member's tenant and its owner equals the member's user id; a billing administrator may read any invoice within that administrator's tenant. The administrator exception does not remove the tenant boundary. Write the rule with parentheses: `active AND sameTenant AND (owner OR billingAdmin)`. An expression that places `billingAdmin` outside the tenant conjunction accidentally creates a global administrator. Define update, download, refund and export separately rather than assuming read permission grants them all; /domains/appsec/authorization/function-property-authorization/ covers those action-level checks. ## Place selection and permission on the same path A vulnerable implementation authenticates Alice, reads the requested invoice by id alone and serializes it. A repaired implementation either scopes the database query to permitted records, for example `WHERE id = ? AND tenant = ? AND (owner = ? OR ? = 'billing_admin')`, or evaluates the complete predicate on the loaded record before serialization. Derive user id, tenant and roles from verified server context. Taking tenant from a request header and user id from a JSON body simply lets the caller provide the answer to the permission question. The same boundary applies to nested routes. A URL containing `/tenants/red/invoices/502` is not proof that invoice 502 belongs to red. Check the actual relationship. Apply the rule to list results, counts, search indexes, attachment endpoints and background exports as well as the primary detail route. A bulk endpoint should authorize every selected object or reject the request atomically according to a documented contract. Checking only the first object leaves later identifiers uncontrolled. The module below is a runnable version of the rule. Save it as `invoice-access.mjs` and run `node invoice-access.mjs` with Node.js 22 or later; it prints `PASS: 7 authorization cases`. `readInvoice` returns the same 404 body for a missing invoice and a forbidden one, and its response is an explicit projection rather than the stored record. Moving the role check outside the tenant comparison makes the Cara/601 assertion fail, which is the regression this test exists to catch. ```javascript // Run: node invoice-access.mjs import assert from 'node:assert/strict'; // Server-side records; in production this is a database lookup. const invoices = new Map([ ['501', {id: '501', tenant: 'red', owner: 'A', total: 40}], ['502', {id: '502', tenant: 'red', owner: 'B', total: 70}], ['601', {id: '601', tenant: 'blue', owner: 'D', total: 90}], ]); // The complete rule, with the role exception inside the tenant boundary. export function canRead(caller, invoice) { return ( caller.active === true && caller.tenant === invoice.tenant && (caller.id === invoice.owner || caller.role === 'billing_admin') ); } // `caller` comes from the verified session, never from the request body or headers. export function readInvoice(caller, id, repo = invoices) { const invoice = repo.get(String(id)); if (!invoice || !canRead(caller, invoice)) return {status: 404, body: {error: 'not_found'}}; return {status: 200, body: {id: invoice.id, total: invoice.total}}; // explicit projection } const alice = {id: 'A', tenant: 'red', role: 'member', active: true}; const cara = {id: 'C', tenant: 'red', role: 'billing_admin', active: true}; const dan = {id: 'D', tenant: 'blue', role: 'billing_admin', active: true}; const denied = {status: 404, body: {error: 'not_found'}}; assert.deepEqual(readInvoice(alice, 501), {status: 200, body: {id: '501', total: 40}}); assert.deepEqual(readInvoice(alice, 502), denied); // not the owner assert.deepEqual(readInvoice(cara, 502), {status: 200, body: {id: '502', total: 70}}); assert.deepEqual(readInvoice(cara, 601), denied); // admin role stays inside its tenant assert.deepEqual(readInvoice(dan, 601), {status: 200, body: {id: '601', total: 90}}); assert.deepEqual(readInvoice(cara, 999), denied); // missing looks the same as forbidden assert.deepEqual(readInvoice({...cara, active: false}, 502), denied); // revoked member console.log('PASS: 7 authorization cases'); ``` ## Use a matrix that changes one fact at a time The fixture below makes the exception and boundary independently visible. Alice owns invoice 501 in red. Ben is another ordinary red member. Cara is a red billing administrator. Dan is a blue billing administrator. Invoice 502 belongs to Ben in red; invoice 601 belongs to Dan in blue. Comparing Alice/501 with Alice/502 isolates ownership. Comparing Cara/502 with Cara/601 isolates tenant scope despite the same privileged role. For the stated contract, every denied lookup returns the same 404 body as a missing invoice. That avoids deliberately advertising existence; it does not guarantee identical timing or hide information available elsewhere. Assert no invoice fields, totals, attachment locations or cache entries for the denied resource escape. Also test revocation: mark Cara inactive before a second request and require denial. The IDOR lab at /labs/authorization/ runs the same ownership and tenant cases against a handler you fix yourself. If permissions are cached, the cache must include identity and relevant policy context, with an explicit invalidation or expiry design. ## Keep authorization understandable during operation A useful decision log contains an internal request id, actor id, action, resource reference, tenant context, outcome and policy reason. It should not contain the invoice document or session token. Group denied requests to spot repeated cross-tenant probing while recognizing that stale bookmarks also cause legitimate denials. A log is evidence of what the service decided, not a substitute for preventing the response. Centralizing a policy helps consistency, but a helper can still be skipped. Review every entry point and make unauthorized resource access difficult by construction, for example through a repository function that requires caller context. For writes, keep permission and state changes consistent with concurrent revocation or ownership transfer; high-impact actions may require a transaction or a conditional update. Test both the policy and the route because a perfect policy function provides no protection when a handler serializes data before calling it. ## Enforce the rule with your framework, not around it Framework features help only when they see the loaded resource. A route-level role check such as an `[Authorize]` attribute or a `@PreAuthorize("hasRole(...)")` annotation knows the caller but not the invoice, so it cannot decide ownership or tenant. Use the resource-aware hook each framework provides (/fix/missing-authorization/ has a fix for each common stack), and keep list endpoints on a filtered query because per-object checks rarely run for every row of a list. - Django REST framework: `has_object_permission` runs only when the view calls `get_object()`; list views need a `get_queryset` filtered to the caller. - Spring Security: method security with `@PreAuthorize` or `@PostAuthorize` that passes the loaded object to a policy bean, or a custom `AuthorizationManager`. - ASP.NET Core: resource-based authorization, `IAuthorizationService.AuthorizeAsync(User, invoice, "CanRead")`, called after loading the invoice. - Rails: scope through the association, for example `current_user.invoices.find(params[:id])`, or Pundit `authorize` with `policy_scope` for lists. - Policy engines such as OPA, Cedar or OpenFGA centralize the rule; the application must still call them with the loaded resource and act on a denial. - PostgreSQL row-level security adds defense in depth for tenant scope. Superusers and `BYPASSRLS` roles always bypass it, and table owners do too unless the table uses `FORCE ROW LEVEL SECURITY`. - GraphQL: authorize in every resolver or in the shared data loader, because one query can reach the same object through several fields. ## A real-world failure In 2019 First American Financial exposed hundreds of millions of title and escrow document images (about 800 million by the SEC's count, 885 million by the reporting journalist's) because any document link could be edited to a sequential number to read another customer's file, with no authorization check and no authentication required. This is a textbook insecure direct object reference; the missing control was authorizing every object access against the caller. The regulators' orders add the governance failure: the company's own penetration test had reported the flaw months earlier, and a clerical error logged it at a lower severity so it was never fixed (/domains/appsec/authorization/case-study-first-american/). ## Choose bulk and delegated-access semantics explicitly Suppose Alice requests invoices [501, 502] in one export. Under the fixture's ownership rule, she may read only 501. The endpoint needs a deliberate contract: reject the entire request, or return a documented partial result that contains only permitted objects. This guide chooses atomic rejection for an explicitly selected export: the result is a denial and zero exported invoices. Quietly dropping 502 may be acceptable for a broad search, but can mislead a caller who believes a requested financial export is complete. The distinction comes from the operation's meaning, not a universal rule that partial responses are always wrong. For delegated access, add an explicit grant with subject, resource scope, action and expiry. A support engineer temporarily allowed to read invoice 501 does not automatically gain permission to refund it, export the entire tenant or retain access after the grant expires. Evaluate grants together with current tenant and resource state. A pre-signed download link is itself a bearer capability: whoever has the link may be able to retrieve the file until expiry. Issuing one is therefore an authorization-sensitive action with a lifetime decision, not merely a performance optimization. When testing delegation, hold the resource constant and vary action and time. With a grant that expires at time 10, a read at time 9 may succeed, while a refund at time 9 and a read at time 10 should both be denied under an exclusive-expiry rule. Record that exact boundary in the policy rather than leaving different services to interpret it differently. ## Guided practice Evaluate a complete invoice permission matrix; then review the route ordering. ### Supplied material ```text Callers: Alice=(id A, tenant red, member, active); Ben=(B, red, member, active); Cara=(C, red, billing_admin, active); Dan=(D, blue, billing_admin, active). Invoices: 501=(red,A,total 40); 502=(red,B,total 70); 601=(blue,D,total 90). Requests: A/501; A/502; C/502; C/601; D/601; C/999. Policy: active AND sameTenant AND (owner OR billing_admin). Denial contract: 404 {error: not_found}; no invoice fields. Buggy cache key: invoice:{id}. ``` ### 1. Compute the decisions Apply every term of the predicate to the six requests and write status plus visible total. Worked answer: A/501 → 200,40; A/502 → 404; C/502 → 200,70; C/601 → 404; D/601 → 200,90; C/999 → 404. Cara's role does not bypass tenant scope. ### 2. Find the route bypass Consider a handler that loads and serializes invoice 601 before calling `canRead(Cara, invoice)`. Identify the earliest forbidden effect. Worked answer: Serialization may already expose fields through a response, log or cache. The decision belongs before any release. Loading internally can be acceptable; releasing data before denial cannot. ### 3. Evaluate revoked membership Set `Cara.active=false` and repeat C/502. Then restore active and transfer invoice 502 to blue. Worked answer: Both variants return 404. The first fails active; the second fails sameTenant. A cached earlier allow must not silently override either change. ### 4. Repair response caching Dan first caches invoice 601 under `invoice:601`. Cara requests the same id. Explain why a cache hit cannot skip the policy. Worked answer: The shared key identifies the object but not permission. Authorize against current object metadata before returning cached content, or use a policy-aware cache with invalidation. Cara must receive 404 even if the document is already cached. Deliverable: A six-row access table, two revocation results and a cache design that cannot return a foreign invoice. Scope: Paper exercise: every caller, invoice and policy term is supplied above. The runnable `invoice-access.mjs` module and the browser lab exercise the same rule in code. ## Executable practice - [IDOR lab: Repair an invoice authorization bug](https://softwaresecurity.com/labs/authorization/) Articles and teaching material: CC BY 4.0. Original executable code: MIT. Attribution: Software Security. ## Sources - [OWASP Authorization](https://cheatsheetseries.owasp.org/cheatsheets/Authorization_Cheat_Sheet.html) - [OWASP Insecure Direct Object Reference Prevention](https://cheatsheetseries.owasp.org/cheatsheets/Insecure_Direct_Object_Reference_Prevention_Cheat_Sheet.html) - [OWASP API1:2023 Broken Object Level Authorization](https://api-security.owasp.org/editions/2023/en/0xa1-broken-object-level-authorization/) - [Django REST framework: permissions and object-level checks](https://www.django-rest-framework.org/api-guide/permissions/) - [ASP.NET Core: resource-based authorization](https://learn.microsoft.com/en-us/aspnet/core/security/authorization/resource-based?view=aspnetcore-10.0). ASP.NET Core 10.0. - [PostgreSQL: row security policies](https://www.postgresql.org/docs/current/ddl-rowsecurity.html). PostgreSQL 18. - [KrebsOnSecurity: First American Financial Corp. leaked hundreds of millions of title insurance records](https://krebsonsecurity.com/2019/05/first-american-financial-corp-leaked-hundreds-of-millions-of-title-insurance-records/). 24 May 2019. - [SEC: In the Matter of First American Financial Corporation, Release No. 34-92176 (order)](https://www.sec.gov/litigation/admin/2021/34-92176.pdf). 14 June 2021. --- # Design a safer file-upload pipeline Treat upload receipt, content processing, storage, and later delivery as separate security decisions. Canonical URL: https://softwaresecurity.com/domains/appsec/secure-features/secure-file-uploads/ ## In short Treat an upload as untrusted bytes, not a file type. Cap the size while streaming, store it privately under a server-generated name, decode only the formats the feature needs, check decoded dimensions before full decoding, and publish a newly encoded copy. Bind approval to the content digest, authorize every retrieval, and serve user files from a separate origin with `Content-Disposition: attachment` and `nosniff`. Extensions and `Content-Type` headers are hints, not proof. - The extension, the `Content-Type` header and magic bytes are attacker-controlled hints; decoding with an allowlist of formats decides. - Enforce the byte cap while reading, and a separate decoded-pixel cap before full decoding. - Quarantine first: nothing is public until processing succeeds for the exact digest being published. - Store under a random server-side id and keep the original filename only as escaped metadata. - Serve user content from another origin with `attachment`, `nosniff` and a sandboxing CSP; never render uploaded SVG or HTML inline in the application origin. ## Recommended defaults | Setting | Value | Note | Source | | --- | --- | --- | --- | | Upload size | A per-feature cap enforced while streaming; 2,097,152 bytes in this example | Stop reading at the limit plus one byte. | OWASP ASVS v5.0.0 V5.2.1 | | Decoded image size | A width × height cap checked before full decoding; 12,000,000 pixels in this example | Pillow's `Image.MAX_IMAGE_PIXELS` default is 89,478,485: above it Pillow warns, above twice it raises `DecompressionBombError`. | OWASP ASVS v5.0.0 V5.2.6; Pillow | | Stored object name | A server-generated random id, such as `secrets.token_urlsafe(16)` | Never use the submitted filename as a path. | OWASP ASVS v5.0.0 V5.3.2 | | Download response headers | `Content-Disposition: attachment`, `X-Content-Type-Options: nosniff` and the verified `Content-Type` | — | OWASP ASVS v5.0.0 V5.4.1 and V3.4.4 | | Inline display of user files | A separate origin plus `Content-Security-Policy: sandbox` | SVG and HTML can carry script; rasterize or sanitize them first. | OWASP File Upload Cheat Sheet | | Archive uploads | Rejected unless the feature needs them; otherwise cap entries and uncompressed bytes before extraction | — | OWASP ASVS v5.0.0 V5.2.3 | ## Verify it 1. Upload exactly 2,097,152 bytes and then 2,097,153 bytes; the second must stop at the extra byte and create no object. 2. Upload 4000×3000 and 4001×3000 images; only the first may publish. 3. Upload HTML or SVG renamed to `.jpg` with `Content-Type: image/jpeg`; the decoder must reject it. 4. Replace a quarantined object after approval and confirm the new bytes are never published. 5. Fetch quarantined, rejected and pending objects through every URL, including thumbnails and signed links; every request must fail. ## What you will learn - Distinguish byte, format and decoded-resource checks. - Bind approval to immutable content. - Prove rejected or pending content remains private. Audience: Developers building upload features, and reviewers checking how uploaded files are stored, processed and served. Before you start: - Trace a request body from temporary storage to the eventual download response. Time: About 27 minutes: 7 to read, 3 on the code and 17 for the exercise ## Specify what receiving a file is allowed to cause An upload is a sequence of trust decisions, not a single extension check. The client controls the filename, claimed media type, bytes and transmission pace. A storage service can faithfully store those bytes while a later previewer, browser or document parser interprets them differently. The security boundary must cover receipt, inspection, processing, publication, retrieval and eventual deletion. Consider a profile-photo feature that needs only JPEG and PNG images, at most 2 MiB compressed, at most 12 megapixels decoded, and one published image per user. It does not need SVG, HTML, archives or the original filename as a filesystem path. Those decisions remove entire parser capabilities. A small compressed image may still require excessive decoded memory, so a byte cap and a pixel cap address different resources. An extension and a `Content-Type` header are useful consistency signals but are controlled by the sender and cannot establish the actual format. ## Make publication an explicit state transition Accept the stream into a private quarantine object with a server-generated id. Enforce the incoming byte budget while reading, before exhausting storage. Store owner, received length and a digest beside it. A worker with bounded CPU, memory and time decodes an allowed format, checks dimensions, strips unnecessary metadata and writes a newly encoded image. Publish the newly produced object only after all required checks succeed. A timeout or missing verdict leaves the upload unpublished. Framework-specific fixes are at /fix/file-upload/. Bind inspection to the exact object version or digest that will be published. Otherwise a client can upload safe bytes, obtain approval and replace them with different bytes before release. Separate receipt credentials from publication credentials. The public read path must check the object's approved state and any required ownership policy, including thumbnails and temporary URLs. A signed URL to quarantine still bypasses quarantine if anyone possessing it can fetch the bytes. Do not put originals in the public bucket and hope an asynchronous scanner removes them quickly enough. The Python program below implements receipt and processing for the photo contract with Pillow. `receive` reads at most one byte past the cap, so a 2,097,153-byte stream is rejected without being stored, and names the object with a random id. `process` opens the bytes with `formats=("PNG", "JPEG")`, rejects more than 12,000,000 pixels from the header before `img.load()` decodes the pixel data, and writes a new image that does not copy the original metadata. Install Pillow, save the file as `upload_pipeline.py` and run `python3 upload_pipeline.py`; it prints `PASS: format, pixel, byte and metadata checks`. The lab at /labs/path-traversal-python/ practices serving stored uploads without path traversal. - States: `RECEIVING` → `QUARANTINED` → `PROCESSING` → `PUBLISHED`, or `REJECTED` at any step. - A processing timeout returns the object to `QUARANTINED` for a bounded retry, or moves it to `REJECTED`. - A public `GET` serves only the generated output of a `PUBLISHED` object. - Approval of digest `d1` for `q17@v1` cannot publish `q17@v2` with digest `d2`. ```python # Requires Pillow (pip install Pillow). Run: python3 upload_pipeline.py import hashlib import io import secrets from PIL import Image MAX_BYTES = 2 * 1024 * 1024 # 2 MiB = 2,097,152 bytes MAX_PIXELS = 12_000_000 FORMATS = ("PNG", "JPEG") class Rejected(Exception): pass def receive(stream): """Read into private quarantine; never read more than MAX_BYTES + 1 bytes.""" data = stream.read(MAX_BYTES + 1) if len(data) > MAX_BYTES: raise Rejected("too_large") return {"id": secrets.token_urlsafe(16), # never the client's filename "sha256": hashlib.sha256(data).hexdigest(), "data": data} def process(item): """Decode an allowed format, check dimensions before decoding pixels, re-encode.""" try: with Image.open(io.BytesIO(item["data"]), formats=FORMATS) as img: if img.width * img.height > MAX_PIXELS: raise Rejected("too_many_pixels") img.load() out = io.BytesIO() img.save(out, format=img.format) # new file; metadata is not copied except (Image.UnidentifiedImageError, Image.DecompressionBombError, OSError): raise Rejected("not_an_allowed_image") from None return {"source": item["sha256"], "format": img.format, "data": out.getvalue()} def png(width, height, **info): from PIL import PngImagePlugin meta = PngImagePlugin.PngInfo() for key, value in info.items(): meta.add_text(key, value) out = io.BytesIO() Image.new("RGB", (width, height)).save(out, format="PNG", pnginfo=meta) return out.getvalue() def rejects(data, reason): try: process(receive(io.BytesIO(data))) except Rejected as e: assert str(e) == reason, e else: raise AssertionError(f"accepted, expected {reason}") ok = process(receive(io.BytesIO(png(800, 600, Comment="gps=51.5,-0.1")))) assert ok["format"] == "PNG" and b"gps=" not in ok["data"] assert process(receive(io.BytesIO(png(4000, 3000))))["format"] == "PNG" # exactly 12,000,000 rejects(png(4001, 3000), "too_many_pixels") rejects(b"", "not_an_allowed_image") gif = io.BytesIO() Image.new("RGB", (8, 8)).save(gif, format="GIF") rejects(gif.getvalue(), "not_an_allowed_image") assert len(receive(io.BytesIO(b"\0" * MAX_BYTES))["data"]) == 2_097_152 try: receive(io.BytesIO(b"\0" * (MAX_BYTES + 1))) raise AssertionError("accepted 2,097,153 bytes") except Rejected as e: assert str(e) == "too_large" print("PASS: format, pixel, byte and metadata checks") ``` ## Verify the entire lifecycle with explicit budgets In this example a 2 MiB limit means 2,097,152 bytes and the image dimension rule means width × height must not exceed 12,000,000 pixels. Test exactly those boundaries rather than only a convenient large file. The valid case must produce a usable image; the rejected cases must create no public object. A response saying upload accepted may mean only that private receipt completed, so distinguish accepted-for-processing from published. Instrument the decoder process and storage transitions in a staging system using synthetic files. Confirm cancellation kills abandoned processing, temporary objects expire and repeated failures cannot grow an unbounded retry queue. For the paper exercise below, supplied decoder results stand in for file bytes so you can evaluate the state machine without manufacturing image files. They establish the design decision only; a deployed decoder also requires package maintenance and integration checks against actual formats. ## Understand what re-encoding and scanning cannot prove Re-encoding can remove unsupported active content and metadata, but the decoder itself processes attacker-controlled input. Isolating the worker limits the consequences of a parser flaw. Antivirus may identify known malicious content; a clean verdict is not proof of harmlessness. Keep the narrow allowlist and publication gate even when scanning is enabled. Download disposition, a separate content origin and `nosniff` can reduce browser interpretation risks, but cannot authorize who may read a private file. Serve uploaded content so the browser cannot treat it as part of your application. Use a separate origin for user files, send `Content-Disposition: attachment` for downloads, `X-Content-Type-Options: nosniff` with the verified `Content-Type`, and `Content-Security-Policy: sandbox` for anything displayed inline. SVG is XML that can carry script, so rasterize it, sanitize it with an SVG-aware allowlist or serve it only as an attachment. A polyglot file is valid as two formats at once and passes magic-byte checks, which is why this pipeline decodes and re-encodes instead of trusting signatures; content disarm and reconstruction applies the same idea to office documents and PDFs, and /domains/appsec/secure-features/untrusted-document-conversion/ shows how to convert those in an isolated parser. If the feature accepts archives, apply /domains/appsec/secure-features/archive-path-safety/; the archive-extraction project linked below tests those rules. Operational failures often appear as stuck uploads or mismatched versions. Expose a stable processing status to the owner, record the failed stage and retry count, and keep sensitive original bytes out of logs. A quarantine-retention policy needs enough time for investigation without keeping rejected personal documents indefinitely. Backpressure should reject new work predictably when capacity is exhausted; silently skipping inspection to reduce queue length converts an availability incident into a security failure. ## A real-world failure CVE-2021-21972 was an unauthenticated file upload in the VMware vCenter Server vSphere Client plugin that let an attacker with network access to port 443 write a file and execute commands with unrestricted privileges on the host operating system. The missing controls were validating the upload and storing it where the web server will never execute it. ## Keep storage ownership and job ownership aligned The pipeline can be correctly sandboxed yet still publish the wrong person's object. Suppose Alice receives upload id U1, then submits `finishUpload(U2)` where U2 belongs to Bob. A finish endpoint that merely finds a clean verdict for U2 may attach Bob's image to Alice's profile. Require the authenticated actor to own the upload session, verify the immutable content version, and bind the resulting published object to the intended feature. A scanner's clean verdict answers content-policy questions; it does not answer who may publish or retrieve the object. Direct-to-storage uploads require the same discipline. Limit the issued upload capability to its object key, method, size constraints where supported and lifetime; an Amazon S3 presigned POST can carry the size limit in its policy as `["content-length-range", 1, 2097152]`. The completion endpoint should verify actual stored properties and ownership rather than trusting client-reported length or digest. If a client can replace the same key after scanning, publication should use an immutable version or copied generated output. Changing the object key to a random string helps prevent collisions but does not establish ownership or immutability. Consider concurrency too. If Alice uploads U1 and U2 and the older worker finishes last, an unconditional profile update can replace her newer choice. Bind publication to the current selected upload or compare a profile revision before committing. The old worker should mark its own processing complete without overriding a newer user decision. This is a correctness issue with privacy consequences when objects have different visibility or retention expectations. ## Guided practice Apply a complete photo-upload contract to supplied receipt and decoder evidence. ### Supplied material ```text Policy: JPEG/PNG only; bytes <=2097152; pixels <=12000000; publish only newly encoded output of approved object version. A: a.png, header image/png, 1000 bytes, decoder PNG 800x600, success, q1@v1=d1. B: b.jpg, header image/jpeg, 900 bytes, decoder HTML, q2@v1=d2. C: c.png, 2097152 bytes, decoder PNG 4000x3000, success. D: d.png, 2097153 bytes; stream still arriving. E: e.png, 1000 bytes, decoder PNG 4001x3000, success. F: f.png, 1000 bytes, decoder timeout. G: q1@v1=d1 approved, but current source q1@v2=d9. ``` ### 1. Classify ordinary and disguised content Decide A and B using decoder identity rather than submitted names. Worked answer: A may proceed to re-encoding and publication. B is rejected because HTML is outside the feature contract despite its JPEG extension and header. ### 2. Calculate boundaries Calculate the pixel counts for C and E; decide C, D and E. Worked answer: C is exactly 12,000,000 pixels and exactly the byte limit, so it may publish after processing. D stops at byte 2,097,153. E is 12,003,000 pixels and is rejected. None of these decisions needs an arbitrary safety margin. ### 3. Resolve incomplete and stale verdicts Decide the public state of F and G. Worked answer: F remains private; retry is bounded or it becomes rejected. G cannot use `d1` approval for `d9`. It must process the selected immutable version, then publish only its generated output. ### 4. Audit retrieval A user requests q2, a thumbnail of F and a signed link to G before approval. State the result for each. Worked answer: All three are unavailable through the public retrieval path. Hiding the main UI or returning a processing message does not suffice if an alternate URL returns the raw object. Deliverable: A seven-case decision table with byte/pixel calculations and a publication-state invariant. Scope: Paper exercise: decoder results stand in for real image files, so no files or services are needed. The Python program above tests the same byte, pixel and format rules on generated images. ## Executable practice - [Path traversal lab: Serve an uploaded file without path traversal](https://softwaresecurity.com/labs/path-traversal-python/) - [File upload lab: Accept avatars without publishing pages](https://softwaresecurity.com/labs/file-upload/) - [Keep extracted files inside their boundary: project ZIP](https://softwaresecurity.com/downloads/projects/archive-extraction-e31e8b17b78e9134.zip) — Python 3.11+ · standard library - [Convert untrusted documents in a locked-down container: project ZIP](https://softwaresecurity.com/downloads/projects/document-conversion-sandbox-62d2ff3fa1f9ec09.zip) — Python 3.11+ offline · Docker for the live tests Articles and teaching material: CC BY 4.0. Original executable code: MIT. Attribution: Software Security. ## Sources - [OWASP File Upload](https://cheatsheetseries.owasp.org/cheatsheets/File_Upload_Cheat_Sheet.html) - [OWASP Input Validation](https://cheatsheetseries.owasp.org/cheatsheets/Input_Validation_Cheat_Sheet.html) - [OWASP ASVS 5.0.0: V5 File Handling](https://github.com/OWASP/ASVS/blob/v5.0.0_release/5.0/en/0x14-V5-File-Handling.md). Version 5.0.0. - [Pillow: Image.open and decompression bomb protection](https://pillow.readthedocs.io/en/stable/reference/Image.html). Pillow 12.3.0. - [Amazon S3: POST policy (content-length-range)](https://docs.aws.amazon.com/AmazonS3/latest/developerguide/sigv4-HTTPPOSTConstructPolicy.html) - [MITRE CWE-434: unrestricted upload of file with dangerous type](https://cwe.mitre.org/data/definitions/434.html). CWE 4.20. - [NVD: CVE-2021-21972, file upload to RCE in VMware vCenter](https://nvd.nist.gov/vuln/detail/cve-2021-21972) --- # Verify webhook deliveries before acting Verify the provider’s signature over the correct bytes, then apply replay, authorization, and idempotency controls before a business action. Canonical URL: https://softwaresecurity.com/domains/appsec/secure-features/verify-webhook-deliveries/ ## In short Verify the provider's signature over the exact raw request bytes before parsing, using its documented scheme and a constant-time comparison. Reject stale timestamps where the scheme signs one; Stripe's libraries allow 5 minutes. Then check that the event belongs to the expected account and order state, and record it under a unique event key in the same transaction as the state change and an outbox entry, so retries and duplicates cause one business effect. - Verify the raw bytes before any body parser runs; re-serialized JSON is a different message. - Use the provider scheme exactly: GitHub signs the body in `X-Hub-Signature-256`, Stripe signs `.` in `Stripe-Signature`. - Compare with `hmac.compare_digest`, `crypto.timingSafeEqual` or `MessageDigest.isEqual`, never `==`. - A valid signature proves origin, not freshness, uniqueness or business validity; check each separately. - Make the inbox key unique in the database and commit it with the state change and outbox intent, then acknowledge. ## Recommended defaults | Setting | Value | Note | Source | | --- | --- | --- | --- | | Signature algorithm | HMAC-SHA256 (GitHub, Stripe, Standard Webhooks `v1`); Ed25519 for Standard Webhooks `v1a` | — | Provider documentation; Standard Webhooks specification | | Timestamp tolerance | 300 seconds | Stripe's library default. A tolerance of 0 disables Stripe's recency check. | Stripe: verify webhook signatures | | Secret rotation overlap | Up to 24 hours, with one signature per active secret | — | Stripe: roll endpoint signing secrets | | Deduplication retention | At least the provider retry window: Stripe retries live events for up to 3 days | — | Stripe: automatic retries | | Response time | Acknowledge after the durable commit and within the provider timeout: 10 seconds for GitHub | — | GitHub: best practices for using webhooks | | Shared secret | Random, high entropy, per endpoint and environment; Standard Webhooks uses 24 to 64 bytes | — | GitHub; Standard Webhooks specification | ## Verify it 1. Run the documented provider test vector, such as the GitHub values in the code below, against your verification function. 2. Change one byte of a signed body and confirm the request fails before parsing, with no stored rows. 3. Replay a delivery older than the tolerance and confirm it is rejected; repeat with exactly the boundary age. 4. Send the same event twice, including concurrently, and assert one inbox row, one state change and one outbox entry. 5. Rotate the secret with an overlap window and confirm both secrets verify only until the old one is retired. ## What you will learn - Explain why parsed JSON is not the signed byte sequence. - Separate signature, freshness, deduplication and business validation. - Design acknowledgement and retry behavior around durable state. Audience: Backend developers receiving signed webhooks from payment, source-control or SaaS providers, and reviewers of those handlers. Before you start: - Read raw HTTP body bytes, headers, and a state-changing handler. Time: About 32 minutes: 7 to read, 5 on the code and 20 for the exercise ## Verify the exact message the provider signed A webhook endpoint receives an ordinary HTTP request from the network. A field named `event` or `paid` is only a claim until the receiver authenticates the delivery according to its provider's protocol. In an HMAC scheme, the provider and receiver share a secret and compute a keyed digest over specified bytes. The receiver compares the supplied signature with its own computation using the protocol's constant-time comparison facility. TLS protects transport; it does not make every caller an authorized provider. GitHub signs the raw request body with HMAC-SHA-256 in `X-Hub-Signature-256`. Stripe's signing scheme also binds a timestamp and its SDK enforces a timestamp tolerance. These are different protocols: do not invent a universal header parser or copy freshness behavior from one provider to another. JSON parsing and serialization can change whitespace, ordering or Unicode representation while preserving the same object. Verify the original bytes before body middleware transforms them; then parse and validate the authenticated content. Reject unsupported signature formats, missing secrets and malformed signatures before business work. ## Separate authenticated receipt from a business effect A valid signature establishes possession of the delivery secret for that message, not that the event is new, belongs to the intended account or is appropriate for the current order state. For this example, a signed paid event names order O17, provider account `shop-red` and amount 2500 USD cents. The receiver loads O17, requires account `shop-red`, currency USD, amount 2500 and an allowed unpaid-to-paid transition. Trusting the event's tenant field to choose an unrestricted database context would create a new authorization failure. Durably record a verified event and enqueue its work before returning the provider's successful acknowledgement. Deduplicate with a unique key scoped to provider/account/event identity, and atomically couple that record to the local state change or durable work intent. Two concurrent deliveries must not both pass an in-memory seen check; this is the check-then-act race in /domains/appsec/business-logic/transaction-races/. For an external effect, use the recipient's idempotency mechanism and retain the operation identity across retries. Arrival order is not necessarily business order; fetch authoritative current provider state when ordering matters. The program below implements the training protocol from the exercise with Python and SQLite. It verifies `HMAC-SHA256(secret, "." + raw body)` with `hmac.compare_digest` before parsing, rejects ages above 300 seconds, and then inserts the inbox row, marks the order paid and writes `fulfill:O17` to the outbox in one transaction. A repeated event id hits the unique constraint and is acknowledged without a second effect; a new event id for an already paid order commits its inbox row but no outbox entry. Run `python3 webhook_receiver.py`; it prints `PASS: 7 delivery cases, one fulfillment`. ```python # Run: python3 webhook_receiver.py import hashlib import hmac import json import sqlite3 TOLERANCE = 300 # seconds, inclusive class Rejected(Exception): pass def sign(secret, timestamp, raw): """The training protocol: HMAC-SHA256 over b"." + raw body.""" return hmac.new(secret, f"{timestamp}.".encode() + raw, hashlib.sha256).hexdigest() def verify(secret, raw, timestamp, signature, now): if not timestamp.isdigit() or len(signature) != 64: return "malformed" if not hmac.compare_digest(sign(secret, timestamp, raw), signature): return "bad_signature" if abs(now - int(timestamp)) > TOLERANCE: return "stale" return "ok" def handle(db, secret, raw, headers, now): result = verify(secret, raw, headers.get("t", ""), headers.get("sig", ""), now) if result != "ok": return 401, result # nothing parsed, nothing stored event = json.loads(raw) # parse only the authenticated bytes try: with db: # inbox row, state change and outbox intent commit together db.execute("INSERT INTO inbox VALUES (?, ?, ?)", ("training", event["account"], event["id"])) account, amount, currency, state = db.execute( "SELECT account, amount, currency, state FROM orders WHERE id = ?", (event["order"],)).fetchone() if (account, amount, currency) != (event["account"], event["amount"], event["currency"]): raise Rejected("context_mismatch") # rolls back the inbox row if state == "paid": return 200, "already_paid" # new event id, same effect: no outbox db.execute("UPDATE orders SET state = 'paid' WHERE id = ?", (event["order"],)) db.execute("INSERT INTO outbox VALUES (?, ?)", (f"fulfill:{event['order']}", event["order"])) return 200, "accepted" except sqlite3.IntegrityError: return 200, "duplicate" # already committed: acknowledge, no new effect except Rejected as e: return 400, str(e) def fresh_db(): db = sqlite3.connect(":memory:") db.executescript(""" CREATE TABLE inbox (provider TEXT, account TEXT, event_id TEXT, UNIQUE (provider, account, event_id)); CREATE TABLE orders (id TEXT PRIMARY KEY, account TEXT, amount INTEGER, currency TEXT, state TEXT); CREATE TABLE outbox (operation TEXT UNIQUE, order_id TEXT); INSERT INTO orders VALUES ('O17', 'shop-red', 2500, 'USD', 'unpaid'); """) return db SECRET = b"training-endpoint-secret" def delivery(t, event_id, account="shop-red", amount=2500, tamper=False): raw = json.dumps({"id": event_id, "account": account, "order": "O17", "amount": amount, "currency": "USD"}).encode() sig = sign(SECRET, str(t), raw) if tamper: raw = raw.replace(b", ", b",") # same JSON object, different bytes return raw, {"t": str(t), "sig": sig} db = fresh_db() assert handle(db, SECRET, *delivery(900, "e1"), now=1000) == (200, "accepted") assert handle(db, SECRET, *delivery(900, "e1"), now=1000) == (200, "duplicate") assert handle(db, SECRET, *delivery(950, "e5"), now=1000) == (200, "already_paid") assert db.execute("SELECT COUNT(*) FROM outbox").fetchone() == (1,) for case, expected in [ (delivery(900, "e2", tamper=True), (401, "bad_signature")), (delivery(699, "e2"), (401, "stale")), # age 301 (delivery(700, "e3", account="shop-blue"), (400, "context_mismatch")), # age 300 (delivery(900, "e4", amount=25), (400, "context_mismatch")), ]: db = fresh_db() assert handle(db, SECRET, *case, now=1000) == expected, (case, expected) assert db.execute("SELECT COUNT(*) FROM inbox").fetchone() == (0,) assert db.execute("SELECT state FROM orders").fetchone() == ("unpaid",) print("PASS: 7 delivery cases, one fulfillment") ``` ## Check the provider's exact signature scheme Use the provider's library or its documented algorithm exactly; the schemes differ in the bytes they sign and in whether a timestamp is included. The function below verifies GitHub's header and is tested with the secret, payload and signature GitHub publishes for this purpose, so you can prove the function before wiring it to a route. Compare with a constant-time function on equal-length values: `hmac.compare_digest` in Python, `crypto.timingSafeEqual` in Node.js, which throws when the lengths differ, so check the length first, and `MessageDigest.isEqual` in Java. Stripe also publishes the addresses it sends from and recommends allowlisting them in addition to verifying signatures; an address allowlist alone does not authenticate a message. ```python # Run: python3 github_signature.py import hashlib import hmac def verify_github(secret: str, raw_body: bytes, header: str | None) -> bool: """Checks X-Hub-Signature-256 against the raw request body.""" if not header or not header.startswith("sha256="): return False expected = "sha256=" + hmac.new(secret.encode(), raw_body, hashlib.sha256).hexdigest() return hmac.compare_digest(expected, header) # Test values published in GitHub's "Validating webhook deliveries" guide. secret = "It's a Secret to Everybody" header = "sha256=757107ea0eb2509fc211221cce984b8a37570b6d7586c22c46f4379c8b043e17" assert verify_github(secret, b"Hello, World!", header) assert not verify_github(secret, b"Hello, World! ", header) # one extra byte assert not verify_github("wrong secret", b"Hello, World!", header) assert not verify_github(secret, b"Hello, World!", None) print("PASS: GitHub test vector verified") ``` | Scheme | Signature header | Signed bytes | Freshness and duplicates | | --- | --- | --- | --- | | GitHub | `X-Hub-Signature-256: sha256=` | The raw body | No timestamp is signed; deduplicate on `X-GitHub-Delivery`, which a redelivery repeats. | | Stripe | `Stripe-Signature: t=