Call a hosted LLM API safely

The most common LLM-integration mistake is calling the provider straight from the client. Anything you ship to a browser, mobile app or desktop binary can be extracted, so an API key embedded there is a leaked key, and build-time environment prefixes such as NEXT_PUBLIC_ and VITE_ copy their values into the browser bundle. Put the LLM call behind your own backend endpoint: the client calls you, you authenticate the user and apply quotas, and only your server holds the provider key. The narrow exception is a provider’s own short-lived client token, such as an OpenAI Realtime client secret minted server-side with POST /v1/realtime/client_secrets, which is scoped and expires; use that mechanism rather than shipping the real key.

Read the key from a secrets manager or an injected environment variable, never from source control. Use a distinct, least-privilege key per service: OpenAI project API keys with restricted permissions, or an Anthropic Console workspace key. Rotate on a schedule, and keep a revoke-first runbook for a leak (see /domains/identity-crypto/secrets/leaked-secret-response/). The SDKs read the key from the environment by default, so you rarely pass it in code.

import os

from openai import OpenAI
from anthropic import Anthropic

# Keys come from the environment, injected from a secrets manager at deploy time.
# They are never hard-coded and never sent to a client.
openai = OpenAI(
    api_key=os.environ["OPENAI_API_KEY"],
    timeout=20.0,       # seconds; SDK default is 10 minutes
    max_retries=2,      # bounded; SDK default is 2
)
anthropic = Anthropic(
    api_key=os.environ["ANTHROPIC_API_KEY"],
    timeout=20.0,
    max_retries=2,
)

# Read the model from config, so it is not hard-coded and can be changed per environment.
MODEL = os.environ["LLM_MODEL"]

Send the model the least it needs. Strip or pseudonymize personal data before it reaches the prompt, so a provider log, a future training-data question or an incident exposes as little as possible; this is the data-minimization half of OWASP LLM02:2026 Sensitive Information Disclosure. Choose reversible pseudonymization, keeping a server-side map so you can re-expand identifiers in the response, when the model needs a stable token, and irreversible redaction otherwise. In Python, Presidio, which Microsoft started and a community project now maintains, detects and anonymizes many entity types; for a few structured identifiers a pattern-based redactor is enough, with the honest caveat that pattern matching misses names, addresses and free-text PII.

from presidio_analyzer import AnalyzerEngine
from presidio_anonymizer import AnonymizerEngine

analyzer = AnalyzerEngine()      # needs a spaCy model, e.g. en_core_web_lg
anonymizer = AnonymizerEngine()


def redact(text: str) -> str:
    results = analyzer.analyze(
        text=text,
        language="en",
        entities=["EMAIL_ADDRESS", "PHONE_NUMBER", "CREDIT_CARD", "US_SSN", "PERSON"],
    )
    return anonymizer.anonymize(text=text, analyzer_results=results).text


# Redact before building the prompt, and redact (or omit) before logging.
safe_prompt = redact(user_record)

On Node there is no Presidio; for structured identifiers a vetted set of patterns, applied before the prompt and before logging, covers the common cases. Tested in Node, this replaced an email, a US national id and a card number with labels. Treat it as a floor, not a complete PII engine: it will not catch names or addresses, so still minimize what you send.

const PATTERNS: Array<[string, RegExp]> = [
  ["EMAIL", /\b[\w.+-]+@[\w-]+\.[\w.-]+\b/g],
  ["CARD", /\b(?:\d[ -]?){13,16}\b/g],
  ["US_SSN", /\b\d{3}-\d{2}-\d{4}\b/g],
];

export function redact(text: string): string {
  return PATTERNS.reduce((out, [label, re]) => out.replace(re, `[${label}]`), text);
}

Know, and record, how long your provider keeps the data you send. As of October 2026, OpenAI keeps abuse-monitoring logs for up to 30 days by default and, for approved customers, offers Zero Data Retention, which treats store as false on /v1/responses and /v1/chat/completions and keeps customer content out of abuse-monitoring logs; endpoints that keep application state, such as Files, Batches, Vector stores, Conversations, Assistants and fine-tuning jobs, are not ZDR-eligible. Anthropic deletes API inputs and outputs within 30 days of receipt or generation by default, except for features whose retention you control (such as the Files API), a zero data retention agreement, Usage Policy enforcement and legal requirements; flagged content is kept up to 2 years and trust-and-safety classifier scores up to 7 years, and since 9 June 2026 the models Anthropic designates as Covered Models require 30-day retention, so they are generally unavailable under ZDR. These values change, so verify them against the provider’s data-control pages for your account and record the date you checked. OpenAI’s Responses API keeps application state for 30 days by default; set store: false when you don’t need it, understanding that this does not remove abuse-monitoring logs unless you are on ZDR.

Structured outputs make the response parse reliably into your schema, which removes a class of brittle string handling. They do not make the content true or safe: a schema-valid object can still carry a prompt-injection payload or a wrong answer, so re-validate on your server and treat the values as untrusted input (see /domains/ai-emerging/llm-security/untrusted-model-output/). With the OpenAI Python SDK, responses.parse takes a Pydantic model as text_format and returns output_parsed; in Node, pass zodTextFormat from openai/helpers/zod. Handle a refusal or a truncated response (hitting the output-token cap) as a failure, not as data.

from pydantic import BaseModel, Field


class Draft(BaseModel):
    title: str = Field(min_length=1, max_length=200)
    summary: str = Field(min_length=1, max_length=2000)


def summarize(text: str, user_id: str) -> Draft:
    resp = openai.responses.parse(
        model=MODEL,
        input=redact(text),                 # PII removed before the prompt
        text_format=Draft,                  # schema-constrained output
        max_output_tokens=800,              # cost and truncation cap
        safety_identifier=hash_user(user_id),  # a hash, never an email
    )
    draft = resp.output_parsed              # already validated against Draft
    log_call(resp)                          # request id and usage, not content
    return draft                            # still treat fields as untrusted downstream

Even when the provider constrains output to a schema, re-validate with your own schema on the server so a provider or SDK change can't slip an unexpected shape through. With zod's z.strictObject, an extra field or a wrong type fails; the tested snippet rejected an object with an empty title and an extra field. Use the parsed, re-validated value, and still encode it for whatever sink it reaches.

import { z } from "zod";

const Draft = z.strictObject({
  title: z.string().min(1).max(200),
  summary: z.string().min(1).max(2000),
});

// Re-validate the model's output on the server, treating it as untrusted.
export function parseDraft(output: unknown) {
  const result = Draft.safeParse(output);
  if (!result.success) {
    throw new Error(`model output failed schema: ${result.error.issues[0].message}`);
  }
  return result.data;
}

An LLM call is a network call to a slow dependency, so bound it. The OpenAI and Anthropic SDKs default to a 10-minute timeout and 2 retries; set a shorter timeout for interactive requests and keep retries bounded with backoff (the SDKs retry connection errors, 408, 409, 429 and 5xx). The Anthropic SDKs also raise an error for a non-streaming request expected to take longer than about 10 minutes unless you stream or set a timeout. Cap the output with max_output_tokens (OpenAI) or max_tokens (Anthropic) so a runaway generation can’t run up the bill. For per-user cost control, reserve budget before the call and reconcile from the usage the response reports. Set provider spend limits as a backstop: an OpenAI organization or project hard limit returns 429 with project_spend_limit_exceeded or organization_spend_limit_exceeded, and Anthropic workspaces take a monthly spend limit. The cost and memory guide at /domains/ai-emerging/agent-security/cost-memory-operations/ covers budget reservation in depth.

# Reserve budget before the call, then reconcile from reported usage.
def call_with_budget(user_id: str, text: str) -> Draft:
    reservation = budget.reserve(user_id, estimated_tokens=1200)  # raises if over cap
    try:
        resp = openai.with_options(timeout=15.0).responses.parse(
            model=MODEL,
            input=redact(text),
            text_format=Draft,
            max_output_tokens=800,
            safety_identifier=hash_user(user_id),
        )
    finally:
        reservation.release()
    budget.charge(user_id, tokens=resp.usage.total_tokens)  # actual spend
    log_call(resp)
    return resp.output_parsed

Log what you need to diagnose a problem and nothing that is sensitive. Record the provider request id (both SDKs expose it as response._request_id, read from the x-request-id header for OpenAI and request-id for Anthropic), the token usage, the model and the latency; a hash of the prompt is fine for correlating. Never log the API key, the raw prompt or a response that may contain PII. Turn off the SDKs’ debug logging in production (OPENAI_LOG and ANTHROPIC_LOG): at debug level the Anthropic TypeScript SDK logs request and response bodies. Make sure your HTTP client isn’t dumping request bodies either. Keep your own logs on a short retention and access them under the same controls as other sensitive data (see /domains/appsec/secure-coding/secure-error-handling/ and /domains/runtime/detection/security-event-logging/).

import logging

log = logging.getLogger("llm")


def log_call(resp) -> None:
    # Request id and usage only: no key, no prompt, no response content.
    log.info(
        "llm_call",
        extra={
            "request_id": getattr(resp, "_request_id", None),
            "model": MODEL,
            "input_tokens": resp.usage.input_tokens,
            "output_tokens": resp.usage.output_tokens,
        },
    )

A minimal FastAPI endpoint shows the whole shape: authenticate the user, enforce a quota, redact the input, call the provider with a bounded timeout and a schema, and return only the validated result. The provider key never leaves the server, and the client never talks to the provider directly.

from fastapi import Depends, FastAPI, HTTPException

app = FastAPI()


@app.post("/summarize")
def summarize_endpoint(body: SummarizeIn, user=Depends(current_user)):
    if user is None:
        raise HTTPException(status_code=401)        # no anonymous provider calls
    if not quota.allow(user.id):
        raise HTTPException(status_code=429)        # per-user cost cap
    try:
        draft = call_with_budget(user.id, body.text)  # redact + timeout + schema
    except TimeoutError:
        raise HTTPException(status_code=504, detail="model timed out")
    return {"title": draft.title, "summary": draft.summary}

The controls are provider-independent. With the Anthropic SDK, the client reads ANTHROPIC_API_KEY from the environment and takes the same timeout and max_retries options; set max_tokens to cap output, pass a hashed end-user id as metadata.user_id for abuse monitoring, and read token counts from response.usage (input_tokens and output_tokens) to reconcile budget. For schema-constrained output, call client.messages.parse() with output_format set to a Pydantic model (the SDK sends it as output_config.format) and read parsed_output, or use a strict tool definition, then re-validate the result on your server exactly as above. Read the model id from configuration rather than hard-coding it.

resp = anthropic.messages.create(
    model=MODEL,                       # from config, e.g. a current Claude model id
    max_tokens=800,                    # output cap
    metadata={"user_id": hash_user(user_id)},  # hashed id, not an email
    messages=[{"role": "user", "content": redact(text)}],
)
# resp.usage.input_tokens / resp.usage.output_tokens for budget reconciliation;
# resp._request_id for logging. Validate and treat resp.content as untrusted.

Sources & further reading

The standards, specifications and documentation behind the guidance above. Each is the evidence for a claim on this page; consult it for detail beyond the defaults and versions listed here.

1. OpenAI: how we use and retain your data (data controls, Zero Data Retention)
2. OpenAI: safety best practices (safety identifiers)
3. OpenAI: structured outputs (responses.parse, zodTextFormat)
4. OpenAI: spend limits
5. OpenAI Python SDK README (timeout, max_retries, request id)
6. OpenAI Node SDK README
7. Anthropic: how long do you store my organization’s data? (API retention)
8. Anthropic: data retention practices for Covered Models
9. Anthropic: API and data retention (zero data retention scope)
10. Anthropic Python SDK (timeouts, retries, request ids, structured outputs)
11. OWASP Top 10 for LLM Applications 2026: LLM02 Sensitive Information Disclosure. 2026 edition (3 August 2026).
12. Presidio: data protection and de-identification SDK
13. OWASP ASVS 5.0.0, V13 Configuration. Version 5.0.0.
14. CWE-798: Use of Hard-coded Credentials. CWE 4.20.
15. CWE-532: Insertion of Sensitive Information into Log File. CWE 4.20.

Reference information

Tags

Denial of service and resource limitsSecrets managementPrivacy engineeringLLM application security

SDLC phase

DesignDevelopment

Role

DevelopersAppSec engineers

ASVS 5.0.0 requirements

v5.0.0-13.1.1 · v5.0.0-1.3.6

Download this guide as Markdown

Continue exploring