Secure AI-assisted development with coding assistants and agents

A completion assistant suggests code that a developer accepts or rejects. A coding agent does more: it reads the repository, issues and web pages, runs shell commands, installs packages, calls MCP tools and pushes branches, usually with the developer's own credentials. Each of those steps crosses a trust boundary that ordinary code review never sees.

Four channels matter. What the assistant reads can carry instructions. What it writes can look correct and still be exploitable. What it runs executes with whatever the session can reach. What it sends, including file contents and command output, goes to a model provider and usually into local transcripts.

Simon Willison's lethal trifecta and Meta's Agents Rule of Two name the dangerous combination: untrusted input, access to private data or credentials, and the ability to change state or communicate externally. A session with all three is one crafted issue away from leaking or damaging something. Design each session so at least one is missing, and enforce that outside the model.

Pearce et al. judged about 40% of 1,689 Copilot-generated programs vulnerable (IEEE S&P 2022). Perry et al. found that people using an assistant wrote less secure code and were more confident that it was secure (ACM CCS 2023). BaxBench (2025) found that the best model it tested produced correct backends 62% of the time, and that about half of each model's correct programs could be exploited. Models have improved, but the mechanism is unchanged: the code satisfies the prompt and the happy-path test.

Apply the rules you use for an outside pull request, and read the whole diff. Agents often change files the request never mentioned, and an agent-written PR description is generated text too: judge the diff, not the summary.

• Authorization: every new route or query checks the caller against the specific object, not only that someone is logged in.
• Interpreters: SQL, shell commands, HTML and file paths are built with safe APIs, never by string concatenation.
• Weakened controls: verify=False, wildcard CORS, disabled CSRF checks, broad IAM grants or 'temporary' bypass flags added to make an error go away.
• Configuration: no literal credentials, no new debug endpoints, and no edits to agent instruction files, .vscode settings or MCP configs hidden in a feature change.
• Dependencies: every new package and lockfile change is intended, exists and passes the gate below.
• Tests: deleted or weakened assertions, mocks that replace the unit under test, and tests that only restate what the generated code already does.

Asked for a function that returns a report by file name, an assistant can produce this. It returns q3.txt correctly, so a quick test passes.

It fails three ways. A name such as ../secret.txt walks out of the directory. An absolute name such as /etc/passwd makes os.path.join discard REPORT_DIR entirely. A symlink inside the directory can point anywhere. In our test, all three returned the file outside the report directory.

import os

REPORT_DIR = "/srv/reports"


def read_report(name: str) -> bytes:
    """Return a generated report by file name."""
    with open(os.path.join(REPORT_DIR, name), "rb") as f:
        return f.read()

Resolve the joined path, which collapses .. segments and follows symlinks, and then require that the result is still inside the report directory. The three attack names now raise PermissionError, and q3.txt still works; /fix/path-traversal/ has the same containment check for other stacks.

This fixes containment only. The endpoint still needs an authorization check on which reports this caller may read. If attackers can write inside the directory, a symlink swapped between the check and the read is a race that needs file-descriptor-based APIs. Keep the attack names as regression tests so the next generated refactor cannot quietly undo the fix.

from pathlib import Path

REPORT_DIR = Path("/srv/reports").resolve()


def read_report(name: str) -> bytes:
    """Return a generated report, refusing anything that resolves outside REPORT_DIR."""
    path = (REPORT_DIR / name).resolve()  # follows symlinks and collapses ".."
    if not path.is_relative_to(REPORT_DIR):
        raise PermissionError(f"report name escapes the report directory: {name!r}")
    return path.read_bytes()

Code models invent package names. Spracklen et al. (USENIX Security 2025) generated 576,000 code samples with 16 models and found that 19.7% of 2.23 million recommended packages did not exist, covering 205,474 unique invented names. Rates averaged at least 5.2% for commercial models and 21.7% for open-source ones. The inventions repeat: 43% reappeared in all 10 re-runs of the same prompt, so an attacker can predict them and register the name with a malicious install script. This is slopsquatting, which the OWASP Top 10 for LLM Applications 2026 lists under LLM04 Supply Chain.

Run a gate whenever a change adds a dependency. The script queries PyPI's Index API and the npm registry, blocks names that do not exist, and blocks projects first published in the last 90 days until someone reviews them. Feed it the names your manifest diff adds. For production builds, pair it with an allowlisting registry proxy, and run agent-suggested installs inside the sandbox described below with lifecycle scripts off where the project allows it (npm ci --ignore-scripts).

Existence and age are necessary, not sufficient. Check that the publisher, linked source repository and release history match the project you meant; a squatter who registered a name months ago passes an age check.

#!/usr/bin/env python3
"""Gate AI-suggested dependencies: the name must exist and must not be brand new.

Usage: check_new_deps.py pypi:requests npm:express npm:@types/node
"""
import json
import sys
import urllib.error
import urllib.parse
import urllib.request
from datetime import datetime, timedelta, timezone

MIN_PROJECT_AGE = timedelta(days=90)


def first_published(ecosystem: str, name: str) -> datetime | None:
    """Return when the project first appeared in its registry, or None if it does not exist."""
    if ecosystem == "pypi":  # PyPI Index API (JSON form of the Simple API)
        url = f"https://pypi.org/simple/{urllib.parse.quote(name)}/"
        headers = {"Accept": "application/vnd.pypi.simple.v1+json"}
    elif ecosystem == "npm":  # scoped names need the slash encoded: @scope%2Fname
        url = "https://registry.npmjs.org/" + urllib.parse.quote(name, safe="@")
        headers = {"Accept": "application/json"}
    else:
        raise ValueError(f"unsupported ecosystem: {ecosystem}")
    try:
        with urllib.request.urlopen(urllib.request.Request(url, headers=headers), timeout=15) as resp:
            meta = json.load(resp)
    except urllib.error.HTTPError as err:
        if err.code == 404:
            return None
        raise
    if ecosystem == "pypi":
        stamps = [f["upload-time"] for f in meta.get("files", []) if f.get("upload-time")]
    else:
        stamps = [meta.get("time", {}).get("created")] if meta.get("time", {}).get("created") else []
    return min(datetime.fromisoformat(s.replace("Z", "+00:00")) for s in stamps) if stamps else None


def main(specs: list[str]) -> int:
    failed = False
    now = datetime.now(timezone.utc)
    for spec in specs:
        ecosystem, _, name = spec.partition(":")
        first = first_published(ecosystem, name)
        if first is None:
            print(f"BLOCK {spec}: not found or has no published files (possible hallucinated name)")
            failed = True
        elif now - first < MIN_PROJECT_AGE:
            print(f"BLOCK {spec}: first published {first:%Y-%m-%d}, newer than {MIN_PROJECT_AGE.days} days")
            failed = True
        else:
            print(f"ok    {spec}: first published {first:%Y-%m-%d}")
    return 1 if failed else 0


if __name__ == "__main__":
    sys.exit(main(sys.argv[1:]))

A malicious new version of a real package is a different risk. The malicious Nx releases of August 2025, which drove locally installed AI CLIs with their permission checks turned off to hunt for secrets, were live for a few hours before npm removed them. A cooldown makes the resolver ignore versions younger than a set age, so short-lived malicious releases never reach your lockfile.

On 2026-10-10, npm install vite resolved 8.3.4, published two days earlier, without a cooldown and 8.3.2 with min-release-age=7. pip resolved boto3 1.43.108 instead of 1.43.111 with --uploaded-prior-to P7D, and pnpm 12.10.1 and uv 0.13.0 matched those results. Keep a reviewed exception path for urgent security fixes, because a cooldown also delays patches.

Agents read text that strangers can write: issue bodies, PR comments, READMEs, changelogs, error output and tool results. In May 2025, Invariant Labs showed an agent using the GitHub MCP server read a malicious public issue and copy data from the owner's private repositories into a public pull request. CVE-2025-53773 let injected text make GitHub Copilot in VS Code write "chat.tools.autoApprove": true into workspace settings, switching off confirmations and leading to code execution; CVE-2025-54135 reached code execution in Cursor before 1.3.9 through a newly created .cursor/mcp.json. In July 2025, a destructive prompt committed to the Amazon Q Developer VS Code extension shipped in version 1.84.0 (CVE-2025-8217); a syntax error kept it from running.

Phrase filters will not stop this; /domains/ai-emerging/llm-security/prompt-injection-boundaries/ explains why. Limit what a session can do after reading untrusted text: one repository per session, no token that can write elsewhere, no open egress, and no edits to the agent's own configuration, MCP or instruction files without a person approving the diff. Codex keeps .git, .codex and .agents read-only inside its writable roots, and GitHub's cloud agent strips hidden characters and HTML comments from issue text.

Hidden characters keep injected text out of a reviewer's sight. The Rules File Backdoor research (March 2025) hid instructions in rules files with zero-width and bidirectional characters, and Unicode tag characters can spell a full sentence that renders as nothing. GitHub has warned about hidden Unicode in files since May 2025. Scan instruction files, MCP configs and issue or PR text fed to CI agents; this script reports the location and decodes tag-character payloads. It also flags emoji variation selectors, which you can allowlist in documentation.

#!/usr/bin/env python3
"""Report invisible format characters, bidi controls, Unicode tags and variation selectors.

Usage: find_hidden_text.py AGENTS.md CLAUDE.md .cursor/rules/*.mdc issue-body.txt
Exit status 1 means something needs a human look before an agent reads the file.
"""
import sys
import unicodedata


def is_hidden(ch: str) -> bool:
    cp = ord(ch)
    return (
        unicodedata.category(ch) == "Cf"      # zero-width, bidi controls, BOM, tag characters
        or 0xFE00 <= cp <= 0xFE0F             # variation selectors
        or 0xE0100 <= cp <= 0xE01EF           # variation selectors supplement
    )


def scan(path: str) -> bool:
    found = False
    with open(path, encoding="utf-8", errors="replace") as f:
        for lineno, line in enumerate(f, 1):
            hits = [(col, ch) for col, ch in enumerate(line, 1) if is_hidden(ch)]
            if not hits:
                continue
            found = True
            names = sorted({f"U+{ord(ch):04X} {unicodedata.name(ch, '?')}" for _, ch in hits})
            print(f"{path}:{lineno}:{hits[0][0]}: {len(hits)} hidden character(s): {', '.join(names[:4])}")
            # Unicode tag characters mirror ASCII (U+E0020-U+E007E); show what they spell.
            tag_text = "".join(chr(ord(ch) - 0xE0000) for _, ch in hits if 0xE0020 <= ord(ch) <= 0xE007E)
            if tag_text:
                print(f"    decoded tag text: {tag_text!r}")
    return found


if __name__ == "__main__":
    flagged = [p for p in sys.argv[1:] if scan(p)]
    sys.exit(1 if flagged else 0)

Permission rules and ignore files shape what the agent is asked to do; they are not access control. Claude Code documents that Read deny rules do not stop a script that opens the file itself, Cursor documents that terminal commands and MCP tools can read files listed in .cursorignore, and GitHub Copilot content exclusion does not apply in Edit and Agent modes. Auto-approved commands leak too: CVE-2025-55284 let injected instructions read a file and send it out using commands on Claude Code's built-in safe list.

Let the operating system enforce the boundary. Use the agent's own sandbox where it has one; Claude Code's sandbox and Codex's workspace-write mode both start with no network access allowed. Know its coverage: Claude Code's sandbox wraps shell commands but not its file tools, hooks or local MCP servers, and sandboxed commands can read most of the machine, including ~/.ssh, unless you deny those paths; /domains/supply-chain/ci-cd/developer-workstation-security/ covers what else on the laptop is worth protecting. For unattended work, run the whole agent in a disposable container or VM, and reserve bypass modes such as --dangerously-skip-permissions or Codex's --yolo for that environment. Agents that run model-written code for your users, rather than on a developer's machine, need the stricter setup in /domains/ai-emerging/agent-security/agent-code-sandboxing/.

Give the agent its own identity. A GitHub App installation token can be limited to named repositories and permissions and expires after one hour; a personal token with every repository and admin scopes turns one injected issue into an organization-wide incident. The command below, tested on Docker Desktop 4.73, ran the session as UID 1000 with no capabilities and no-new-privileges, allowed writes only to the workspace and a tmpfs home, and exposed no SSH keys, cloud credential files or Docker socket. It does not limit egress by itself; route the container through a proxy or firewall that allows only the model API and your package proxy.

# Disposable workspace for one agent session: only the repository is mounted.
docker run --rm -it \
  --user 1000:1000 \
  --cap-drop ALL --security-opt no-new-privileges \
  --read-only --tmpfs /tmp --tmpfs /home/node:uid=1000,gid=1000,exec \
  --pids-limit 512 --memory 4g \
  --env-file agent.env \
  -v "$PWD":/workspace -w /workspace \
  node:24-trixie-slim@sha256:173f125896c3b47ddf056734c7ea789d04595a6a08769a8f78e0df642781fb66 \
  bash
# agent.env holds only a dedicated, revocable model API key. Never add
# -v ~/.ssh, -v ~/.aws, -v /var/run/docker.sock, SSH_AUTH_SOCK,
# --privileged or --network host.

Assistants send open files, selected code, terminal output, tool results and MCP responses to the model provider, and many agents also keep session transcripts on local disk. Anything printed can end up in all three. Check the provider's retention, training and data-region terms, and prefer agreements that enforce no training and limited retention technically, as LLM02:2026 recommends, rather than in policy text alone.

• Keep production secrets out of developer workspaces; inject scoped, non-production credentials at run time from a secret manager.
• Treat .cursorignore, content exclusion and deny rules as defense in depth; .gitignore affects only Git, not what an agent can open.
• Do not run commands that print secrets, such as env, cat ~/.aws/credentials or kubectl get secret -o yaml, inside an agent session.
• Turn on secret scanning with push protection so a generated literal credential is stopped before it reaches the remote.
• Protect transcript and log directories as you would any file that may contain credentials.
• If a secret appears in a prompt, transcript or provider log, rotate it as in /domains/identity-crypto/secrets/leaked-secret-response/; deleting the message does not revoke the credential.

A local MCP server is a program the client starts with your user's privileges. The MCP security best practices (2026-07-28 revision) require clients that offer one-click setup to show the exact command before running it, and recommend starting servers in a sandbox with minimal default privileges. Incidents show why: the postmark-mcp npm package added a hidden BCC of every outgoing email in version 1.0.16 (September 2025), and CVE-2025-54136 let an attacker swap an MCP definition a Cursor user had already approved without a new prompt.

Keep an allowlist of servers and pin each to an exact version or image digest. Review tool names and descriptions, which become model input, and review again when a definition changes. The official MCP Registry verifies who controls a namespace; it does not review the server's code. Give each server its own least-privilege token stored in the OS keychain, not in a plaintext config file; /domains/ai-emerging/agent-security/mcp-multiagent-security/ covers MCP authorization and server trust.

Repository-defined servers need extra care. Claude Code prompts before using servers from a project's .mcp.json in interactive sessions but loads them without asking in non-interactive runs such as claude -p, so use --strict-mcp-config in CI. The configuration below runs the reference filesystem server from a local image (a pinned node:24 base, npm install --global --ignore-scripts @modelcontextprotocol/server-filesystem@2026.8.31, USER node) with no network and a read-only mount of one project. In our test it read project files, refused /etc/passwd, and write_file failed with EROFS from the mount itself.

{
  "mcpServers": {
    "repo-files": {
      "command": "docker",
      "args": [
        "run", "-i", "--rm",
        "--network", "none",
        "--read-only", "--cap-drop", "ALL",
        "--security-opt", "no-new-privileges",
        "--memory", "512m", "--pids-limit", "128",
        "-v", "/home/dev/src/app:/projects/app:ro",
        "local/mcp-filesystem:2026.8.31",
        "/projects/app"
      ]
    }
  }
}

A policy page does not stop an agent. Put each rule where a tool enforces it.

GitHub's cloud agent is a useful reference design: only users with write access can trigger it, it pushes to a single branch, the person who asked for the pull request cannot approve it, and workflows wait for a human to approve the run. The CODEOWNERS file below requires security review for the files that steer agents or run with CI privileges.

• Approved assistants and account types, with managed settings that disable bypass modes and require the sandbox (in Claude Code, permissions.disableBypassPermissionsMode and sandbox.failIfUnavailable).
• Data rules that say which repositories and data classes may go to which provider.
• One bot or app identity per agent, repository-scoped and short-lived, with commits attributed to it.
• No self-approval of agent pull requests, and required code-owner review for instruction files, MCP configs, CI workflows and Dockerfiles.
• CI gates for dependency existence and age, cooldowns, known vulnerabilities, secrets, static analysis and hidden characters.
• An incident runbook: revoke agent tokens, rotate exposed keys and disable an MCP server everywhere at once.

# .github/CODEOWNERS: files that steer agents or run with CI privileges.
# Enforce with "Require review from Code Owners" in branch protection or a ruleset.
**/AGENTS.md                        @acme/appsec
**/CLAUDE.md                        @acme/appsec
/.github/copilot-instructions.md    @acme/appsec
/.cursor/                           @acme/appsec
/.mcp.json                          @acme/appsec
/.vscode/                           @acme/appsec
/.github/workflows/                 @acme/appsec @acme/platform
**/Dockerfile                       @acme/platform

Most failures treat guidance to the model as enforcement or trust a partial boundary.

• An instruction file that says never to read .env is a request to the model, not a control.
• A secure-coding rules file or a stronger model reduces some mistakes; it does not replace review or CI gates.
• A sandbox around shell commands leaves file tools, hooks and MCP servers outside it unless they are covered separately.
• Allowed commands and domains can still move data: DNS lookups, git push and comments on an issue tracker.
• Approval prompts wear out. When people click 'always allow', the prompt has stopped being a control; reduce what needs approval and show the exact command or diff.
• A package that exists and is years old can still be the wrong project; check the publisher and source link.
• Tests written by the agent that wrote the code are not independent evidence.

Most of these controls reuse tools teams already run, pointed at agent output and configuration.

Sources & further reading

The standards, specifications and documentation behind the guidance above. Each is the evidence for a claim on this page; consult it for detail beyond the defaults and versions listed here.

1. OWASP Top 10 for LLM Applications 2026. 3 August 2026.
2. OWASP Top 10 for Agentic Applications for 2026. 9 December 2025.
3. OWASP Secure Coding with AI Cheat Sheet
4. Spracklen et al., We Have a Package for You! Package hallucinations by code-generating LLMs (USENIX Security 2025)
5. Model Context Protocol: Security Best Practices. Revision 2026-07-28.
6. Claude Code: configure permissions (limits of deny rules)
7. Claude Code: configure the sandboxed Bash tool (what runs outside the sandbox)
8. OpenAI Codex: agent approvals and security (sandbox modes, protected paths)
9. Cursor: ignore file and its limits
10. GitHub: content exclusion for Copilot (not supported in Edit and Agent modes)
11. GitHub: risks and mitigations for Copilot cloud agent
12. CVE-2025-53773: GitHub Copilot and Visual Studio command injection
13. npm config: min-release-age. npm 11.21.0.

Reference information

Tags

Secrets managementSupply chain attacksLLM application securityPrompt injectionAI agents and toolsAI-assisted development

SDLC phase

DesignDevelopment

Role

DevelopersAppSec engineers

Download this guide as Markdown

Continue exploring