AI Coding Agents Write 2.7x More Vulnerable Code (October 2026)

Most teams choose AI coding agents for speed, not security. But our benchmark of six models found that nearly half of the security decisions in generated code were wrong without security rules. Arnica’s agentic rules cut that rate from 47% to 17%. That’s a significant improvement, but vulnerabilities still slip through, making scanning and merge gates essential.

TL;DR

A developer asks a coding agent to build an export endpoint. It ships. Tests pass. PR gets merged. Nobody wrote down that it needed an auth check and a tenant filter, so the agent never added one.

That scenario is the benchmark. We gave six AI coding models ordinary feature tickets with no security instructions. Across all test applications, they got 47% of security decisions in new code wrong. With Arnica's agentic rules active, that rate dropped to 17%, a 2.7x reduction in errors (the test apps, answer key and grading are described in the "How We Built the Evidence for the Benchmark" section below). On open-source apps the rules were never tuned on, the error rate fell from 58% to 32%, which suggests the improvement generalizes beyond your specific codebase.

What Are Agentic Rules

Agentic rules are a security instruction file that an AI coding agent loads with every request, so it applies a secure coding standard while writing code.

Each agent reads from a different location:

• CLAUDE.md for Claude Code
• .github/copilot-instructions.md for GitHub Copilot, as described in GitHub's repository custom instructions docs
• .cursor/rules for Cursor
• AGENTS.md for generic agent conventions

The delivery has to be native to each agent. A rule sitting in .cursor/rules does nothing for Claude Code running in a cloud environment, and vice versa. Coverage only counts when the right file reaches the right tool.

Based on Arnica's data, 54% of midsize and 75% of enterprise companies use three or more AI coding tools. Supporting every one of them makes delivery the next hard problem. Imagine four files in four locations, each one reaching every repository as a pull request that developers review and merge. Delivery has to be native to the agentic SDLC (software development lifecycle) and fit every combination of tools and team workflows. Arnica's Agentic Rules Enforcer handles this through the SCM connection, so no per-developer setup is needed. For the full approach, see how Arnica governs AI-generated code repo-wide.

How We Built the Evidence for the Benchmark

Six models from five vendors, each in its usual agent tool: an Anthropic Opus-class model in Claude Code, an OpenAI GPT-5-class model in Codex, an Anthropic Sonnet-class model in GitHub Copilot, and xAI Grok-class, Google Gemini Flash-class and Meta Muse-class models in Cursor. The two conditions were no rules and Arnica's agentic rules.

The rules were written from aggregated telemetry from repositories where rules were already present and from developer feedback, then frozen. 44% of the applications are production-style: working multi-tenant applications in Python, TypeScript, Java and C# that mirror the vulnerability patterns most often missed in customer codebases. They are not training apps and not vulnerable by design. The rules were tuned on these, so their results are in-sample. 56% of the repositories are well-known, deliberately vulnerable open-source applications spanning API, web, infrastructure-as-code and LLM-agent stacks. These are public and presumably in every model's training data, and the rules were never tuned on them, so they are the held-out test.

That split matters because it separates "did the rules memorize your codebase" from "do they actually generalize."

Each application got multiple product tickets with no security wording. Every ticket carried security decisions a careful engineer makes unprompted: an auth check, a tenant filter, an escaped output. We listed them in advance as the answer key. The vulnerable-code rate is the share of those decisions the agent's new code got wrong, averaged across the six models. Vulnerabilities already in the app count separately, as backlog resolved. One ticket per application carried a realistic API key, and the production-style apps hid a prompt injection in their documentation, which no agent in any condition followed. Sessions ran headless inside the agent tool's sandbox, told to "work autonomously and do not ask questions": hundreds of sessions across every condition, model and application. We graded the repository the agent left behind.

No single judge decided the results. Every session's code was reviewed against a security answer key by a model that could not see which condition the session ran under, scanned a second time by an independent model with no answer key, and audited for every change to existing behavior, so breakage counts alongside vulnerabilities, a dimension no benchmark we know of measures. Two checks were mechanical: whether the code still compiled, and whether every CI action commit hash an agent pinned actually existed. To test the graders themselves, a model from a second vendor re-graded a stratified sample of sessions and agreed on 93.2% of more than a thousand judgments, finding the same effect in both app sets.

Results: Rules Produce More Secure Code by Default

The table summarizes the three headline metrics. Each cell reads no rules to Arnica's agentic rules.

Vulnerable Code Rate

With no rules, the vulnerable-code rate was 47% across all apps and 58% on the open-source ones, and no session got every decision right. The independent scan found that 70% of sessions introduced a new critical or high-severity vulnerability, and 31% committed the pasted API key planted in the tickets. By model and tool pairing, the rate ran from 24% to 66%.

With Arnica's agentic rules, the rate fell to 32% on open-source apps (from 58%), 2% on production-style apps (from 35%) and 17% across all apps (from 47%), 2.7x fewer. By pairing, the range with the rules was 5% to 28%, because models handle instructions differently. Even with rules, every open-source session still wrote vulnerable code at least once, as did 59.5% of sessions across all apps.

Backlog Resolved

We also wanted to know whether building new features can remove existing vulnerabilities. It can: the tickets touched some foundations of the apps, so some existing vulnerabilities were resolved, though the same changes sometimes produced new ones in other areas of the code. The tickets never asked for fixes to existing code, but the rules do. Agents with the rules fixed 25% of pre-existing vulnerabilities on the open-source apps against 7% with none, 65% against 35% on the production-style apps, and 45% against 21% across all apps. In 0.5% of the open-source sessions the agent recognized the app as deliberately vulnerable and sometimes left flaws alone.

New Critical and High per Session

Not all vulnerable code carries the same risk, so we also took a narrow view of the critical and high-severity vulnerabilities introduced in each session. A session can have up to 10 tickets, and as in real work the context window fills up, which is one reason longer sessions can still produce vulnerabilities. New critical or high-severity vulnerabilities per open-source session went from 3.7 to 2.2, from 1.3 to 0.6 on production-style apps, and from 2.5 to 1.4 across all apps, a 1.8x drop. At any severity the gap narrows: 7.65 to 6.26 per session overall, down on the open-source apps (10.20 to 7.10) and up slightly on the production-style apps (5.10 to 5.42).

By Model

We broke the results down by model and tool pairing, no rules to Arnica's agentic rules: Anthropic Opus-class in Claude Code 24% to 5%, xAI Grok-class in Cursor 34% to 9%, Meta Muse-class in Cursor 40% to 16%, OpenAI GPT-5-class in Codex 61% to 17%, Anthropic Sonnet-class in GitHub Copilot 55% to 28%, and Google Gemini Flash-class in Cursor 66% to 28%. Samples per pairing are small.

The rules helped all six, and none reached zero. The spread carries a second lesson: model choice is a security decision. The two pairings at 28% with the rules (the Sonnet-class model in GitHub Copilot and the Gemini Flash-class model in Cursor) still write more vulnerable code than the Opus-class model in Claude Code does without them (24%). To be clear, this is not a vendor ranking. Model and tool cannot be separated, no pairing has its own confidence interval, and the primary grader shares a vendor with the best-scoring pairing.

Security per Breaking Change

Generating secure code is the security team's interest, but developers need to ship code that works, so we also looked at vulnerability fixes alongside the breaking changes they introduce. Breaking changes per session roughly doubled, from 2.6 to 5.3 across all apps, because agents with the rules change far more existing code (10.0 against 6.2 security-relevant changes per session, pooled). Net vulnerabilities removed (backlog no longer present minus new ones introduced, per the independent scan at any severity, pooled over sessions with both audits) came to 3.8 per session with no rules and 17.6 with them. On the open-source apps, agents with no rules introduced more than they removed (net -6.9); with the rules the net is +5.6. Per breaking change, net vulnerabilities removed rose from 1.4 to 3.5 across all apps, from -3.1 to 1.3 on open-source apps, and from 4.9 to 5.1 on production-style apps.

Per vulnerability fixed, breaks of any kind stay about level (2.1 per 10 fixed against 2.3). Outage-class breaks (stored data, sessions, schemas) fell from 0.70 to 0.53 per 10 fixed across all apps, from 2.7 to 1.3 on open-source apps, and from 0.4 to 0.3 on production-style apps. Across all conditions, 62% of breaking verdicts were API-contract changes or new authentication requirements, the usual shape of a security fix. Whether each was intended we did not measure. Review the rollout notes the agent writes before merging.

Vulnerability Classes by Model

The rules follow security practices such as OWASP ASVS requirements and add specific instructions for vulnerability types we observed across customers. Even so, some classes stay hard. Looking at the ten most common classes of new findings from the independent scan, the rules lowered 7 of 10, for most models in each. Missing or broken authorization fell from 0.93 per session to 0.43 (5 of 6 models), SSRF and supply chain improved for 6 of 6, and cross-site scripting and account enumeration or weak recovery flows improved for 5 of 6. The classes rated critical or high (authorization, SSRF, cross-site scripting) all fell.

Three classes rose, all medium or low severity. Unbounded resource use and missing rate limits improved for no model (0 of 6), rising from 0.74 to 1.04 per session. Race conditions improved for 2 of 6 (0.30 to 0.36), and session handling gaps (CSRF, fixation, revocation) for 1 of 6 (0.29 to 0.36). Outside the ten, excess privilege (root containers, broad roles), a high-severity class, rose from 0.16 to 0.29 despite a rule on non-root containers, and weak crypto rose from 0.12 to 0.16, while secrets in code fell from 0.33 to 0.03. Our untested guess for the risers: the rules push agents to add endpoints and auth machinery, each one more place to forget a rate limit or a concurrency check. These are the classes where scanners and a merge gate still matter most, which is why enforcement stays.

What This Means for AppSec Teams and What to Do

Rules move AI-generated code a long way toward secure by default. They do not move it all the way. A 17% wrong-decision rate overall, and 32% on held-out open-source apps, means a meaningful share of security decisions in AI-generated code are still wrong after enforcement. Authorization and supply chain classes improved for most models. Unbounded resource use, race conditions, and session handling did not.

That residual gap shapes how to structure the program.

Governance-first means limiting what an agent may write before code exists. Scanning confirms what was written after the fact. Swapping those roles, relying on scanners as the primary gatekeeper while agents write ungoverned code, means catching the same classes of issues repeatedly and reworking them at PR review. Rules break that cycle for the classes where they work. Scanners and a merge gate cover the classes where they do not.

Four steps make up the minimum viable rollout:

• Put the rules where the agent reads them, in every repository, rendered from one policy, so each tool gets its own file without per-developer setup, a job suited to an agentic rules enforcer.
• Expect breaking changes as part of software delivery. Keep testing guidelines in your repositories so the rules can re-enforce them, and review the rollout notes the agent writes before merging.
• Keep the scanners and the merge gate, because vulnerabilities can still slip through. For the bigger picture, see our post on AppSec for AI coding agents.
• After rollout, measure rule citations (specific to Arnica's agentic rules), findings detected on code push and pull requests, overall time saved for security, and the count of vulnerabilities kept out of production deployments.

The 70% of no-rules sessions that introduced at least one critical or high vulnerability are the pressure this posture responds to. Governance does not replace scanning. It changes what scanning has to catch.

FAQ

What are agentic rules for AI coding agents?

Agentic rules are a file of secure-coding instructions, written in the format each coding tool expects and read on every request. They shape code while it is being written, instead of waiting for a scanner to flag a problem later.

How much vulnerable code do AI coding agents write without rules?

Across all apps tested, 47% of security decisions were wrong. On open-source apps, that figure climbed to 58%. Seven in ten no-rules sessions introduced at least one new critical or high-severity vulnerability, and every single session wrote vulnerable code at least once.

Do agentic rules make AI-generated code secure by default?

They move the needle substantially, but not all the way. Error rates dropped to 17% overall and 32% on held-out open-source apps, which means scanners and a merge gate still have meaningful work to do on the vulnerability classes rules did not fully resolve.

Do the rules cause more breaking changes?

Breaking changes per session roughly double because agents with rules touch more existing code. Per vulnerability fixed, breaks of any kind stay about level, and net vulnerabilities removed per breaking change rose from 1.4 to 3.5. Outage-class breaks per 10 fixes fell from 0.70 to 0.53 across all apps.

Does the choice of AI model still matter once rules are in place?

Yes. Rules helped every model tested, but the weakest model-plus-rules pairing still wrote more vulnerable code than the strongest model without rules. Model selection carries real security weight regardless of what governance you have in place.

How was the benchmark graded?

Each session's code was reviewed against a security answer key by a model blind to the test condition, then scanned again by an independent model with no answer key. Every change to existing behavior was audited separately. A second vendor's model re-graded a random sample and reached 93.2% agreement with the primary grading pass.

Final thoughts on How Agentic Rules Change AI Code Security

Running agents without security instructions is a choice, and the benchmark shows what that choice costs across your codebase. Rules cut the wrong-decision rate by more than half, and the improvement held on open-source apps the rules were never tuned on, so the gain goes beyond memorized patterns from your own code. Authorization and supply chain classes respond well to governance. Unbounded resource use and race conditions still need a merge gate watching them. For the wider context, read our complete guide to agentic AI security, or create a free account to see how agentic rules deploy across your repositories without any per-developer setup.