AI Code Skip Audit

The refund endpoint passed two tests and a four-minute review, then paid one order out twice.

A coding assistant wrote the route and its tests. Two days later, finance found the double refund, eleven seconds apart; next, security showed a customer could refund a stranger's order by editing the URL. Every defect was an absence.

(Composite example, assembled from patterns common in AI-assisted delivery. The figures are illustrative.)

Review can't comment on a missing line, and generated tests share the code's gaps. This worksheet starts from the ticket: state seven requirements a model won't infer, score how often recent PRs dropped them, and make each PR prove all seven. Veracode's Spring 2026 update shows syntax pass rates climbing to 95% (from roughly half in 2023) as security stayed at 45–55%, and its July 2026 report puts the average security pass rate at 56% and found that model size made no difference to security. Interpretation (ours): upgrades fix what you asked for, not what you never stated.

Step 1: write the unstated requirements into the ticket

Fill it in before generation with a domain expert (roughly ten minutes, in our experience). n/a needs a challengeable reason.

Who may do this:      role=______, scope (tenant/region/owner)=______. Everyone else: ___.
Hostile input:        field types=______, bounds=______, rejection code=___.
If it fails halfway:  state saved before each external call=______; reconciliation on timeout=______.
If it runs twice:     idempotency key=______; repeated key returns ___, causes ___ effect(s).
If it runs at once:   lock, version check or constraint=______; the loser gets ___.
What gets recorded:   audit event {actor, ______, correlation ID, timestamp}.
Limits/cost:          max rows=___, timeout=___, rate=___ (or n/a + reason)

For the composite refund endpoint: own-tenant agents, else 403; a positive Decimal within the unrefunded balance, else 422; PENDING before the provider call; a required idempotency key; a row lock; an audit event; limits n/a. Many payment APIs, Stripe's among them, accept an idempotency key, so line four is often configuration.

Step 2: require evidence for seven rows on the PR

Link the covering line or test, or an n/a reason. No link, no merge.

#	Question	Why it's skipped	Evidence	Link or n/a reason
1	Who may call it?	Roles and tenancy aren't in the prompt	Wrong role and wrong tenant each get 403	☐ ______
2	Hostile input?	Prompt examples are well formed	Bad inputs tested at each entry point	☐ ______
3	Fails halfway?	Prompts never mention timeouts	State left after each failed external call	☐ ______
4	Runs twice?	Retries come from networks and users	Duplicate request, one effect	☐ ______
5	Runs at once?	Generated tests are sequential	Shared state and its lock or constraint	☐ ______
6	What's recorded?	Tickets don't ask for audit trails	Audit event with actor and correlation ID	☐ ______
7	Limits and cost?	Test data is small	Pagination and query counts at real volume	☐ ______

• ☐ High-risk changes: each present row also has an independent test, written from the Step 1 block, not the code.

Scanners tend to be weak on rows 1, 3, 4 and 5, where the flaw is an absence. Run one, but gate on this table.

Step 3: score your last ten AI-assisted PRs

Score each cell P (present, with evidence), A (absent) or N (n/a, reasoned).

PR	1 Who	2 Input	3 Halfway	4 Twice	5 At once	6 Recorded	7 Limits	Absences
Refund (composite)	A	A	A	A	A	A	N	6
1	___	___	___	___	___	___	___	___
2	___	___	___	___	___	___	___	___
3	___	___	___	___	___	___	___	___
4	___	___	___	___	___	___	___	___
5	___	___	___	___	___	___	___	___
6	___	___	___	___	___	___	___	___
7	___	___	___	___	___	___	___	___
8	___	___	___	___	___	___	___	___
9	___	___	___	___	___	___	___	___
10	___	___	___	___	___	___	___	___
Per row	___	___	___	___	___	___	___	Total: ___ of 70

• ☐ Over about 10 absences: gate now. ShiftQuality's rule of thumb, not a benchmark.

• ☐ Absences bunched in one row: make it the first mandatory ticket line.

• ☐ Can't identify AI-assisted PRs? Label first (Step 5).

Quick checks:

• ☐ Sort five recent PRs' review comments into "missing" and "wrong". An empty first pile means reviewers miss gaps.

• ☐ Tag each requirement one AI-written function meets as written down or guessed. Few guesses means unwritten needs are absent.

• ☐ If AI fixes its own findings, diff first and final versions. Shukla et al. (2025) found a 37.6% rise in critical vulnerabilities after five rounds of LLM "improvement" in a controlled 400-sample experiment.

Step 4: match the gate to the risk

Context	Gate	Wrong if
Throwaway prototype	None, or rows 1–2	It gets a production URL "for now"
Internal script, one operator	Rows 3, 6, 7	Someone schedules it or points it at production data
Strong shared middleware	Full; rows 1 and 4 confirmed in a line	Step 3 still finds absences there
Money, personal or regulated data	Full, plus independent tests	None

• ☐ Assign each repo with AI-assisted merges last quarter to a row; one you can't place has no agreed risk level.

Step 5: install the gate

Targets are ours, not industry benchmarks.

Action	What to do	Owner	Done when	☐
1	Label AI-authored PRs (label or Co-authored-by: trailer)	Tech lead	90% or more labelled, two weeks running	☐
2	Make the Step 1 block a mandatory ticket field	Product owner and QA lead	It can't be skipped	☐
3	Store defaults (authorisation, idempotency, audit shape) in agent instructions	Tech lead	The file is code-reviewed	☐
4	Make the Step 2 rows a mandatory PR checklist	QA lead	Incomplete checklists block merge	☐
5	Independent tests on high-risk present rows	QA lead	Visible on high-risk PRs	☐

Label first: unlabelled code can't be measured. Georgia Tech's Vibe Security Radar had attributed 74 public CVEs to AI coding tools as of March 2026 from AI-signed commits only, and the lead researcher estimates the real figure at five to ten times that.

Step 6: check the gate monthly

Thresholds are ours.

Failure	Detect	Prevent	☐
Checklist theatre: ticks without links	Audit five merged PRs' links monthly	No link, no merge	☐
Mirror tests: tests that confirm the bugs	Test and code from one session	Step 5, action 5	☐
Reflexive n/a	Over two n/a on customer-facing work	Each n/a needs a challengeable reason	☐
Unlabelled bypass	Labels fall while licence usage holds	Keep the gate light (Step 4)	☐

How to use it

• Who: an engineering or QA lead with a domain expert. Start with Step 3.

• Output: a scored grid, a ticket block, an evidence checklist and a gate level per repo.

• Done when: high-risk PRs merge only with all seven rows linked or reasoned, and monthly checks find no bare ticks.

Where this changes

Small teams without money or personal data can rerun Step 3 quarterly. Regulated teams should keep grids and links as audit evidence and test every customer-facing change independently.

The reasoning is in what AI-generated code structurally skips. See also this review tutorial, the acceptance criteria template and the register of AI failure modes.

Final takeaway

Generated code delivers what the ticket says. Write the seven usual omissions into it, count them, and make every PR show where each is handled.

Sources

• Veracode, Spring 2026 GenAI Code Security Update: Despite Claims, AI Models Are Still Failing Security (24 March 2026). https://www.veracode.com/blog/spring-2026-genai-code-security

• Veracode, 2026 GenAI Code Security Report press release (28 July 2026). https://www.veracode.com/news/llms-are-getting-smarter-but-not-safer-veracode-2026-genai-code-security-report-finds-ai-generated-code-security-has-stalled-at-56%25-pass-rate/

• Shukla, S., Joshi, H., Syed, R., Security Degradation in Iterative AI Code Generation: A Systematic Analysis of the Paradox, arXiv:2506.11022 (2025). https://arxiv.org/abs/2506.11022

• Security Researchers Sound the Alarm on Vulnerabilities in AI-Generated Code, Infosecurity Magazine (26 March 2026), reporting Georgia Tech SSLab's Vibe Security Radar. https://www.infosecurity-magazine.com/news/ai-generated-code-vulnerabilities/

• Stripe API Reference, Idempotent requests. https://docs.stripe.com/api/idempotent_requests