Agile InsightsBriefing
A tall stack of printed pages with a single page pulled out under a magnifying glass, a rubber approval stamp lying unused, a large undo key and a studio microphone with headphones on a dark workbench, labelled Risk tier, Owner, Undo and Episode 12.

Episode 12 · Primer and discussion · About 15 minutes in two parts

Who reads the code when the AI writes it?

Why asking whether a person looked at it is a weak control for AI written work, and how to match the check to the risk, keep people on the risky calls and make mistakes cheap to undo.

#oversight

Part 1: Primer · About 7 minutes

Download the MP3 · Linda and Leonard, AI voices, read a script written from the sources below.

Part 2: Is it safe to stop reading? · About 8 minutes

Download the MP3 · Linda and Leonard, AI voices, read a script written from the sources below.

In this episode

01A signature is a weak proxyHuman reading was never a strong bug filter. A 2013 Microsoft study found defect comments a small share of what reviewers wrote, and a 2026 study found reviewers kinder to AI written changes.
02The change sets the tierAnthropic's Claude Code team gives low risk changes no human approval once automated checks pass, ordinary changes one person and the highest risk code two, with a short list that always needs a person.
03Rules, not featuresAn AI's approval never counts as a person's, whoever prompted the AI owns the change, and an approval covers one version. Anthropic says its review product posts findings but never approves or blocks.
04Read the revert number carefullyAnthropic says merged changes more than doubled from June to September while the share reverted fell over forty per cent. It is a share, so total reverts may be similar. The figures are unaudited.
05Tier it, own it, undo itFind controls that amount to a person checked it, tier the work by risk, never let an AI approval fill a human slot, pick a guardrail measure that tightens the gate, and make undo cheap.
Sources and links (10)
How the Claude Code team does code review · Read on the claude.dev blog
Anthropic's own account of its risk tiered review process, 15 October 2026; the source of every Anthropic figure in the episode.
Code Review documentation · Read the docs
Anthropic's product documentation, including that a review never approves or blocks a pull request.
Announcing the 2024 DORA report · Read on Google Cloud
Google Cloud's summary of the 2024 DORA research, including the estimated fall in delivery stability as AI adoption rose.
State of AI-assisted Software Development 2025 · Read on dora.dev
DORA's 2025 report, which describes AI as an amplifier of an organisation's existing strengths and weaknesses.
The AI Productivity Paradox · Read on Faros AI
Faros AI, July 2025: telemetry showing more merged changes and much longer review times on teams heavy on AI.
Early-2025 AI and experienced open-source developers · Read on METR
METR's randomised trial, July 2025: developers were slower with AI tools yet believed they were faster.
More Code, Less Reuse · Read on arXiv
Huang and colleagues, MSR 2026: reviewers reacted more positively to AI written pull requests that carried more redundant code.
Expectations, Outcomes, and Challenges of Modern Code Review · Read the paper
Bacchelli and Bird, Microsoft Research, 2013: review is expected to find defects but mostly does other things.
ASA 330 The Auditor's Responses to Assessed Risks · Read on the AUASB site
The Australian auditing standard that asks for more persuasive evidence the higher the assessed risk.
Ironies of Automation · Read on Wikipedia
A summary of Lisanne Bainbridge's 1983 paper on why automating the routine leaves people the hardest, least practised work.

Written from public sources and checked on 21 October 2026. The Anthropic article and documentation, the METR study, the Google Cloud DORA 2024 summary, the Faros AI summary and ASA 330 were read in full; the DORA 2025 report and the 2026 study were read in summary or abstract only. Anthropic's figures are its own and unaudited, and Anthropic sells an AI review product. Please check the originals before quoting. This is general information, not professional advice.

Our fact check · 25 claims: 15 confirmed, 9 reported, 1 not yet known

Before we record, every claim the hosts make is checked against the sources (this table was last updated 21 October 2026). Confirmed means a primary source or two independent reports; Reported means one party says so; Partly right, Disputed, Unverified and Not yet known mean what they say, and the hosts say so on air. This is the table as we left it.

  1. ReportedClaude writes about 90% of Claude Code’s own codeClaude.dev, 15 Oct 2026; “We’ve said before”; method not public
  2. ReportedMerged PRs to Claude Code’s codebase more than doubled, June to September 2026Claude.dev; Claude Code repos only; unaudited
  3. ReportedThe share of merged PRs reverted fell by more than 40%Claude.dev; Relative fall of a share, no baseline. On doubled volume, 2 × 0.6 = 1.2: total reverts may be about the same or higher. Never say “reverts fell 40%”
  4. ReportedAbout 2,500 changes a day in May to over 30,000 in SeptemberClaude.dev; Anthropic’s monorepo, separate from Claude Code; “changes landing”, not PRs
  5. ReportedMost PRs to merge with at most one human, no rise in reverts or incidentsClaude.dev; A goal, not a result
  6. ConfirmedTiers: low none, medium one, high two; classifier sets the tier; some paths always need a personClaude.dev; Stated policy; Classifier accuracy not disclosed
  7. ConfirmedA bot’s approval never fills a human slot; prompter owns the change; approval covers one commitClaude.dev; Stated policy
  8. ReportedInside Anthropic, open AI or security review findings block the mergeClaude.dev
  9. ConfirmedThe sold product never approves or blocks; check run never fails; it reads but does not run the codeCode Review docs; launch post 9 Mar 2026; The gate is something the customer builds
  10. ReportedSubstantive review comments 16% → 54% of internal PRs at previewLaunch post; claude.dev
  11. ReportedOne-round pass 65% → 78%; five-plus rounds 8.9% → 1.5%; mean reviews 2.08 → 1.32Claude.dev; After a round cap and diff-only re-review; one PR had run 50+ rounds overnight
  12. ConfirmedIncidents become the reviewer’s daily eval setClaude.dev; Stated practice
  13. ConfirmedNo circuit breakers yet that tighten approval when revert or incident numbers moveClaude.dev; Admission against interest
  14. ConfirmedShared understanding now comes from asking Claude for an explainer; “no formal process”Claude.dev; Stated
  15. ConfirmedProduct GA on 15 Oct 2026, the same day as the articleAnthropic GA post
  16. ConfirmedDORA 2024: as AI adoption rose, throughput fell an estimated 1.5% and stability 7.2%Google Cloud, 23 Oct 2024; Survey-model estimate
  17. ConfirmedDORA 2025: AI is an amplifier of existing strengths and weaknessesDora.dev; InfoQ; ~5,000 respondents; full report not read
  18. ReportedHigh-AI teams: +98% PRs merged, +91% review time, +154% PR size, +9% bugs per developerFaros AI, 23 Jul 2025; Vendor telemetry, correlational
  19. ConfirmedExperienced developers 19% slower with early-2025 AI, believed 20% fasterMETR, 10 Jul 2025; RCT; METR’s Feb 2026 update says the follow-up signal is unreliable and developers are likely faster now
  20. ConfirmedReviewers react more neutrally or positively to AI PRs, which carry more redundant codeHuang et al., MSR 2026; Abstract; Direct evidence that human reading is a weak proxy
  21. ConfirmedCode review finds fewer defects than expected; it mostly spreads understandingBacchelli and Bird, ICSE 2013
  22. ConfirmedAutomating the routine leaves people rarer, harder interventions with less practiceBainbridge, 1983; Via summary
  23. ConfirmedAuditors must obtain more persuasive evidence the higher the assessed riskAUASB ASA 330 paras 6–7; The risk-based check outside software
  24. ConfirmedStandard IT changes are low risk and pre-approvedITIL via Atlassian; Established practice
  25. Not yet knownIndependent audit or replication of Anthropic’s numbers—; None found

Want to know more?

Sign up for our podcast at agile-insights.com.au.

Read the transcript · Part 1: Primer

Linda G'day, and welcome to the Agile Insights briefing. I'm Linda. Here is the main idea. Asking whether a person looked at it is a weak way to check work an AI has written. A stronger check matches the effort to the risk, lets machines do most of the reading, and keeps people on the risky decisions. By the end of this part you will know how that works, and how to use it beyond software. One disclosure first. Our main source is Anthropic, which sells an AI review product, and this script was drafted with Claude, Anthropic's model, then checked by a person.

Leonard I'm Leonard, and I'll be the newcomer today. I have never reviewed code in my life. So what is code review, and why is everyone suddenly talking about it?

Linda Code review means a second person reads a proposed change before it joins the shared code. It has two jobs. It catches problems before they ship, and it spreads understanding of the code across the team. It is news because of volume. On the fifteenth of October, three Anthropic authors described how the Claude Code team reviews code that Claude mostly writes. Anthropic says Claude writes about ninety per cent of Claude Code. In Anthropic's larger shared code base, changes landing each day went from about two and a half thousand in May to over thirty thousand in September.

Leonard Nobody reads that much.

Leonard That sounds like a staffing problem to me. The obvious answer is to hire more reviewers, or slow everyone down until the reading catches up. Why doesn't that work?

Linda Because human reading was never as strong a filter as we assumed. In a Microsoft study from 2013, finding defects was the main reason people gave for review. Yet defect comments were a small share of what reviewers wrote. A study this year found reviewers kinder to AI written changes, even though the AI code carried more redundancy. Plausible work gets an easier ride. The Anthropic authors put it plainly. Did a human read the code is an imprecise proxy for what could go wrong, and which check should catch it.

Leonard We have all waved one through.

Linda So here is the method, in four steps. Step one is to sort changes by risk, and let the change set the tier, not the author. Anthropic uses three tiers. Low risk, like documentation, needs no human approval once the automated checks pass. Ordinary changes need one person. The highest risk code needs two. A model reads each change and picks the tier, and a short list of sensitive areas always needs a person...

Leonard That is less exotic than it sounds. Plenty of IT teams already pre approve standard changes, like adding storage, and send the unusual ones to a change board. Auditors work the same way. The Australian auditing standard asks for more persuasive evidence the higher the assessed risk. Nobody checks every receipt. What is new is the speed, and that a machine does the sorting.

Linda Exactly.

Linda Step two is to let machines do most of the reading, in layers. Before a person looks, the AI runs the tests and shows the output, and a second AI reviews the change with fresh eyes. Then every change gets an automated review, a separate security review and the usual tests. Inside Anthropic, an open finding blocks the merge until someone fixes it or writes down why it is wrong. Each layer misses something. Few problems get through all of them...

Leonard Here is the newcomer's worry, though. If a machine clears the low risk work, who is actually accountable when it goes wrong? Does the machine get the blame?

Linda A person, always. Step three is to write down who owns what. An AI's approval never counts as a person's. Whoever prompted the AI owns the change, as if they had typed it. And an approval covers one version, so if the work changes, they sign again. Those are rules an organisation writes, not features it buys. Anthropic's own documents say its review product posts findings but never approves or blocks a change.

Leonard Rules, not features.

Linda Step four is to make mistakes cheap to undo, and to measure whether the shortcut works. New behaviour ships behind switches that turn off without a new release, to staff first, then to users in stages. Anthropic counts the human time saved, and watches reverts, meaning changes that had to be undone, and incidents. If those rise, it says, the time was not saved, only moved.

Leonard So what does Anthropic say it got for all that effort? And, given who is saying it, how much weight should a listener put on the numbers?

Linda Here is what is confirmed and what is not. Anthropic says merged changes to Claude Code more than doubled from June to September, while the share reverted fell by more than forty per cent. Notice the word share. Double the volume and cut the share by forty per cent, and the total can stay about the same, or rise. The figures are Anthropic's own and unaudited. And Anthropic admits it has nothing yet that tightens approval automatically when quality slips.

Leonard Huh. I'd have misread that.

Leonard The common misunderstanding is that this takes people out. It doesn't. Anthropic keeps people on five decisions. Whether to build it at all, risky code like billing, design choices others will copy, reading the evidence, and owning the outcome. And this stops being a software story. Any organisation drafting contracts or board papers with AI has a control that says a person checked it.

Linda So, what to do. List where your control is simply that a person checked it. Sort the work by risk, and name the short list that always needs a person, such as money, access and anything hard to reverse. Never let an AI approval fill a human slot. Pick a measure, like reversals or complaints, and watch it before you loosen anything. And make undo cheap.

Leonard That's part one. In part two, Linda and I argue about whether any of this is safe.

Read the transcript · Part 2: Is it safe to stop reading?

Linda G'day, and welcome back to the Agile Insights briefing. I'm Linda. The main idea again. Asking whether a person looked at it is a weak control for work an AI has written, and the stronger one matches the check to the risk. In this part we test that idea. Is it safe to stop reading most of the work, and what does the wider evidence say? Same disclosure as part one. Our main source is Anthropic, which sells an AI review product, and this script was drafted with Claude and checked by a person.

Leonard I'm Leonard. A quick recap for anyone who skipped part one. Code review means a second person reads a change before it joins the shared code. Anthropic's Claude Code team now sorts every change into a risk tier. Low risk needs no human approval once the automated checks pass, ordinary changes need one person, and the riskiest need two. Machines do most of the reading. An AI's approval never counts as a person's, whoever prompted a change owns it, and mistakes are built to be undone quickly.

Linda So the question is whether that is brave or reckless. Let me make the case for it. Human reading was never the strong filter we pictured. Microsoft's 2013 study found defect comments were a small share of what reviewers wrote. A study this year found reviewers kinder to AI written changes, even though those changes carried more redundant code. Meanwhile, volume has exploded. The analytics firm Faros AI looked at teams heavy on AI in 2025. They merged ninety eight per cent more changes, and review time rose ninety one per cent. A tired person skimming a queue is not a control. Layers of checks, fast undo, and a measure that tells you when it is failing, together make a control...

Leonard I'll take the other side, because I think the evidence is thinner than the confidence. Every outcome number in the article is Anthropic's own and unaudited. It went up the same day Anthropic's review product went on general sale. The wider data is less rosy. Google's DORA research in 2024 found that as AI adoption rose, delivery stability fell by an estimated seven point two per cent. Its 2025 report calls AI an amplifier. Strong organisations get stronger, and the rest see their weaknesses magnified. Anthropic may be showing what excellent foundations make possible, not what a typical organisation will get.

Linda That's fair, and the headline number shows why care matters. Anthropic says merged changes to Claude Code more than doubled from June to September, and the share reverted fell by more than forty per cent. A share, not a count. Double the volume, cut the share by forty per cent, and the total is roughly one point two times what it was. So the article does not claim fewer reverts overall. It claims quality per change held up while volume doubled. That is still useful, but nobody outside Anthropic has checked it.

Leonard Arithmetic, the quiet hero.

Leonard My bigger worry is the people left on the high risk tier. In 1983 the psychologist Lisanne Bainbridge wrote a paper called Ironies of Automation. Her point was that when machines take the routine work, people are left with the rare hard interventions, and less practice at them. If two senior people only ever see the scariest changes, are they sharper, or rustier? There is a second loss too. Review used to spread understanding of the code. Anthropic says people now ask Claude to explain the part they are about to change. It admits it has no formal process for this.

Linda I think that is the strongest objection, and I don't think Anthropic has fully answered it. But the alternative is not careful reading of everything. That ended when the volume went up. The real question is where attention goes. Anthropic keeps people on five decisions. Whether to build the thing, risky code like billing, design choices others will copy, reading the evidence, and owning the outcome. That is a better use of a senior person than reading a one line wording fix. The skill problem is real, so it needs its own plan.

Leonard Fair. It does.

Leonard Here's where I think we agree. The control is the rule, not the tool. Anthropic's own documents say its review product posts findings but never approves or blocks a change. Inside Anthropic, an open finding blocks the merge because the team built that gate around the tool. An organisation that buys a review product and calls its review automated may have findings with no enforcement at all. Whatever you buy, you still have to write the rule that says this cannot go ahead until that is resolved.

Linda We agree. Mark the date.

Linda And there is one admission I would underline for any board. Anthropic says it does not yet have circuit breakers that tighten approval when revert or incident numbers move. Its engineers want them before more code merges with no human at all. That is the gap. Loosening a control with a measure is good practice. Loosening it with nothing that pulls it tight again when the measure goes bad is how a sensible shortcut becomes a habit nobody remembers agreeing to.

Leonard So, what is known and what isn't. Confirmed are the rules Anthropic describes, including the tiers and the list that always needs a person. So is the fact that the product itself does not block. Reported by Anthropic alone are the doubling, the revert share and the review improvements. Independent research adds two cautions. Perceived speed is unreliable. In METR's 2025 trial, developers were nineteen per cent slower, yet believed they were faster. And DORA finds stability is still where AI hurts. Not yet known is whether this works outside Anthropic.

Linda And none of this is really about software. Auditors already work this way. The Australian auditing standard asks for more persuasive evidence the higher the assessed risk, and nobody checks every receipt. IT teams pre approve standard changes and send the unusual ones to a board. What AI changes is the volume, and that a machine now does the sorting. Contracts, policies, payments and board papers drafted with AI all need the same thinking, so let's finish with what to do...

Leonard First, find every control that amounts to a person checked it, and ask what it is meant to catch. Second, write risk tiers set by the work, not the author, plus a short list that always needs a person, such as money, access and anything hard to reverse. Third, never let an AI approval fill a human slot, and make the person who directed the AI the owner. Fourth, make an approval lapse when the work changes after it was signed.

Linda Fifth, pick a guardrail measure, like reversals, complaints or rework, and decide in advance what tightens the gate when it moves. Sixth, make undo cheap, with staged roll outs and changes small enough to reverse. And seventh, plan on purpose for the understanding and skill that reading used to build, because that will not happen by itself. If you'd like to know more, sign up for our podcast at agile-insights.com.au.

Leonard Thanks for listening.

More on #oversight

Episode 37 · PrimerWhat is an AI skill (and why you should care)Episode 36 · Primer and discussionChecking agents in production, even when you can’t read their workEpisode 35 · DiscussionWho watches the agents? Detect, stop, and let a person approve the rulesReference 13 · PrimerThe Commonwealth AI rulebook: the Policy, the technical standard and the APS AI PlanReference 12 · PrimerThe DTA AI Reference Architecture: one blueprint for government AIEpisode 32 · Primer and discussionAgentic video is real: what to try, and what to check

Every episode tagged #oversight

All of the briefings (38) · Follow Briefing in Apple Podcasts or any podcast app

Everything from Agile Insights: briefings, Making You Think and the reference series