Briefing
Episode 12 · Primer and discussion · About 15 minutes in two parts
Download the MP3 · Linda and Leonard, AI voices, read a script written from the sources below.
Download the MP3 · Linda and Leonard, AI voices, read a script written from the sources below.
Written from public sources and checked on 21 October 2026. The Anthropic article and documentation, the METR study, the Google Cloud DORA 2024 summary, the Faros AI summary and ASA 330 were read in full; the DORA 2025 report and the 2026 study were read in summary or abstract only. Anthropic's figures are its own and unaudited, and Anthropic sells an AI review product. Please check the originals before quoting. This is general information, not professional advice.
Before we record, every claim the hosts make is checked against the sources (this table was last updated 21 October 2026). Confirmed means a primary source or two independent reports; Reported means one party says so; Partly right, Disputed, Unverified and Not yet known mean what they say, and the hosts say so on air. This is the table as we left it.
Sign up for our podcast at agile-insights.com.au.
Linda G'day, and welcome to the Agile Insights briefing. I'm Linda. Here is the main idea. Asking whether a person looked at it is a weak way to check work an AI has written. A stronger check matches the effort to the risk, lets machines do most of the reading, and keeps people on the risky decisions. By the end of this part you will know how that works, and how to use it beyond software. One disclosure first. Our main source is Anthropic, which sells an AI review product, and this script was drafted with Claude, Anthropic's model, then checked by a person.
Leonard I'm Leonard, and I'll be the newcomer today. I have never reviewed code in my life. So what is code review, and why is everyone suddenly talking about it?
Linda Code review means a second person reads a proposed change before it joins the shared code. It has two jobs. It catches problems before they ship, and it spreads understanding of the code across the team. It is news because of volume. On the fifteenth of October, three Anthropic authors described how the Claude Code team reviews code that Claude mostly writes. Anthropic says Claude writes about ninety per cent of Claude Code. In Anthropic's larger shared code base, changes landing each day went from about two and a half thousand in May to over thirty thousand in September.
Leonard Nobody reads that much.
Leonard That sounds like a staffing problem to me. The obvious answer is to hire more reviewers, or slow everyone down until the reading catches up. Why doesn't that work?
Linda Because human reading was never as strong a filter as we assumed. In a Microsoft study from 2013, finding defects was the main reason people gave for review. Yet defect comments were a small share of what reviewers wrote. A study this year found reviewers kinder to AI written changes, even though the AI code carried more redundancy. Plausible work gets an easier ride. The Anthropic authors put it plainly. Did a human read the code is an imprecise proxy for what could go wrong, and which check should catch it.
Leonard We have all waved one through.
Linda So here is the method, in four steps. Step one is to sort changes by risk, and let the change set the tier, not the author. Anthropic uses three tiers. Low risk, like documentation, needs no human approval once the automated checks pass. Ordinary changes need one person. The highest risk code needs two. A model reads each change and picks the tier, and a short list of sensitive areas always needs a person...
Leonard That is less exotic than it sounds. Plenty of IT teams already pre approve standard changes, like adding storage, and send the unusual ones to a change board. Auditors work the same way. The Australian auditing standard asks for more persuasive evidence the higher the assessed risk. Nobody checks every receipt. What is new is the speed, and that a machine does the sorting.
Linda Exactly.
Linda Step two is to let machines do most of the reading, in layers. Before a person looks, the AI runs the tests and shows the output, and a second AI reviews the change with fresh eyes. Then every change gets an automated review, a separate security review and the usual tests. Inside Anthropic, an open finding blocks the merge until someone fixes it or writes down why it is wrong. Each layer misses something. Few problems get through all of them...
Leonard Here is the newcomer's worry, though. If a machine clears the low risk work, who is actually accountable when it goes wrong? Does the machine get the blame?
Linda A person, always. Step three is to write down who owns what. An AI's approval never counts as a person's. Whoever prompted the AI owns the change, as if they had typed it. And an approval covers one version, so if the work changes, they sign again. Those are rules an organisation writes, not features it buys. Anthropic's own documents say its review product posts findings but never approves or blocks a change.
Leonard Rules, not features.
Linda Step four is to make mistakes cheap to undo, and to measure whether the shortcut works. New behaviour ships behind switches that turn off without a new release, to staff first, then to users in stages. Anthropic counts the human time saved, and watches reverts, meaning changes that had to be undone, and incidents. If those rise, it says, the time was not saved, only moved.
Leonard So what does Anthropic say it got for all that effort? And, given who is saying it, how much weight should a listener put on the numbers?
Linda Here is what is confirmed and what is not. Anthropic says merged changes to Claude Code more than doubled from June to September, while the share reverted fell by more than forty per cent. Notice the word share. Double the volume and cut the share by forty per cent, and the total can stay about the same, or rise. The figures are Anthropic's own and unaudited. And Anthropic admits it has nothing yet that tightens approval automatically when quality slips.
Leonard Huh. I'd have misread that.
Leonard The common misunderstanding is that this takes people out. It doesn't. Anthropic keeps people on five decisions. Whether to build it at all, risky code like billing, design choices others will copy, reading the evidence, and owning the outcome. And this stops being a software story. Any organisation drafting contracts or board papers with AI has a control that says a person checked it.
Linda So, what to do. List where your control is simply that a person checked it. Sort the work by risk, and name the short list that always needs a person, such as money, access and anything hard to reverse. Never let an AI approval fill a human slot. Pick a measure, like reversals or complaints, and watch it before you loosen anything. And make undo cheap.
Leonard That's part one. In part two, Linda and I argue about whether any of this is safe.
Linda G'day, and welcome back to the Agile Insights briefing. I'm Linda. The main idea again. Asking whether a person looked at it is a weak control for work an AI has written, and the stronger one matches the check to the risk. In this part we test that idea. Is it safe to stop reading most of the work, and what does the wider evidence say? Same disclosure as part one. Our main source is Anthropic, which sells an AI review product, and this script was drafted with Claude and checked by a person.
Leonard I'm Leonard. A quick recap for anyone who skipped part one. Code review means a second person reads a change before it joins the shared code. Anthropic's Claude Code team now sorts every change into a risk tier. Low risk needs no human approval once the automated checks pass, ordinary changes need one person, and the riskiest need two. Machines do most of the reading. An AI's approval never counts as a person's, whoever prompted a change owns it, and mistakes are built to be undone quickly.
Linda So the question is whether that is brave or reckless. Let me make the case for it. Human reading was never the strong filter we pictured. Microsoft's 2013 study found defect comments were a small share of what reviewers wrote. A study this year found reviewers kinder to AI written changes, even though those changes carried more redundant code. Meanwhile, volume has exploded. The analytics firm Faros AI looked at teams heavy on AI in 2025. They merged ninety eight per cent more changes, and review time rose ninety one per cent. A tired person skimming a queue is not a control. Layers of checks, fast undo, and a measure that tells you when it is failing, together make a control...
Leonard I'll take the other side, because I think the evidence is thinner than the confidence. Every outcome number in the article is Anthropic's own and unaudited. It went up the same day Anthropic's review product went on general sale. The wider data is less rosy. Google's DORA research in 2024 found that as AI adoption rose, delivery stability fell by an estimated seven point two per cent. Its 2025 report calls AI an amplifier. Strong organisations get stronger, and the rest see their weaknesses magnified. Anthropic may be showing what excellent foundations make possible, not what a typical organisation will get.
Linda That's fair, and the headline number shows why care matters. Anthropic says merged changes to Claude Code more than doubled from June to September, and the share reverted fell by more than forty per cent. A share, not a count. Double the volume, cut the share by forty per cent, and the total is roughly one point two times what it was. So the article does not claim fewer reverts overall. It claims quality per change held up while volume doubled. That is still useful, but nobody outside Anthropic has checked it.
Leonard Arithmetic, the quiet hero.
Leonard My bigger worry is the people left on the high risk tier. In 1983 the psychologist Lisanne Bainbridge wrote a paper called Ironies of Automation. Her point was that when machines take the routine work, people are left with the rare hard interventions, and less practice at them. If two senior people only ever see the scariest changes, are they sharper, or rustier? There is a second loss too. Review used to spread understanding of the code. Anthropic says people now ask Claude to explain the part they are about to change. It admits it has no formal process for this.
Linda I think that is the strongest objection, and I don't think Anthropic has fully answered it. But the alternative is not careful reading of everything. That ended when the volume went up. The real question is where attention goes. Anthropic keeps people on five decisions. Whether to build the thing, risky code like billing, design choices others will copy, reading the evidence, and owning the outcome. That is a better use of a senior person than reading a one line wording fix. The skill problem is real, so it needs its own plan.
Leonard Fair. It does.
Leonard Here's where I think we agree. The control is the rule, not the tool. Anthropic's own documents say its review product posts findings but never approves or blocks a change. Inside Anthropic, an open finding blocks the merge because the team built that gate around the tool. An organisation that buys a review product and calls its review automated may have findings with no enforcement at all. Whatever you buy, you still have to write the rule that says this cannot go ahead until that is resolved.
Linda We agree. Mark the date.
Linda And there is one admission I would underline for any board. Anthropic says it does not yet have circuit breakers that tighten approval when revert or incident numbers move. Its engineers want them before more code merges with no human at all. That is the gap. Loosening a control with a measure is good practice. Loosening it with nothing that pulls it tight again when the measure goes bad is how a sensible shortcut becomes a habit nobody remembers agreeing to.
Leonard So, what is known and what isn't. Confirmed are the rules Anthropic describes, including the tiers and the list that always needs a person. So is the fact that the product itself does not block. Reported by Anthropic alone are the doubling, the revert share and the review improvements. Independent research adds two cautions. Perceived speed is unreliable. In METR's 2025 trial, developers were nineteen per cent slower, yet believed they were faster. And DORA finds stability is still where AI hurts. Not yet known is whether this works outside Anthropic.
Linda And none of this is really about software. Auditors already work this way. The Australian auditing standard asks for more persuasive evidence the higher the assessed risk, and nobody checks every receipt. IT teams pre approve standard changes and send the unusual ones to a board. What AI changes is the volume, and that a machine now does the sorting. Contracts, policies, payments and board papers drafted with AI all need the same thinking, so let's finish with what to do...
Leonard First, find every control that amounts to a person checked it, and ask what it is meant to catch. Second, write risk tiers set by the work, not the author, plus a short list that always needs a person, such as money, access and anything hard to reverse. Third, never let an AI approval fill a human slot, and make the person who directed the AI the owner. Fourth, make an approval lapse when the work changes after it was signed.
Linda Fifth, pick a guardrail measure, like reversals, complaints or rework, and decide in advance what tightens the gate when it moves. Sixth, make undo cheap, with staged roll outs and changes small enough to reverse. And seventh, plan on purpose for the understanding and skill that reading used to build, because that will not happen by itself. If you'd like to know more, sign up for our podcast at agile-insights.com.au.
Leonard Thanks for listening.
Every episode tagged #oversight
All of the briefings (38) · Follow Briefing in Apple Podcasts or any podcast app
Everything from Agile Insights: briefings, Making You Think and the reference series