Briefing
Episode 27 · Primer and discussion · About 15 minutes
Download the MP3 · Linda and Leonard, AI voices, read a script written from the sources below.
Download the MP3 · Linda and Leonard, AI voices, read a script written from the sources below.
Written from the Ai2 paper, blog, code and data, checked on 2 November 2026. The study is new and not yet replicated, and it tested research novelty only, so check the originals before quoting.
Sign up for our podcast at agile-insights.com.au.
Linda Here is the idea for today. An A I judge is a configuration, not an instrument. A new study from A I 2, the Allen Institute for A I, found that one model gives opposite verdicts on the same cases when you change how you ask. So before you trust a judge's score, test the whole set up on the cases it will really judge. By the end you will know what a judge is, what the study found, and three checks to ask for.
Leonard I keep hearing the phrase L L M as a judge. Vendors tell us their outputs are A I reviewed, and our own teams use one model to mark another model's homework. It sounds sensible, because nobody can read ten thousand outputs a day. So before we get to the study, what actually is a judge, and why has everyone started using them?
Linda A judge is a large language model given a grading job instead of a writing job. You hand it something another system produced and ask for a verdict. Pointwise means scoring one thing on its own. Pairwise means showing it two things and asking which is better. Judges scale. A person can review fifty documents in a day, and a judge can review fifty thousand. So they are used for quality checks, routing work, compliance review and testing new versions of a system. The catch is simple. A judge's verdict is not the truth. It is another model's opinion, and it needs testing like any other model.
Leonard Judges all the way down.
Linda Pretty much.
Leonard That is the bit I want to understand. To test a judge, you need cases where you already know the answer. For something as slippery as whether a research idea is original, that sounds close to impossible. I would not know where to start. So how did the researchers get an answer key they could actually trust?
Linda They used peer review. I C L R is a major A I research conference, and its reviews are public. The team mined the 2026 reviews for passages where reviewers explicitly praised or disputed a paper's originality. They took 154 ideas from top rated accepted papers whose originality was praised and never disputed. They took 145 from low rated rejected papers with originality concerns. A second set paired those 154 strong human ideas with ideas from a simple A I generator, given no search and no tools.
Leonard Then they ran six judges over those pairs. Three were from Open A I, G P T five point one, five point two and five point four. Three were from Anthropic, Claude Sonnet four point five, Opus four point five and Opus four point six. They changed one thing at a time: the prompt, the reasoning budget, the formatting, and whether the judge could see related papers. As a newcomer, I would have guessed the model matters most...
Linda The prompt mattered most. The researchers showed G P T five point four the same pairs, each with one strong human idea and one generated idea, under two prompts. The first told the model that reviewers had found exactly one idea in each pair novel, and asked which. The second simply asked which idea is novel. Under the first, it picked correctly 93 and a half per cent of the time. Under the second, 40 point 9 per cent. A coin toss gets fifty. So it fell below guessing, and it changed its verdict on more than half of the identical pairs.
Leonard Oh, that is not good.
Linda It is not.
Leonard That is a big swing, though the first prompt does hand the model a useful clue, that one of the two is the winner. So is the lesson just to write better prompts and pick the best one? If it is, that is a familiar problem. Most teams already spend time tuning their prompts before anything goes live.
Linda That is the common misunderstanding, and the study rules it out. No prompt was best for every model. The same change helped one judge and hurt another. Judges also looked steadier on human writing than on A I writing. On the human only pairs, that change moved G P T five point four by about seven points. With a generated idea in the pair, it moved it by more than fifty. More machinery did not fix it either. Extra reasoning and related papers helped little, and two purpose built novelty checkers did worse than a cheap, plain prompt.
Leonard This study is about research ideas, though. Our listeners run judges over customer emails, policy documents and agent logs. I can imagine a board member asking how far a study about academic novelty really reaches. I would want a straight answer rather than a stretch. So what should a governance lead take from it?
Linda Be clear about the stretch. The study tested novelty only, and not the new purpose built judgment models. So the wider lesson is our inference, to be tested, not assumed. But the method carries. The model, prompt, format, reasoning budget and any retrieval are all part of the judge. Change one and you have a different judge. Its record on human work says little about A I work. And the authors caution against using current novelty judges in consequential settings, especially in place of human peer review.
Leonard That changes how I would read a vendor claim that something is A I reviewed. The useful question is not which model they used. It is how they tested the judge, on what kind of cases, and what happens to its verdicts when they change the set up. Which brings us to what an organisation can do about its own judges...
Linda Three checks. First, write the judgment as one fixed question, with each possible answer described in words. Label a set of your own cases, including A I generated ones if the judge will see them. Second, run two or three reasonable prompts or models over that set. How often they disagree is your instability, and it needs no labels at all. Third, treat any change to the model, prompt or settings as a change to the control. Rerun the set, and keep a person on the calls that matter.
Leonard In Part 2 we argue about what this means for the judges you are already running. Thanks for listening.
Linda Here is the idea for this discussion. An A I judge is a configuration, not an instrument, so the real question is not whether to use one, but whether you have tested the set up you are actually running. By the end you will know the strongest case against relying on A I judges, the strongest case for them, where the evidence settles, and what an organisation should ask of every judge it runs.
Leonard A quick recap for anyone who skipped Part 1. A judge is a large language model asked to grade or compare outputs instead of writing them. A study from A I 2, the Allen Institute for A I, tested six judges from Open A I and Anthropic on telling more original research ideas from less original ones. The answer key came from public peer reviews. The headline result: telling G P T five point four that one idea in each pair had been judged novel, versus simply asking which is novel, moved its accuracy from 93 and a half per cent to 40 point 9. That is below a coin toss.
Linda So here is the question. Organisations are already using judges as quality gates on generated documents, to route work, to review compliance, and to test new versions of agents before release. Some vendors sell outputs as A I reviewed. If one model's verdict can flip on a change of framing, should organisations stop relying on A I judges for anything that matters, or should they keep using them and run them better? I want to give each answer real room before we settle it.
Leonard I will take the sceptic's side. A judge used as an assurance gate is a yes or no control. If a reasonable change of wording swings it by fifty points, that control is not doing what the risk register says it does. The study found more than that. Depending on the set up, even Claude Opus four point six gave an undecided verdict on about a third of the pairs, and how you count those ties can change which system looks best. The authors themselves caution against using current novelty judges in consequential settings. So my position is that judges can triage and sort, but they should never sign anything off...
Linda That is a fair warning, but the same study also contains the case for judges. The best result came from the plainest set up. Claude Opus four point six, with low reasoning effort and a majority vote across three calls, scored zero point eight five on a combined accuracy measure where one is perfect. It cost two dollars and fifty nine cents for the whole evaluation set. The problem the study exposes is not the idea of a judge. It is the untested judge. A configuration you have measured on your own cases is a usable instrument for that configuration, and only that one.
Leonard Measured on which cases, though? That is where I part company with you. On pairs of human ideas, the prompt change moved G P T five point four by about seven points. With an A I generated idea in the pair, it moved it by more than fifty. A team that validated its judge on last year's human written documents would have signed it off. Then the work it reads becomes mostly A I generated, and the judge they tested is no longer the judge they have. Nothing in the dashboard would tell them.
Linda I accept that, and it sharpens my position rather than ending it. The test set has to look like the traffic, including the A I generated part, and it has to be refreshed as the traffic changes. That makes judge testing a standing duty, like any other monitoring, not a one off sign off at go live. So we land in the same place. Judges for scale and sorting, with a tested and retested configuration, and a person on the calls that carry real consequences.
Leonard We agree, then.
Linda Mostly.
Leonard There is a third answer people reach for, which is to buy a bigger judge. Give it more reasoning, let it search the literature, or use a purpose built evaluator. The study tested that too. An agentic judge running G P T five point six Sol, with extra high reasoning and its own search, scored eleven points below Claude Opus four point six in the standard set up, at about ten times the cost per decision. Two dedicated novelty checkers also scored below the cheapest plain prompt, and cost thirty dollars and eighty two dollars for the set, against two dollars fifty nine. Expensive is not the same as tested.
Linda And there is one use where this bites hardest, which is choosing between systems. Teams often run a bake off, where a judge compares outputs from two models or two vendors and counts the wins. The study did exactly that with two pairs of idea generators. Changing the judge's prompt alone flipped a quarter to a third of the verdicts on average, and moved the share of wins for the stronger system by up to 29 points. No answer key is needed to see that. So a claim that one product wins most head to heads, judged by A I, means little until you know how sensitive that judge was.
Leonard I have sat through those.
Linda We all have.
Leonard So where does the evidence settle? The instability is solid. When two reasonable set ups give different verdicts on the same pair, at most one of them can be right, whatever the true answer is. That finding does not depend on the labels. What is not settled is just as important. The study tested novelty of research ideas, nothing else. It is about a week old and nobody has replicated it yet. The generated ideas carry assumed labels, although a computer science professor checked thirty of the hardest pairs and agreed every time. And it did not test the new purpose built judgment models...
Linda That last gap matters for the judgment models now on sale, which return a probability for one fixed question rather than a written opinion. That design is closer to the discipline this study points towards. Whether it removes the instability is not known, and the study gives no evidence either way. So my reading, and it is an inference, is that they should be tested exactly the same way: on your own cases, with the disagreement across set ups measured, before the score drives anything a customer or a regulator would see.
Leonard So, what to do. Start by writing down every place a judge already makes a call in your organisation, including the ones inside vendor products. For each one, name an owner, and record the full configuration: the model, the prompt, the format, the reasoning setting and any retrieval. Then build a labelled set of your own cases, with A I generated content in the same proportion the judge will see in practice. Run two or three reasonable prompts or models over it, and record how often they disagree. That disagreement rate is the number to report.
Linda Then treat any change to that configuration as a change to the control, and rerun the set before it goes live. Keep a person on borderline and consequential calls, and say in writing which calls those are. When a vendor says its output is A I reviewed, ask how the judge was tested, on what kind of content, and how much its verdicts move when the set up changes. If you'd like to know more, sign up for our podcast at agile-insights.com.au.
Leonard Thanks for listening.
Every episode tagged #oversight
All of the briefings (38) · Follow Briefing in Apple Podcasts or any podcast app
Everything from Agile Insights: briefings, Making You Think and the reference series