Briefing
Episode 36 · Primer and discussion · About 15 minutes in two parts
Download the MP3 · Linda and Leonard, AI voices, read a script written from the sources below.
Download the MP3 · Linda and Leonard, AI voices, read a script written from the sources below.
Checked against Microsoft’s blog and documentation and the Latent Space transcript on 10 November 2026. Microsoft’s loop is the vendor’s own description and its optimiser is in preview; Legora’s methods are its CTO’s account and have not been audited. Check the original before quoting.
Before we record, every claim the hosts make is checked against the sources (this table was last updated 4 November 2026). Confirmed means a primary source or two independent reports; Reported means one party says so; Partly right, Disputed, Unverified and Not yet known mean what they say, and the hosts say so on air. This is the table as we left it.
Sign up for our podcast at agile-insights.com.au.
Linda Here is the idea for today. An A I agent in production can get worse without anyone touching it. So checking it is not something you do once before launch. It is a loop that keeps running. Measure it, diagnose what moved, test changes on cases it has never seen, and release slowly. By the end, you will know the four parts of that loop. You will also know the one question to ask when someone shows you a falling score.
Leonard Some context first. On the sixth of November, Microsoft published a how to guide on its Foundry blog. It covers closed loop evaluation for agents. Foundry is Microsoft's platform for building and hosting agents. The guide opens with a useful claim. A production agent can lose accuracy, or cost too much, without a single code change. New kinds of requests expose gaps in its instructions. New models arrive that might be better or cheaper. It is a vendor's guide, so the product details are Microsoft's own description.
Linda Let me build the loop up in four steps. Step one is to measure the live agent all the time. Microsoft calls this continuous evaluation. A sample of real conversations is scored as they happen. Does the reply rest on what the tools actually returned? Did the agent follow its own rules, and pick the right tool? Did it obey instructions hidden in data it should only have read? Each conversation leaves a trace: the request, every tool call, and the reply. Those scores become the baseline for every later change.
Leonard So the score drops one morning. Do we go straight in and fix the prompt?
Linda Not yet, and this is the step I would underline. Step two is to diagnose before you tune. The guide lists three things to rule out first. Is a tool or a back end system failing? Has the mix of requests changed, with a surge in something new? Or has the grader changed? The grader is whatever does the scoring, often another model working from a written rubric. Microsoft's point is that a changed grader shifts scores even when the agent behaves exactly as before. Sometimes the ruler moved, not the agent.
Leonard That is the one question from the top of the show. When someone shows you a falling score, ask whether the measuring stick changed. Say the rubric was rewritten last month. A chart for the quarter can then show a decline that never happened in the agent. Or an improvement that never happened. So the grader needs its own version number, the same as the agent does. Otherwise the board is reading a story about the ruler.
Linda A story about the ruler. I like that.
Leonard I have read a few.
Linda Step three turns real conversations into a fair test. Foundry can sample recent traces into a varied test set, and it redacts private content by default. Then a person reviews each case and writes down the right business outcome. That is called the ground truth. Each case is replayed against a sandbox, with stand in tools and a test ledger. Never live payment systems or customer records. Then the cases are split. Some become practice material for whatever is improving the agent. The rest are held back as the exam.
Leonard Why keep the exam separate? Isn't more test data always better?
Linda Because anything that tunes against a test will eventually learn the test. Microsoft has a tool for the tuning, called Agent Optimizer, which is still in preview. It rewrites the agent's instructions, skills and tool descriptions, and tries other approved models. The guide keeps the release exam where that tool cannot see it. The business rules also sit outside what the optimiser can edit. A plain piece of code checks, for example, that a refund was never submitted twice.
Leonard The guide's examples make this concrete. It uses a refund agent with three failures. In the first, a refund request times out. The agent sends it again before checking whether the first one went through. In the second, it tells the customer the refund is complete while it is still pending. In the third, it obeys text hidden inside a customer's return note. Each of those can read perfectly well in a transcript. You only catch them by checking what the agent did, not just what it said.
Linda Confident and wrong.
Leonard The worst kind.
Linda Step four is to release slowly, with a way back. An engineer reviews the improved version first. An automated gate then runs the old and new versions on the held back exam, and blocks any shortfall. Then comes a canary. In the guide's example, five per cent of new sessions go to the new version, then twenty five per cent. Each stage is compared with the old version running alongside. And the rollback plan covers more than switching back. Refunds already submitted need to be reversed safely too.
Leonard The part people forget.
Linda Every time.
Leonard So the common mistake is thinking evaluation is a test you pass before launch. Testing before launch covers the cases you thought of, on the model you had. It was graded the way you graded it then. All three of those change. The other mistake is thinking monitoring means someone reads the logs. Nobody can read thousands of conversations. And in many organisations, nobody is allowed to. That is Part 2: running this loop when you cannot read a single trace...
Linda Exactly. For anyone governing A I, an agent's instructions, skills and model are production settings. Changing them is a release. Reviewed, tested against evidence it did not train on, staged, and reversible. Three things to do. First, write down the handful of ways this agent is most likely to fail, and measure those in production. Second, keep a held back test set the tuning cannot touch. Version your graders. Third, give every change a canary, and a rollback that covers the business side effects.
Leonard That's Part 1. In Part 2, we look at what to do when you are not allowed to read the traces at all. Stay with us.
Linda If you'd like to know more, sign up for our podcast at agile-insights.com.au.
Leonard Thanks for listening.
Linda Here is the idea for today. Being unable to read what your agent does is no excuse for not checking it. Privacy changes what feeds the checking loop. It does not change whether the loop runs. By the end, you will know three ways one company keeps checking an agent it cannot watch. You will also know where those methods are strong, and where they are still a guess.
Leonard A quick recap for anyone joining here. A trace is the record of one agent conversation: the request, every tool call, and the reply. An evaluation scores that behaviour, and a grader does the scoring. The loop from Part 1 runs in four steps. Measure live traffic. Diagnose before you tune. Test changes on a held back set the tuning never sees. Then release by stages, with a way back. Microsoft's version of that loop starts from production traces, with private content redacted by default.
Linda So here is the question. What if you cannot use the traces at all? That is the everyday position for a lot of organisations. Privacy law, client confidentiality or a works agreement can stop anyone reading the conversations. Our example comes from the Latent Space podcast on the second of November. The guest was Jacob Lauritzen, chief technology officer of Legora, which builds A I tools for lawyers. He says Legora cannot read any of its traces. Almost everything lawyers upload is confidential, and much of it belongs to the firm's clients, not the firm...
Leonard Let me give the first approach some room, because it is the most interesting. If you cannot watch the real work, you rebuild it. Lauritzen says three engineers and three lawyers build simulated environments of what customers do. One is a data room of ten thousand files. The agent runs a due diligence review, and has to produce a report that catches the right red flags. They plot the results as a heat map, by area of law and type of task. His point is that a new model often gets better at some kinds of legal work, and worse at others.
Linda That is a lot of reading.
Leonard For the agent, anyway.
Linda And that heat map matters beyond law. Most organisations treat a model upgrade as a single yes or no. Better or worse. The heat map says the honest answer is usually both, depending on the task. That is a strong argument for testing every model change by type of work. And for recording which model handles which work. The second approach is how they watch production. Lauritzen says they sort failures into ten to twenty known failure types, tagged by task and area of law. Cheap model based checks do the sorting, behind feature flags. So nobody reads the work, but the counts by failure type still move.
Leonard Which gives them a signal when they roll something out. He describes comparing the old and new feature side by side, to see whether it improved or badly declined. The third approach is the human one. When a client is engaged and something is not working, Legora asks them to anonymise the documents and the problem. If the team can recreate it, they can fix it. And on turning real conversations into synthetic ones, he describes a range of risk. Changing the names alone may not be enough. They stay at the cautious end.
Linda Now, a fair objection. Microsoft redacts private content from trace test sets by default. So why not just use the traces, redacted, and skip all this rebuilding?
Leonard I will argue that side, because I think it is half right. Redaction is real and it is on by default. Microsoft's documentation also says to keep personal data out of telemetry in the first place. And to treat traces like any other production log, with the same access and retention rules. Say the agent works on your own organisation's data, with a lawful basis and good controls. Then redacted traces are the most representative test material you will ever get. Rebuilding everything from scratch would be throwing away the best evidence you have.
Linda Half right is generous.
Leonard I know.
Linda I agree for your own data, and disagree for someone else's. Redaction answers the question of whether a name is visible. It does not answer whose permission you need to use the material at all. Much of Legora's material belongs to its customers' clients. Removing the names would not change who owns the facts of a client's merger. So the decision is not technical first. It is about permission. Where the permission is yours, use redacted traces. Where it is not, the simulated work and the failure counts are the loop.
Leonard That is fair, and it settles it for me. The test is permission, then redaction, in that order. So the practical design for most organisations is both. Use redacted traces where the data and the permission are yours. Use simulated work, failure types and consented examples where they are not. Microsoft's own documentation says trace data and synthetic test data are complementary, rather than alternatives. Which brings us to where the evidence stops.
Linda Here is what is confirmed, and what is not. Microsoft's loop is described in its own guide and its product documentation. The optimiser at the centre of it is still in preview, and needs your subscription on an allow list. Legora's methods are what its chief technology officer said on a podcast. They are his description of his own company, and nobody has audited them. We found no published data showing that either approach reduces failures in practice. So treat both as designs worth copying, not as proof.
Leonard There is also a limit built into simulation. A simulated data room is your best guess at what customers do. If customers start doing something new, the simulation will not know until someone notices. The failure types help here, because a rise in failures the list did not expect is itself a signal. But it means the list of failure types needs an owner, and a review date. Otherwise it quietly becomes the list of failures you expected two years ago.
Linda Lists get old.
Leonard They do.
Linda So what does this mean for governing A I? Three things. First, the inability to read traces is the normal case in regulated work, not an edge case. Plan the evaluation design before go live, not after the first complaint. Second, a fixed list of failure types, counted in production, is evidence a reviewer can actually use. Having someone read the logs is not a control, and often it is not even lawful. Third, the grader and the failure list are part of the control, so they need owners and versions too.
Leonard And what to do on Monday. Decide, for each agent, whether you are allowed to use its traces, and on what terms. Write down the ten or twenty ways it is most likely to fail, and count those in production. Where you cannot use the traces, budget for simulated test work, built with people who know the domain. Agree a route for customers to share anonymised failures. And keep a held back test set that nothing tunes against, whichever way the test cases are made.
Linda If you'd like to know more, sign up for our podcast at agile-insights.com.au.
Leonard Thanks for listening.
All of the briefings (38) · Follow Briefing in Apple Podcasts or any podcast app
Everything from Agile Insights: briefings, Making You Think and the reference series