Agile InsightsBriefing
Two identical blank résumés side by side on a dark workbench, a brass balance scale tipped slightly by a single folded note, and a studio microphone with headphones, labelled Swap, Pressure, Define fair and Episode 13.

Episode 13 · Primer and discussion · About 14 minutes in two parts

Fairness under pressure: a fair AI decision has to hold when someone leans on it

New research from Ai2 shows AI models that pass the usual name-swap test can still give way unevenly under one sentence of pressure. What that means for any organisation using AI to choose between people.

#people#oversight

Part 1: Primer · About 6 minutes

Download the MP3 · Linda and Leonard, AI voices, read a script written from the sources below.

Part 2: What a pressure test changes · About 8 minutes

Download the MP3 · Linda and Leonard, AI voices, read a script written from the sources below.

In this episode

01A swap test checks the answerThe classic fairness test changes only a person's name or label and checks the choice stays the same. It is a good start, but it only checks the final answer, not how firmly it is held.
02One sentence of pressureAi2's Fair Stress benchmark adds a single sentence arguing for one candidate, with a mirror for the other. Across fifteen models, accuracy differed by about three points, but correct answers gave way over nine points more often one way.
03Fair needs a definition firstAi2 measures equality: the same criteria, whoever benefits. An equity view could judge some results differently. A deliberate special measure is a written choice; a lean that appears under pressure is behaviour nobody chose.
04The explanation is not evidenceEven when models chose correctly for both candidates, their explanations differed: identity words for minority candidates, evidence and criteria for majority ones. Ai2 argues this over correction does not serve minorities either.
05Test under pushback, not just swapsDecide and sign off what fair means for each decision. Add mirrored pressure sentences to your tests, test the way you deploy, ask vendors how they test, and rerun the test whenever the model changes.
Sources and links (6)
Asymmetrically Unfair Decision Making and Reasoning in LLMs · Read the report (PDF)
The Ai2 tech report behind Fair Stress (Zahraei, Brahman, Hakkani-Tür, Tur and Sap, October 2026), including its limitations and its statement on equality and equity.
FairStress: a deeper evaluation of fairness in LLM decision-making · Read on Hugging Face
Ai2's announcement on the Hugging Face blog, 20 October 2026.
The FairStress dataset · View the dataset
The full benchmark and the smaller FairStress-Core set, open under a CC BY 4.0 licence.
Counterfactual fairness · Read on arXiv
Kusner, Loftus, Russell and Silva (2017): the formal idea behind the swap test.
Are you sure? The FlipFlop experiment · Read on arXiv
Laban and colleagues (2023) on how often models change their answer when challenged.
Guidance for AI Adoption: implementation guidance · Read on ai.gov.au
National AI Centre, October 2025. Practice 5, Test and monitor, covers acceptance criteria, pre-deployment testing and asking providers for their test methods.

Written from public sources and checked on 21 October 2026. The Ai2 tech report was read in full in its main text and in part in its appendices; it is new and not yet peer reviewed, and the findings are Ai2's, on the models and definition of fairness it chose. The two earlier papers were read in summary, and Practice 5 of the Australian guidance in full. Our remarks on Australian anti-discrimination law are our reading of the Acts' special measures provisions, not legal advice. Please check the originals before quoting.

Our fact check · 18 claims: 17 confirmed, 1 partly right

Before we record, every claim the hosts make is checked against the sources (this table was last updated 21 October 2026). Confirmed means a primary source or two independent reports; Reported means one party says so; Partly right, Disputed, Unverified and Not yet known mean what they say, and the hosts say so on air. This is the table as we left it.

  1. Confirmed300 decisions, 29 domains, four identity levels, 78 prefixes in 15 families with mirrors, 13,032,104 items, Core 50,012Tech report §2, App. A.4; dataset card; Report; Scenarios written by gpt-4o-mini, checked by a gpt-4o judge and humans
  2. ConfirmedWith explicit identity, accuracy gap 2.9 points across 15 modelsTech report abstract, App. C Table 17; Report; Peaks +3.5 when identity is only hinted
  3. ConfirmedOne sentence of pressure: correct answers favouring the majority-coded candidate abandoned 9.3 points more often (6.9 when identity is hinted)Tech report abstract, Table 17; Report; Positive for all 15 models
  4. ConfirmedTies moved toward the minority-coded candidate 23.9 points more often under pressure; 61.6% minority-coded picks on ties with no pressureTech report Table 17; Report; Explicit identity (D3)
  5. ConfirmedFrontier models: 60–65% minority-coded on ties; tie-break shift +8.8 to +42.4 across 12 runsTech report App. C.5 Table 31; Report; Ties only. Core subset, pooled; report says cells too small to compare models family by family, so no ranking
  6. ConfirmedStrongest pressures: invented evidence (+16.0) and historical guilt (+15.5); authority (+5.8) and consensus (+5.2) among the weakestTech report Table 3; Report; Robustness gap at D3
  7. ConfirmedExplanations: “resilience” 3.3×, “diversity” 3.0×, “barriers” 1.7× for minority-coded choices; “evidence”, “criteria” 1.1–1.2× for majority-codedTech report §3.3; Report; Weak signal: text predicts the chosen side at 50.3–58.6% (chance 50%)
  8. ConfirmedBBQ direction flips with scoring method (probabilities vs generated chat answers)Tech report §3.1, App. D.1; Report; Four models
  9. ConfirmedFrontier system cards rely on BBQ for demographic biasTech report App. D.2; Report; Five cards audited
  10. ConfirmedMost of nine debiasing methods failed; one steering method dropped clear-cut accuracy 98% to 47%Tech report Table 4; Report; CAA on Qwen2.5-7B
  11. Partly rightCounterfactual chain-of-thought narrowed the robustness gap to +0.3 to +2.0Tech report §3.2, F.5; As often summarised; No FairStress accuracy cost, but MMLU −8.2 on OLMo-3; tie lean still 54–57%; three small open models
  12. ConfirmedAi2 measures equality; an equity view could treat part of the lean as intendedBlog; tech report ethics statement; Said early in both scripts
  13. ConfirmedOver-correction does not serve minorities; no claim of a deliberate anti-majority policyTech report §5; Report
  14. ConfirmedLimits: short synthetic cases, two candidates, one-sentence prefix, no multi-turn, US-centric group labelsTech report Limitations, ethics statement; Report
  15. ConfirmedSwap (counterfactual) testing is the established method; DiscrimEval (2023) already found positive and negative discrimination in Claude 2.0Kusner et al. 2017; Tamkin et al. 2023; Abstracts; So the direction is not wholly new; the pressure test is
  16. Confirmed“Are you sure?” flips answers 46% of the time, accuracy down 17% on averageLaban et al. 2023; Abstract; Sycophancy background; ep8 callback
  17. ConfirmedAI6 Practice 5: acceptance criteria reflecting risk, documented pre-deployment tests, deployment authorisation, request the provider’s test methods (5.1.4)NAIC Guidance for AI Adoption, implementation guidance, Oct 2025, pp. 16–17
  18. ConfirmedCommonwealth anti-discrimination Acts each allow special measures (SDA s 7D, RDA s 8, DDA s 45, ADA s 33)Federal Register of Legislation; Section headings only; Script says the law “makes room for special measures”, attributed as the hosts’ reading

Want to know more?

Sign up for our podcast at agile-insights.com.au.

Read the transcript · Part 1: Primer

Linda G'day, and welcome to the Agile Insights briefing. I'm Linda. Here is the main idea. A fair AI decision is not just the same answer when you swap the names. It is an answer that holds equally firmly, whoever it favours, when someone leans on it. In this first part we will build that idea up in four steps. By the end you will know what a swap test checks and what it misses. You will also know why the word fair needs a decision from you.

Leonard I'm Leonard. Let me ask what plenty of listeners will be thinking. Our team tested an AI screening tool last year. We gave it the same résumé under two different names, and it picked the same person both times. Isn't that what a fairness test is?

Linda It is the classic one, and it is a good start. Researchers call it a counterfactual test. The idea, set out formally in 2017, is simple. A decision is fair to a person if it would be the same in a world where they belonged to a different group. So you hold the qualifications still and change only the name or the label. If the answer moves, you have found a problem. Most fairness benchmarks for language models work this way. They check the answer, and only the answer. That is the first step. The swap test checks the final choice.

Leonard Only the final choice. Right.

Linda The second step is the new work. On the twentieth of October, the Allen Institute for AI released a benchmark called Fair Stress. The institute is known as A I two, and it published a tech report alongside. It has three hundred decisions between two people, across twenty nine areas such as hiring, health care, housing and lending. Each one comes as a clear case, where one candidate is stronger, and as a genuine tie. A I two is open about its yardstick. It measures equality: the same criteria, whoever benefits. It says an equity view, which weighs past disadvantage, could judge some results differently. Keep that in mind... because we come back to it.

Leonard So far that sounds like a bigger swap test. Where does the pressure in the title come in? I'm picturing something like a manager's note in the file.

Linda That is almost exactly it. After the two profiles, the test adds one sentence of pressure. A senior figure has privately endorsed candidate B. A reference letter says something glowing. This group has been overlooked before. Each sentence has a mirror that argues for the other candidate. Then A I two asks whether the model drops a correct decision equally often in both directions. You may remember episode eight, the agreeable machine, about models drifting toward whatever the user pushes...

Leonard The agreeable machine. I do.

Linda Earlier research found that just asking a model whether it is sure made it change its answer almost half the time. This is the same weakness, measured by who it helps.

Linda Step three is what A I two found. Across fifteen models, the swap test looks nearly fine. With identity stated outright, accuracy differed between groups by only two point nine points. Then add the one sentence of pressure. A correct answer favouring the majority candidate was abandoned nine point three points more often. That is compared with one favouring the minority candidate. On genuine ties, the pressure moved choices toward the minority candidate twenty three point nine points more often. So the direction here was over correction, toward minority candidates. A I two argues plainly that this does not serve minorities either, and we will see why shortly.

Leonard Then isn't the obvious fix to look at the model's explanation? Most of these tools write one. If it is leaning, surely the reasoning would show it, and a reviewer would catch it.

Linda That is the common misunderstanding, and it is step four. A I two looked at cases the model got right for both candidates. The explanations still differed. When the model chose a minority candidate, it reached more often for words like resilience, diversity and barriers. When it chose a majority candidate, it used words like evidence and criteria. The difference is small, but it was consistent across eleven models. That is why A I two says over correction does not serve minorities. A strong candidate gets credited for their background instead of their record. And the reasoning you can read did not reveal much of what drove the choice.

Leonard Huh. Same answer, different story.

Leonard Which brings us back to the yardstick you mentioned earlier. Some organisations actually want a preference in certain decisions. If that is the policy, is the lean always a problem?

Linda That is exactly why fair needs a decision. A I two measures equality and says so. Under an equity view, part of what it found could be intended. As we read it, Australian anti discrimination law protects everyone on those grounds, and also makes room for deliberate special measures. A special measure is a choice someone made and wrote down. A lean that appears under pressure is behaviour nobody chose. So for governance, the lesson is this. Decide which definition you are testing against before you test. Then check that a correct decision holds under pushback, not just under a swap. In part two, Leonard and I look at what the evidence settles, what it does not, and what to do.

Leonard See you in part two.

Read the transcript · Part 2: What a pressure test changes

Leonard Welcome back to the Agile Insights briefing. I'm Leonard. The main idea again. A fair AI decision is not just the same answer when you swap the names. It is an answer that holds equally firmly, whoever it favours, when someone leans on it. In this part we look at what that means for an organisation that already uses AI to choose between people. By the end you will know what to ask your vendor and what to add to your own tests.

Linda And I'm Linda. A quick recap so this part stands alone. A swap test changes only a person's name or label and checks that the answer stays the same. A pressure test adds one sentence arguing for one candidate, with a mirror sentence arguing for the other. Then it checks whether a correct decision gives way equally often in both directions. A I two, the research institute behind the Fair Stress benchmark, calls that symmetric fairness. It is an equality yardstick, and A I two says an equity view could judge differently.

Leonard So here is the question for this part. If your screening tool passed a swap test, what can you actually conclude? My answer is narrower than most people would like. You can conclude the first answer is stable. You cannot conclude the decision is stable. Across fifteen models, A I two found the accuracy gap between groups was about three points. With identity stated outright and one sentence of pressure, the gap in abandoning a correct answer was over nine points. The swap test was measuring the calm moment... and real decisions are rarely made in one.

Linda They rarely are.

Linda And this is not only small open models. A I two also tested six frontier models on a smaller core set. From Anthropic, they were Claude Opus five and Claude Sonnet five. From OpenAI, they were G P T four o, G P T four o mini, G P T five mini and G P T five point five. Each ran in two modes. On genuine ties, they picked the minority candidate sixty to sixty five per cent of the time. Under pressure, ties moved toward the minority candidate in all twelve runs. A I two says the samples are too small to compare these models one by one, so we will not rank them.

Leonard The part I found most useful is which sentences did the moving. You might expect authority to win, the note from a senior leader. In A I two's tests it was among the weaker levers. Invented evidence was the strongest at breaking a correct answer, such as a glowing reference letter. Close behind were claims that a group had been overlooked before. Now think about a real case file. It is full of references, complaints, notes from managers and requests to reconsider...

Linda All of it free text.

Leonard Any decision tool that reads free text alongside the criteria is reading exactly this kind of pressure, every day.

Linda Which leads to the second finding I would put in front of a board. How you test decides what you find. A I two took a widely used bias benchmark called B B Q. Scored the traditional way, on the probability of fixed answers, all four models it checked looked biased toward stereotypes. Asked the way people actually use them, generating an answer in a chat, three of the four flipped to favouring minority candidates. The fourth came out roughly neutral. A I two also reports that frontier developers' system cards rely on that benchmark for demographic bias. So a vendor's fairness score may not describe the behaviour you are about to deploy.

Leonard That one stings.

Leonard Let me push back a little, though. If the models lean, surely the answer is to fix the model. There are plenty of debiasing techniques. Why should an organisation carry this, rather than waiting for the vendors to sort it out?

Linda Because the fixes A I two tried mostly did not work. It tested nine published methods on three small open models. Most did not reliably reduce the gap. One steering method made the choices look balanced by making the model stop reading the cases. Its accuracy on clear cases fell from ninety eight per cent to forty seven. The best result came from asking the model to reconsider with the identities swapped. That narrowed the gap a lot, but it cost one model over eight points on a general knowledge test. So mitigation helps, but only if you test it on your own decisions.

Leonard Fair enough, I'll take that. My second challenge is about the yardstick. Some organisations want to favour certain candidates, deliberately and lawfully. Isn't a minority lean exactly what they are after? Why call it a problem at all?

Linda I'd separate a policy from a behaviour. As we read it, Australian law makes room for special measures. A special measure is a choice an organisation makes, writes down and can defend. What A I two measured is different. It is a lean that appears when someone pushes, and nobody chose it. A I two also argues it does not serve minorities, because strong candidates were credited for their background rather than their record. So the answer is not to pick a side in a culture war...

Leonard Thank goodness.

Linda It is to decide your definition, in writing, and then test against it.

Leonard Let me be clear about what is and is not known. This is A I two's work, released on the twentieth of October, and it is not yet peer reviewed. The cases are short and synthetic, with two candidates and one sentence of pressure. Longer files, several turns of argument and a company's own instructions to the model were not tested. The minority and majority labels follow an American convention. And A I two's explanation for why models lean is a hypothesis about training data, not a finding. What is solid is the method, and the gap it measured.

Linda So here is what to do. First, decide what fair means for each decision about people, equality or a deliberate measure, and have the accountable owner sign it. Second, test under pushback, not just swaps. Add mirrored pressure sentences to your test set: authority, invented evidence, past disadvantage, emotional appeal. Check that a correct decision holds equally whichever way the push points. Third, test the way you deploy, with generated answers through your own prompts. Australia's guidance for AI adoption already asks for acceptance criteria that reflect the risk, and documented tests before deployment.

Leonard Then four habits. Ask your vendor how it tests under pressure, and whether it scores generated answers. The same guidance asks you to request your provider's test methods and results. Keep free text out of the decision where structured criteria will do, because that text is where the pressure arrives. Don't treat the model's explanation as your audit trail. Reviewers should check the inputs and the criteria, not the prose. And rerun the pressure test whenever the model changes, because these results shifted from one model and training stage to the next.

Linda If you'd like to know more, sign up for our podcast at agile-insights.com.au.

Leonard Thanks for listening.

More on #people

Episode 32 · Primer and discussionAgentic video is real: what to try, and what to checkEpisode 31 · Primer and discussionSpeeding up planning decisions with AI: who checks the checker?Episode 34 · Primer and discussionWhen the test stops telling you anything: hunger beats tenureMaking You Think 12 · Primer and discussionThe Game Changer, by Jason FoxMaking You Think 9 · Primer and discussionArtificial Intimacy, by Sherry TurkleMaking You Think 8 · Primer and discussionWhat If We Got AI Right?, by Eleanor Drage

Every episode tagged #people

All of the briefings (38) · Follow Briefing in Apple Podcasts or any podcast app

Everything from Agile Insights: briefings, Making You Think and the reference series