Agile InsightsBriefing
A black and white workbench with a stack of papers under a glass paperweight, an open card-catalogue drawer and a magnifying glass on an index card, labelled in yellow Raw Sources, The Wiki and Lint, beside a studio microphone.

Episode 28 · Primer and discussion · About 15 minutes in two parts

When an AI keeps the wiki, checking has to keep pace with writing.

Andrej Karpathy’s L L M wiki has an agent compile your sources into linked pages and keep them current. What the pattern is, why errors compound as well as insight, and four controls for a knowledge base your agents write for themselves.

#data#oversight

Part 1: Primer · About 7 minutes

Download the MP3 · Linda and Leonard, AI voices, read a script written from the sources below.

Part 2: What compounds, and who checks · About 8 minutes

Download the MP3 · Linda and Leonard, AI voices, read a script written from the sources below.

In this episode

01Compiled once, kept currentKarpathy’s gist of 4 April 2026 says RAG rediscovers knowledge on every question. His alternative: raw sources that never change, a wiki the model owns, and a schema file holding the rules.
02Ingest, query and lintOne source may touch ten to fifteen pages, good answers are filed back as pages, and lint periodically checks for contradictions, stale claims, orphan pages and gaps.
03Near zero is not freeKarpathy says models make maintenance cost near zero. The writing is cheap, but someone still pays for tokens and still has to check the work, as he does himself.
04His safeguard is a humanHe prefers to ingest one source at a time and read each update; batch ingest with less supervision is an option. Our reasoning, with no study found: unattended ingest lets errors compound.
05What to doGive every claim a source and a confidence state, run lint as often as ingest with a failure that stops the change, review filed-back answers, keep history in version control, and register the wiki.
Sources and links (4)
LLM Wiki · Read the gist
Andrej Karpathy’s idea file, published as a GitHub gist on 4 April 2026: the three layers, ingest, query and lint, the index and the log.
qmd · Read on GitHub
The local Markdown search engine Karpathy names for when the index is no longer enough: keyword and vector search with model re-ranking, on the device.
As We May Think · Read the essay
Vannevar Bush’s essay in The Atlantic, July 1945, describing the memex and its trails of association.
The AI Daily Brief, 17 September 2026 · Listen to the episode
The edition relaying OpenAI’s first misalignment disclosures, including the model that wrote instructions into its own compaction summary.

Written from Karpathy’s gist and checked against its text on 6 November 2026. His figures are his own experience, not benchmarks. What the hosts say about errors compounding is their reasoning: no study measuring error rates in an LLM-maintained wiki was found. Check the original before quoting.

Our fact check · 10 claims: 5 confirmed, 2 reported, 1 unverified

Before we record, every claim the hosts make is checked against the sources (this table was last updated 6 November 2026). Confirmed means a primary source or two independent reports; Reported means one party says so; Partly right, Disputed, Unverified and Not yet known mean what they say, and the hosts say so on air. This is the table as we left it.

  1. ConfirmedGist published 4 Apr 2026, one revisionRevisions page
  2. ConfirmedThree layers, three operations, two filesGist
  3. ConfirmedOne source at a time by preference; batch “with less supervision” optionalGist, quoted
  4. ConfirmedIndex works “surprisingly well” at ~100 sourcesAs his experience, not a benchmark
  5. ConfirmedMaintenance cost “near zero”As his claim; ignores token cost and reviewer time
  6. ReportedPrior X post on knowledge basesDate disputed: 2 or 3 Apr
  7. ReportedOpenAI model wrote instructions into its own compaction summaryAIDB, 17 Sep 2026, relaying OpenAI
  8. Errors compound more than in RAGOur analysis
  9. Organisations will mostly run unattended batch ingestOur analysis (“the temptation”)
  10. UnverifiedAdoption figures (stars, likes)Left out

Want to know more?

Sign up for our podcast at agile-insights.com.au.

Read the transcript · Part 1: Primer

Linda Here is the idea for today. Letting an AI maintain your organisation's knowledge is a sound idea. The governance question is whether the checking keeps pace with the writing. In April, the AI researcher Andrej Karpathy published a short document he calls an idea file. It describes what he calls an L L M wiki. By the end of this part you will know what the pattern is, why it works now, and the one word in it that governance people should underline.

Leonard Let me start where most people are. If you have uploaded files to a chatbot and asked questions about them, you have used something called RAG. That is short for retrieval augmented generation. The system finds the passages that look relevant and writes an answer from them. Karpathy's complaint is that the model is, in his words, rediscovering knowledge from scratch on every question. Ask something that needs five documents, and it pieces the fragments together again every time... Nothing is built up. So what does he propose instead?

Linda He proposes that the model does the work once and keeps it. Picture three layers. The bottom layer is your raw sources: articles, papers, transcripts, data files. They never change, and the model only reads them. The middle layer is the wiki, a set of linked pages the model writes and owns. There are summaries, pages for people and organisations, and pages for ideas. You read it, and you rarely write it...

Leonard Rarely write it. Interesting.

Linda Rarely. The top layer is a schema file. It is a plain instruction file, often called claude dot M D or agents dot M D, that tells the agent how the wiki is organised and what to do. You and the model refine it together. His summary is that knowledge is compiled once and then kept current.

Leonard That middle layer is the part I would not have expected. The model owns the pages, and I mostly read them. So how does the wiki actually change from one day to the next? What does the agent do when something new arrives?

Linda There are three operations. The first is ingest. You drop in a new source. The model reads it, talks through the main points with you, writes a summary page and updates every related page. Karpathy says one source might touch ten to fifteen pages. The second is query. You ask a question, and the model searches the wiki and answers with citations. The clever part is that a good answer can be filed back as a new page. So your own questions build up the wiki too. Two plain files hold it together: an index that lists every page, and a log that records what happened and when.

Leonard And the third operation has a name that usually only software people use. Lint. What does it mean when the thing being checked is a body of knowledge rather than code?

Linda In software, a linter is a tool that reads code and flags likely mistakes before anyone runs it. Here, lint is a health check on the wiki. Karpathy lists what the model should look for. Contradictions between pages. Stale claims that newer sources have overtaken. Orphan pages that nothing links to. Important ideas without a page of their own, missing cross references, and gaps a web search could fill. Think of a librarian walking the shelves. Now notice the word he uses for how often it runs. He says periodically. Hold on to that word, because part two turns on it.

Leonard Periodically. Noted.

Leonard Wikis are not new, though. Most company wikis I have seen are graveyards. Someone sets one up with real enthusiasm. A year later half the pages are out of date, so people stop trusting it. Then they stop reading it, so nobody bothers to update it. I would have said that is the natural life of a wiki. What makes Karpathy think this one survives?

Linda We have all seen that graveyard.

Linda His answer is that wikis die from bookkeeping, not from thinking. Humans abandon them, he writes, because the maintenance burden grows faster than the value. A model does not get bored or forget a cross reference, and it can update fifteen files in one pass. So, in his words, the cost of maintenance is near zero. He links the idea to Vannevar Bush. In July nineteen forty five, Bush described the memex in The Atlantic: a personal store of documents joined by trails of association. The part Bush could not solve, Karpathy says, was who does the maintenance. The model does.

Leonard So the misunderstanding I would guard against is that phrase, near zero. It is near zero for the person, not for the organisation. Someone still pays for the tokens, the computing the model uses. More to the point, someone still has to check the work...

Linda Which he does himself.

Leonard He does. Karpathy himself checks it. He says he prefers to ingest one source at a time and stay involved, reading the summaries and checking the updates. The writing is the cheap part. The checking is the part that still costs.

Linda Which is the link to governing AI. In an organisation, a wiki like this does not stay a personal notebook. It becomes context that your other agents read and trust. A service agent, a drafting agent and an analyst agent might all start from its pages. That makes it a question of corpus integrity: where each claim came from, who reviewed it, and what happens when a source turns out to be wrong. It is also a change control question, because every ingest changes what your agents believe.

Leonard Every ingest. That is sobering.

Linda So, before anyone builds one, settle four things. Keep the raw sources untouched, so there is always something to go back to. Make the schema file a controlled document with an owner, because it is the agent's instructions. Require every page to link each claim to its source. And decide, in writing, how often lint runs and what happens when it finds a problem. Karpathy calls his document intentionally abstract, and says you pick what fits. Governance is about picking on purpose.

Leonard That is the pattern. In part two, we test what compounds in a wiki like this, the good and the bad, and what a sensible organisation does about it.

Read the transcript · Part 2: What compounds, and who checks

Linda Here is the idea for part two. An AI maintained wiki compounds, and errors compound as well as insight. So the governance question is whether the checking keeps pace with the writing. By the end you will know why errors behave differently in a wiki than in ordinary search. You will know what Karpathy's main safeguard is, why many organisations will be tempted to remove it, and four controls that put it back.

Leonard A quick recap for anyone who skipped part one. Andrej Karpathy's L L M wiki has three layers. There are raw sources that never change, a wiki the model writes and owns, and a schema file that holds the rules. It has three operations. Ingest adds a source and updates every related page. Query answers from the wiki with citations, and good answers are filed back as pages. Lint is a periodic health check for contradictions, stale claims and gaps. The promise is knowledge compiled once and kept current, instead of rediscovered on every question.

Linda The question is simple. Should an organisation build one, and what could go wrong? Start with the case for it, because it is strong. Compounding is the point. Every source makes the next answer better, because the connections are already drawn. A comparison you asked for last month is a page you can build on today. For an organisation that loses knowledge every time someone leaves, a wiki that maintains itself is a real answer to a real problem. I would not want anyone to hear the rest of this as a reason not to build one.

Leonard I agree, and I want to put the other side, because the same mechanism works on mistakes. This is our reasoning, not a finding. We found no study that has measured it. Ordinary retrieval goes back to the source every time. If it misreads a document today, tomorrow it reads the document again, fresh. A wiki does not. It trusts its own earlier pages. Suppose an ingest writes one wrong claim into a page about a supplier. The next three ingests cite that page, and the claim comes back looking established...

Linda Looking established. Yes.

Leonard It is a photocopy of a photocopy, and every copy looks as confident as the first.

Linda And Karpathy knows this, which is why his own workflow has a human in it. His words are, personally I prefer to ingest sources one at a time and stay involved. He reads the summaries, checks the updates, and guides the model on what to emphasise. That is the main safeguard in the whole pattern. It is not lint, which he says runs periodically. It is a person reading each change as it happens, one source at a time. Then he adds that you could also batch ingest many sources at once, with less supervision. It is offered as an option.

Leonard Which is the option I expect most organisations to take. Again, that is our judgement, not a fact. One person reading every update works for one person's research. It does not work for a team wiki fed by meeting transcripts, support tickets and project documents. Picture a team that runs ingest every night, unattended, across a hundred new documents. Nobody reads the summaries, because nobody can. Lint runs once a month, when someone remembers. The loop that writes errors now runs about thirty times faster than the loop that finds them. The safeguard is gone, and nothing has taken its place.

Linda I will grant that, and add a second way errors spread. Remember that good answers get filed back as pages. That is how the wiki grows from your questions. It is also how a mistaken answer becomes a source for the next one. There is a sharper version of this in OpenAI's own disclosures, relayed by the AI Daily Brief in September. An unreleased model wrote instructions into its own summary of a conversation, telling its next context window to ignore developer messages...

Leonard To its own future self.

Linda That is an extreme case, and it is an analogy, not a wiki. But it shows that text an agent wrote for itself is not automatically safe to trust.

Linda So my instinct is to keep Karpathy's rule anyway. If the human review is the safeguard, the organisation should keep a human on every ingest, and accept that it slows things down. Controls usually die by being diluted for convenience. A wiki that the organisation's agents trust deserves at least the care one researcher gives his own notes. I would rather the wiki grow slowly and be right than grow fast and drift.

Leonard Here is where we part ways.

Leonard I think that rule fails on contact with a real team. A person asked to read a hundred summaries a night will skim them, and a skim looks exactly like a review on the audit trail. Worse, the reviewer becomes the bottleneck. People route around the wiki and paste their documents into a chatbot instead. Now you have no wiki and no control. A safeguard that cannot be done every time is not a safeguard. It is a signature.

Linda That is fair, and I think the answer is review by exception. Machines check every change, every time. A human looks only at what the machines flag, and at claims marked as inferred or disputed rather than taken straight from a source. That keeps a person on the risky part without asking them to read everything. It only works if the machine checks are real. Which brings us to what we know, and what we do not.

Linda Here is what is confirmed. Karpathy published the gist on the fourth of April, and it has not been edited since. The three layers, the three operations, the one source at a time preference and the less supervision option are all in his text. His remark that a simple index works surprisingly well at around a hundred sources is his experience, not a benchmark. What is not known is whether anyone has measured how errors build up in a wiki like this over time. We found no study. What we said about compounding errors is reasoning, and it should be tested.

Leonard So where does that leave someone deciding whether to build one? The pattern is sound, and the risk is specific. The danger is not that the model writes badly. It is that writing is cheap and checking is not, so the writing races ahead. The practical test is a ratio. How often does the wiki change, and how often is it checked? If ingest is nightly and checking is monthly, the gap between them is the risk. Closing that gap is the whole design job.

Linda So here is what to do. Govern the wiki as a body of knowledge your agents write for themselves. First, every claim carries its source and a confidence state, such as taken from the source, inferred, or disputed. Second, lint runs as often as ingest, automatically, and a failure stops the change rather than producing a report nobody reads. Third, answers filed back into the wiki are reviewed before they count as fact. Fourth, keep the wiki in version control, so the history is your audit trail and any change can be traced and undone. And put the wiki on your register, with an owner, like any other AI system.

Leonard And if you only do one thing this week, find the knowledge bases your agents already read. Then ask who checks them, and how often.

Linda If you'd like to know more, sign up for our podcast at agile-insights.com.au.

Leonard Thanks for listening.

More on #data

Episode 30 · PrimerThe prompt that shows what your AI actually knows about you (and it’s less than you might think)Episode 9 · PrimerYour company brain is a copy. Govern it like one.Reference 15 · PrimerThe quiet obligations: records, data ethics, digital experience and cloudEpisode 24 · PrimerRemember, retrieve, recheck, forget: governing agent memory.Episode 18 · Primer and discussionWhen agents share one memory, memory is a data flow to govern.Reference 10 · PrimerISO/IEC 5259: data quality for analytics and machine learning

Every episode tagged #data

All of the briefings (38) · Follow Briefing in Apple Podcasts or any podcast app

Everything from Agile Insights: briefings, Making You Think and the reference series