Briefing
Episode 20 · Primer and discussion · About 14 minutes in two parts
Download the MP3 · Linda and Leonard, AI voices, read a script written from the sources below.
Download the MP3 · Linda and Leonard, AI voices, read a script written from the sources below.
Written from Microsoft's own blog post and documentation, and checked on 27 October 2026. Everything about how the feature behaves is Microsoft's description; we found no independent testing. Whether comments carry names, who counts as the owner, the release date and any extra cost are not yet known. Check the originals before quoting.
Before we record, every claim the hosts make is checked against the sources (this table was last updated 27 October 2026). Confirmed means a primary source or two independent reports; Reported means one party says so; Partly right, Disputed, Unverified and Not yet known mean what they say, and the hosts say so on air. This is the table as we left it.
Sign up for our podcast at agile-insights.com.au.
Linda Here is the idea for this episode. A usage number tells you an agent is being used. It does not tell you the agent is any good. On the twenty second of October, Microsoft described a preview that lets agent owners read what people say about their agents. That helps, but only when someone decides who sees the feedback, who reads it, and what changes because of it. By the end of this part, you will know the numbers Microsoft reports and what feedback adds. You will also know the caveat that matters most for a certified organisation.
Leonard Let me set the scene from where a lot of listeners sit. Many organisations now have a handful of Copilot agents, built in Copilot Studio or Agent Builder, answering questions about policies or products. Someone in the executive team eventually asks whether they are working. So what is this Insights Agent, and why does it matter to a governance lead now rather than next year?
Linda Insights Agent is itself a Copilot agent. You open it in Copilot Chat and ask, in plain words, how the agents in your organisation are being used. Microsoft's documentation says you can ask about any agent you can see, and you don't need to own it. Asking in chat is included in the Copilot licence. To pull the same numbers into your own dashboards, you use a service called Work I Q. That is charged in Copilot credits. It matters now because it is the first place most organisations will look for evidence about their agents. So it is worth knowing exactly what that evidence can and cannot show.
Leonard I suspect most executives will look at one chart and stop there. So walk me through the numbers. What does it actually report, and how much weight can each one carry?
Linda There are three to understand. The first is active users: how many different people used the agent in a day, a week, or a month. The second is stickiness, which is daily users divided by monthly users. Say a hundred people use an agent this month, and on an average day twenty of them do. That is twenty percent, or roughly six days a month for a typical user. The third is what Microsoft calls a rolling retention proxy. It divides this period's users by the previous period's users. Microsoft is careful here.
Leonard Careful how?
Linda It says this is not retention of the same people, because the agent collects no individual identity, by design. A hundred users last week and a hundred this week could be entirely different people.
Leonard So all three are counts. They tell me people turned up, and how often. They say nothing about whether the answer they got was right. Which is where the new feedback part comes in, I take it?
Linda Exactly right. The preview is called Insights Agent V two, Frontier. When a person uses an agent, they can give it a thumbs up or a thumbs down, and write a comment. With the preview, the agent's owner can read recent comments and count the thumbs for a week. They can also ask for a summary of themes. There is a switch first, though. An administrator chooses, in the Microsoft three six five admin centre, whether feedback is shared for no agents, all agents, or selected agents. That switch does not stop people rating agents. It only decides whether the people who build them ever see the ratings. Microsoft gives an example. If several comments say an answer is unclear, check the agent's instructions or knowledge, change it, and test again.
Leonard And the word Frontier in that name. My understanding is that it marks an early preview, not a finished product?
Linda That's right, and it carries a caveat worth hearing slowly. Frontier is Microsoft's opt in programme for features before general release. An administrator turns it on for everyone or for named groups, and it is off by default. Microsoft says Frontier features can change, be suspended, or be withdrawn, and may not have standard support. Its documentation also says Frontier features do not comply with I S O twenty seven thousand and one, twenty seven thousand and two, or twenty seven thousand and eighteen.
Leonard That one matters.
Linda For an organisation that holds those certifications, a preview feature should not become the evidence of record for a control. Use it to learn, with a small group, and keep your formal evidence somewhere stable.
Leonard There are two misunderstandings I can see people falling into here. The first is that a popular agent must be a good one. Usage climbs, everyone relaxes...
Linda We have all seen that dashboard.
Leonard But an agent can be popular and still be wrong, especially if it answers confidently. The second is about that theme summary. It is tempting to read the summary and skip the comments. Microsoft itself warns against that. It says the summary is an A I generated synthesis, not a substitute for reading the feedback. Review the original comments before deciding what to change.
Linda And that is the governance point in one line. Running an agent in production needs three things for quality. First, a written statement of what good looks like. Second, a way to notice when it falls short. Third, a fallback when it does. Usage numbers give you none of those. Feedback can help with noticing. But only if a named person reads it on a regular cadence, and records what changed and why. Two things are not yet known. Microsoft does not say whether the comments owners see carry the commenter's name, and it gives no date for general release. In Part 2, we will argue about what to do with that.
Leonard That's the groundwork for Part 2. See you there.
Linda Here is the idea for this part. Usage tells you an agent is being used, not that it is any good. Feedback from the people using it can close that gap. But only when an organisation decides who sees it, who reads it, and what changes. By the end, you will have three views on how far to switch this on. You will know what is settled and what is not, and five things to do this quarter.
Leonard A quick recap for anyone joining here. On the twenty second of October, Microsoft described a preview of its Insights Agent, a Copilot agent that reports how the agents in your organisation are used. The usage side counts active users and stickiness, which is daily users over monthly users. It also gives a retention proxy, which does not track the same people coming back. The preview, called V two, Frontier, adds thumbs up and down, written comments, and an A I summary of themes. An administrator decides whether that feedback is shared for no agents, all agents, or selected agents.
Linda So here is the question. Your organisation has a dozen agents, some popular, some quiet, and a preview that would let their builders read what users say. How far should you switch it on? I want to take three views. The first is to switch it on for every agent, and Leonard is going to make that case. The second is to switch it on only for agents that have a named owner and a stated standard. The third is about the word Frontier, and what a preview should and should not carry.
Leonard Let me make the case for every agent. People are already rating these agents. Microsoft's documentation says the sharing switch does not change whether users can rate. It only decides whether builders ever see the ratings. So with sharing off, staff are giving feedback into a void. That is a poor way to treat the people you asked to adopt the thing.
Linda Feedback into a void. Lovely.
Leonard And an agent nobody is watching is exactly the one most likely to drift. If you only open feedback for the agents you already care about, the forgotten ones stay forgotten. Visibility is cheap. Turn it on and let the builders learn...
Linda I'd push back on one part of that, though. Switching it on everywhere sends comments to whoever counts as the builder, and Microsoft's pages don't agree on who that is. The blog says the owner. The admin page says developers. Neither defines it. We also don't know what travels with a comment. Microsoft does not say whether owners see the commenter's name. Its feedback settings let users attach logs, screenshots and a contact email, and those settings are on by default. Whether any of that reaches an owner through this preview is not stated. A comment can easily include the question someone asked, and that question can be personal.
Leonard That's fair, and I'll concede the scope. My worry was the forgotten agents, and you can answer that a different way. An agent with no owner and no standard should not be in production in the first place. So the real fix for a forgotten agent is to give it an owner, or retire it. Opening its feedback to an unnamed builder doesn't do either. Selected agents, tied to whoever is named on your register, gets most of the benefit with far fewer surprises.
Linda Good, so that's settled between us. Now the third view, which is the word Frontier. Frontier is Microsoft's opt in programme for features before general release. Microsoft says these features can change, be suspended or withdrawn, and may not carry standard support. And its documentation says plainly that Frontier features do not comply with I S O twenty seven thousand and one, twenty seven thousand and two, or twenty seven thousand and eighteen.
Leonard Worth reading twice.
Linda For an organisation that holds those certifications, or is building an A I management system on top of one, that matters. A preview should not be the place your quality evidence lives.
Leonard I'll add the good news to that. The useful part here is not the preview at all. It is the habit. Read what users say, pick one thing to change, change it, test it, and see what they say next. Microsoft describes exactly that loop in the post. You can run that loop today with a sampled review of real answers, whether or not you have the preview. The preview makes the reading faster. It doesn't make the deciding happen. So pilot the preview with a small Frontier group, and keep the record of what you changed somewhere stable, in your own register.
Linda Let me be clear on where the evidence settles. Confirmed, from Microsoft's own pages: what the sharing switch does and does not control. That the usage figures are aggregate counts with no individual identity. That Microsoft says not to treat usage alone as a measure of quality or value. That the theme summary is an A I synthesis and you should read the comments themselves. And the Frontier compliance statement. Not yet known: whether comments carry names, who exactly counts as the owner, whether the preview costs extra, and when it reaches general release. We found no independent testing, so everything about how it behaves is Microsoft's own description.
Leonard One more limit, and it comes from Microsoft's own wording. Feedback tells you what users have chosen to share. The people who press thumbs down are not a random sample. Some are frustrated, some are generous, and most say nothing at all.
Linda Most of us, honestly.
Leonard So a quiet agent with no complaints may be fine, or may simply be ignored. That's why feedback works as one signal of quality, beside a sampled review, and never as the whole measure. A handful of angry comments is a reason to look. It isn't a verdict.
Linda So, five things to do this quarter. One, pick the two or three agents that matter most, and make sure each has a named owner. Two, set feedback sharing to selected agents, for those owners, and write down why. Three, before anyone reads a chart, write one line on what good looks like for each agent. Add how you would notice it falling short. Four, have the owner read the raw comments on a fixed cadence, monthly is fine, and log each change and its reason. Five, keep the Frontier preview inside a small enrolled group, and out of your certified control evidence, until it reaches general release.
Leonard And one thing to ask Microsoft, or test in your own tenant, before you widen it. When an owner reads a comment, what do they see about the person who wrote it? Settle that, and the scope decision gets much easier. The answer may be harmless. But you want to know before a complaint makes the decision for you. Until then, treat every comment as possibly personal information, and handle it that way.
Linda If you'd like to know more, sign up for our podcast at agile-insights.com.au.
Leonard Thanks for listening.
Every episode tagged #microsoft
All of the briefings (38) · Follow Briefing in Apple Podcasts or any podcast app
Everything from Agile Insights: briefings, Making You Think and the reference series