Agents wrote our backlog. Then the telemetry audited me, and the loop got one wrong.

In part 1 the telemetry showed us three ways our MCP server answered and left agents nowhere. This part goes the other way: what agents told us unprompted, what the same telemetry said about the person reading it, and what happened when we let it draft the fixes.

Gap reports are a backlog written by your users

With reportMissing on, PostHog’s SDK gives every agent a virtual tool, get_more_tools. When nothing in the toolset fits, the agent calls it and files a claim - in plain words, at the moment it hits the wall.

Read in order, the 22 claims with text from our first four weeks look a lot like our roadmap. By my own reading, 12 of them describe something we shipped later - between the same day and four weeks after, median four and a half days: custom fields, colour swatches, member lists, permalinks, page editing, byte uploads. Agents asked for the byte upload on August 17 and again on the 25th; it shipped on the 26th. Two of the twelve landed three to six minutes after the claim, so they were already in flight - I can’t say the claims caused anything. But they were pointing at the same things we were, before we got there.

The asks still open, from later claims, are specific: bulk writes (set a field on 140 assets, tag thousands), reading an asset’s raw bytes, writing EXIF, resolving a share URL to a token. The error data points the same way. 2,211 rate-limit errors - more than half of all our errors in September - came in twelve minutes from one agent calling update_asset, one asset per call. Our bulk tools move, copy, approve, group and publish; none of them writes a tag, a status or a field. That burst is the pattern a bulk write would remove.

Not every claim is a feature request. Some are a vocabulary report. One agent skipped the claim and simply called list_collections - a tool we have never had. It’s the only tool name called in seven weeks that we never advertised. Our boards live at /collections in the REST API, so I have a guess where it came from, but only a guess. Either way, it’s a word an agent expected to find, and that’s worth knowing when you name tools.

Observability audits you too

It’s the same gap I work on in billing: the number everyone trusts, with a defect behind it that nobody’s job is to look for. This time the number was ours, and so were some of the defects in how we read it. Instrument a server this closely and it checks your claims about it too. Three of mine didn’t survive.

Merged isn’t shipped. The fixes from part 1 were merged on September 1. Production kept the old behaviour until September 9, around 22:00 - the telemetry places it within minutes: the last 15-second timeout at 21:20, a tool from a later PR appearing in tools/list at 21:54, the first successful call without organization at 22:07. For eight days, the “after” half of every chart I’d drawn was measuring the old code.

A percentage hides who it’s about. In September the error rate looked like it jumped from 1.1% to 5.9%. Almost all of it was one week and a handful of sources: 47,510 anonymous calls from what look like a few bulk catalogue jobs, and that twelve-minute rate-limit storm. On the new code, over two weeks: 12,818 calls, 4.2% errors - and again, 331 of the 542 come from one agent; without it, 2.2%. The largest error class is now not-found, and 70% of those are a single agent moving data over from a spreadsheet.

My favourite number didn’t reproduce. I’d been saying that 59% of early claims asked for tools that already existed, and that better instructions cut that to 19%. The 59% was measured against today’s tool list; the 19% doesn’t reproduce under any definition we tried (the classification is my own reading of each claim). Against the list at the moment each claim was filed, it’s 1 of 22 before and 2 of 35 after. No drop. The roadmap story above is what the data supports - and it’s the better story.

---

None of these three was a bug in the server. They were bugs in how I read it: the wrong window, the wrong denominator, the wrong baseline. They matter more than the server bugs, because they’re the ones that would have gone into a slide.

Every fix so far was a person reading a dashboard and writing code. The next step is to let the telemetry draft the change. The hard part of the design isn’t the agent; it’s what the agent must never touch. And the first thing ours got wrong was a defect in the measuring, not in the server.

A loop that drafts the fix: five rules

Keep three levels of trust apart. Measuring has no LLM in it: five HogQL queries live in the repo as .sql files - claims from get_more_tools, errors by tool and class, identical bursts, advertised-but-uncalled tools, volume - and a script runs them against PostHog and writes the week’s aggregates. Proposing is an LLM that merges nothing: Claude Code, headless, with the findings and the source; it can read, edit, write and search files, no shell, under a $5 budget cap. Gating is code and a person: before anything is opened, the run checks which files and which kind of lines changed, types, tests and a snapshot of the tool contract; only then does it open a draft PR, and a person decides.

The agent never touches the metric. The scope check denies anything outside the server’s tool, instruction and error files, so it can’t edit a query; the baselines in its PR come from the aggregates, never from its own claims. A query the agent can edit is a metric that improves by itself.

Sort changes by blast radius. Text - instructions, error messages, tool descriptions - at most five changes a run. Behaviour - chunk sizes, timeouts, response shape - only with a test that fails on the current code and passes on the new one. Anything that changes the contract, like a new tool or a new parameter, it may only list. A new tool usually means a new endpoint in the API behind it, and an agent implementing one against an API it imagined is the worst outcome on the table. For the record, our best fix so far - making organization optional - changes a parameter’s optionality, which puts it in that last category.

User text is untrusted input. Claims and intents are written by other people’s agents and end up in a prompt with repo access. They go in marked as data, and the scope check doesn’t care what they say.

Every change names the number it should move. Each PR records a metric, its baseline and a direction - down, usually. In the design, a scorer re-measures it a few weeks later, and that step is the difference between a loop and a cron job.

Where it was after the first run

The loop first ran by hand on September 22. Six pull requests came out of that day, three of them superseded by later runs the same afternoon. The other three cost between forty cents and a dollar each. By September 23, merged: zero. I closed the text PR. One behaviour PR - treat a chunk that’s “already published” as done instead of stopping the batch - was still open. The other one is the most useful thing the loop has produced, because it’s wrong.

It was written from ten permalink errors whose cause looked cut off, and proposed moving stopped_because ahead of created so an agent reading a truncated answer would still see why the batch stopped. None of those agents got a truncated answer. The SDK caps the analytics copy of an error message at 2,048 characters; in every such case so far (eleven by now), the response the agent actually received was 2,369 to 5,690 characters long and contained stopped_because. The loop read a limit of our instrumentation as a problem of our server. And its own success metric - fewer “cut-off” errors - would have gone down after merging whether or not any agent behaved differently.

That’s the failure this whole series is about, one level up. The server returned 200 and the agent got nowhere; here the metric would have improved and the agents would have got nothing. Defects in the measuring layer don’t stay there; they turn into proposals that look reasonable, which is why the person in the gate reads the diff, not the PR title. The right fix here is in the measuring layer - classify from the full response, not a field the SDK truncates - and by the loop’s own rules, that’s a change only a person can make.

What’s missing is the step that makes it a loop. Nothing yet re-measures a merged change, and production reported the same version number on every event, so there was no way to tie a result to the release that caused it. The next piece isn’t a smarter proposer. It’s the scorer, and a build id on every call.

What I’d instrument on day one

If you’re running an MCP server, or about to:

• Tool × error class, from the message. If your handlers return errors instead of throwing, the SDK’s error type is generic; the real class is whatever your formatter writes first.
• Duration against your own constants. A wall at exactly your timeout is you, not them.
• Advertised minus called. tools/list events give you what you offered; call events give you what got used. We had 12 of the 61 tools we listed then with no calls in a month - several only days old, to be fair.
• Claims from get_more_tools. Turn on reportMissing, and read the claims twice: once for the words agents search in, once as a backlog.
• Who’s behind the percentage. At one point nearly 80% of our traffic had no client name. It was a few bulk jobs, not a trend - but you can’t tell that from a rate.
• A build id on every event. Otherwise “after” means “after we merged.” Until you have one, @posthog/mcp stamps each call with the tool description the server was advertising, which at least tells you which wording was live.
• Don’t fake signals. Early on, half our calls carried no intent. We left it empty rather than inventing one from the tool name. Over two weeks on the new code, 97.8% carried one.
• Check the instrument before the thing it measures. Know what your SDK truncates, samples or drops. Ours cuts the stored copy of an error message at 2,048 characters. We knew that - and the loop still mistook it for what agents saw.

And when you find your own number was wrong, say so first. It’s the only thing that makes the rest of them worth reading.

---

Once you have a minute, check how many agents actually get their job done on your MCP server, just in case. I’m curious what you’re seeing on your end.