The tool responded. The agent didn't get anywhere.

Every tool on our MCP server passed its tests. In the first four weeks with analytics on, agents made 7,956 tool calls and 89 of them failed: 98.9% success. The dashboards were green. And agents were still working around us.

When you look at MCP for the first time, it’s easy to treat it as just another programmatic interface. It is one, to a certain extent. The problem is which programs call it. AI agents are a very particular kind of client: they call your tool, they get a response, and they may never get their job done. Eventually that becomes your problem.

MCP makes this easy to miss. A tool error doesn’t come back as an HTTP error. Our handlers return an error result inside an ordinary response, so at the transport layer every one of those calls looks fine. And a call that succeeds can still leave the agent nowhere.

That gap - between “the tool responded” and “the agent finished what it came for” - is the whole story. An MCP server is a product whose users you can’t interview. They don’t file tickets or churn where you can see it; they retry, improvise around you, or give up and tell a human “I couldn’t do that.” The telemetry is the only interview you get. Read it well and it tells you what to change; wire it into a loop and it drafts the change.

This is the first of three parts. This one is the setup and three things the telemetry showed us that our error rate never would. Part 2 is about what agents asked for, and what instrumenting the server so closely did to my own numbers. Part 3 is the loop that turns all of it into pull requests, the first pull request it got wrong, and what I’d instrument on day one. Both are coming in the next few days.

The setup

Playbook is the media backend for AI products: agents generate the media, Playbook stores it, versions it, keeps a human in the loop before it ships, and serves it to production. Our MCP server is how agents reach it - 69 (^..^) tools on Cloudflare Workers, in front of our Rails API.

On August 4 we wrapped it with @posthog/mcp, PostHog MCP analytics SDK. It’s one call: instrument() patches initialize, tools/list and tools/call, so every tool is covered without per-tool code. Three things did most of the work:

• context: true adds a required context parameter to every tool. The agent writes one sentence on what the user is actually trying to do, and our handlers never see it.
• reportMissing: true adds a virtual tool, get_more_tools. When an agent can’t find what it needs, it calls it and files a claim, in its own words.
• Raw events every call lands with its tool name, arguments, duration, error text and - when the client sends one - the client’s name. Nothing is aggregated at write time, which turned out to matter: most of the questions below are ones we didn’t think to ask when we instrumented.

1. Sort failures by who could have prevented them

The default view of MCP errors is “the client sent something wrong”. Sort them instead by whether the server could have made that call succeed.

25 of our 89 failures - 28% - were the same schema rejection: a missing organization. Every workspace-scoped tool required a workspace slug. For an account with one workspace, the token already says which workspace it is; asking the agent to look the slug up and pass it anyway is our requirement, not its mistake. An agent could call list_organizations first. Many didn’t.

The fix was to make it optional on the wire and resolve it from the token, without touching a single handler:

// each tool still declares `organization: z.string()`;
// at registration the advertised schema swaps it for this
export const organizationArg = z
  .string()
  .optional()
  .describe(
    "Workspace slug, as returned by list_organizations. Omit it when the account has a " +
      "single workspace - it is then resolved from your token. Pass it only when the " +
      "account has several, or when you mean a workspace other than the obvious one.",
  );

A wrapper fills the slug in before the tool runs, with a cache that outlives one request - on Workers every tool call is its own HTTP request, so an in-memory memo covers exactly one call.

Since it went live, it accounts for zero of 542 failures. The day after, 124 calls succeeded without the parameter at all. Accounts with several workspaces now get a different error - “which one?” - and that’s the right failure to have.

2. Your tool description is an instruction

Put every duration next to your own constants.

Publishing permalinks for a batch of assets: in the tool’s first twelve calls, the median, the 95th percentile and the maximum were all exactly 15,000 ms - our own request timeout, cutting the call off. And the tool description, the text agents read to decide how to call a tool, said: “Publishing copies each file in the storage bucket, so a bigger batch is what times the request out - split a large set and call again.”

That’s exactly what agents did, through August and on into September - the fix sat merged but not deployed for eight days, which is its own story in part 2. One agent sent 13 assets, timed out, and came back with three calls of 3. Another went from 39 to batches of 10, most of them landing at 13.4–13.9 seconds, just under the wall. One sent 15, timed out, retried with fewer and got back “all assets already have permalinks”: the call that “failed” had in fact published.

They were doing what we told them: working around our limit by hand. The fix was to take that job off them - give this call 30 seconds instead of 15, chunk the batch server-side, 25 at a time, and stop starting new chunks after 60 seconds; when a chunk fails, re-read what actually landed before reporting; and tell the agent what was created, what remains, and why we stopped:

{ "created": [...], "remaining": [...], "stopped_because": "..." }

The 15-second wall is gone. And there’s a new one at 20–21 seconds: every call in that bucket is an error, and at least 82 of the 92 are 500s from our own Rails API in the middle of a batch. Where exactly those 20 seconds come from is still open. The next “agent failure” is ours again, one layer down.

3. Error text is read by a loop, not a person

An MCP error message goes to a model, which reads it in a loop and decides what to do next. That makes error text part of the interface, and it deserves the same care as a tool description. Throw something - and you get something in turn.

Our API answers “this doesn’t exist” and “you can’t see this” with the same 404. The old text, 23 times in production, was Not found: Not found (not_found) - nothing a loop can act on. Actually, some agents found a way - trying to call the tool again over 1.5k times (guess the token burn).

The new one says what it can: either the identifier matches nothing in this workspace, or it’s real and this account can’t see it; retrying unchanged will fail the same way; here’s where to re-read the token; and if it still looks right, ask the user rather than guessing again.

Did it change what agents do? Repeats of the same call within a minute of a 404 went from 3 of 26 before to 44 of 222 after - but 37 of those 44 are one agent. Without it, 10.4% against 11.5%. That’s not an effect I can claim - yet.

---

Three findings, and not one of them showed up as a red number. The organization rejections were a 1% problem. The timeout looked like slow clients. The 404s looked like agents asking for things that don’t exist. In each case the server answered, and the agent was left to work it out.

Next in part 2: what agents asked for when nothing in the toolset fit, and the three of my own claims that this telemetry didn’t let me keep.