Discovery documents, and the robots.txt trap

There is a gap at the centre of WebMCP that is easy to miss until it bites: tools live in a page, so an agent cannot discover them without visiting first. It cannot include you in a plan it forms before the visit, and it cannot answer which sites can do this with your name in the list.

Discovery documents close that gap. They are small, static files at conventional paths that say what you offer to a machine that has not arrived yet.

The documents worth publishing

Most sites need one or two of these, not all six. A server card if you run an MCP server; an OpenAPI document if you have a public API. The auth documents only matter once something is protected.

A server card

{
  "$schema": "https://static.modelcontextprotocol.io/schemas/2025-07-09/server.schema.json",
  "name": "com.example/catalog",
  "description": "Search the catalog, check accessory compatibility, read stock.",
  "version": "1.0.0",
  "websiteUrl": "https://example.com/developers",
  "remotes": [
    { "type": "streamable-http", "url": "https://example.com/api/mcp" }
  ]
}

The trap

It is a good failure to understand because it is invisible from the inside. You verify the endpoint with curl, it answers, you move on. The scanner that respects robots.txt reports manifest found, handshake failed, and you conclude the scanner is broken.

User-agent: GPTBot
Allow: /api/mcp
Allow: /.well-known/
Disallow: /api/

Repeat it in each user-agent group you maintain. Our discovery check reads the manifest, extracts the URL it points at, and evaluates that path against your own robots.txt — which is how it catches this specific contradiction rather than just reporting that a file exists.

Two more things that quietly break discovery

• A soft 404. Many frameworks serve a styled error page with status 200 for unknown paths. A probe then sees a 200 and HTML, which is not a discovery document. We check the content type for exactly this reason.
• A manifest that does not match reality. A card advertising streamable-http at a URL that only accepts a different transport, or a version that moved on. If the handshake fails, the card is worse than absent — it sends agents to a dead end.

How we grade it

3 points in the interop pillar. Any responding discovery document passes. A document whose target path is disallowed by your own robots.txt scores partial, with the offending path named in the evidence. Nothing responding is a fail — and for a site with no API and no server, that is a fair thing to lose two or three points on rather than a crisis.