What a Random Draw from the MCP Registry Contains, and What Tool-Use Benchmarks Contain Instead
Haseeb Mohammed Afsar (independent researcher) probes a seeded random sample of 400 servers from the 24,135-server MCP registry and finds that fewer than half start, and that popular tool-use benchmarks contain far more duplicated tools than real MCP servers do.
Ask this paper
Sampling: Earlier MCP studies used curated lists or repaired servers until they ran. This one draws 400 npm/stdio servers from the full registry with a published seed and probes each over the wire without repair.
Startup failures: Only 48.8% complete an initialize handshake, against 66.7% for a hand-curated set measured the same way. The main failure is servers that never start (37.5%), with missing credentials at 13.3%.
Conformance and annotations: The 195 working servers show zero fatal JSON Schema violations across 2,766 tools. Safety annotations vary most, with a 58.8% omission rate on the random draw against 41.5% on the curated set.
Benchmark comparison: Real MCP tools show 2.8% near-duplication, all within single servers. BFCL v4 shows 16.7%, and 68.8% of raw BFCL rows and 85.6% of raw UltraTool rows are exact repeats, against 0.4% for real MCP.
Implication: Statistics over these benchmark releases without global deduplication measure repetition rather than tool diversity.
Abstract
Studies of the Model Context Protocol (MCP) server ecosystem draw their samples in ways that quietly select for servers that work: reference sets, popularity lists, hand-curated frames, or pipelines that repair a server until it starts. We report what an unrepaired probability sample actually contains. From a 24,135-server registry census we draw 400 npm/stdio servers with a published seed and probe each one over the wire. Only 48.8% complete an initialize handshake, against 66.7% for a hand-curated frame measured with the same instrument, and the dominant failure is not missing credentials (13.3%) but servers that never start at all (37.5%). Among the 195 that do run, hard conformance is total: zero fatal JSON Schema violations across 2,766 advertised tools. Optional safety annotations are the real variance, and the tool-level omission rate on a random draw is 58.8% against 41.5% on the curated frame, so curation flatters this figure too. We then compare the tool descriptions these servers advertise against two tool-use benchmark corpora using one method held constant. Real MCP tools show 2.8% near-duplication at cosine 0.70, and all of it lies within single servers: cross-author near-duplication is 0.0% at every threshold tested. BFCL v4 shows 16.7%, of which 16.4 points lie between independently presented tasks. UltraTool shows 0.3%, cleaner than real tools, so this is a property of BFCL and not of synthetic corpora as a class. Separately, 68.8% of raw BFCL rows and 85.6% of raw UltraTool rows are exact name-plus-description repeats, against 0.4% for real MCP, so any statistic computed over these releases without global deduplication measures repetition rather than tools. All figures regenerate from released scripts and a published seed.