Zed benchmarks 25 approaches to MCP tool search
A classical retrieval baseline beating neural methods changes how developers should think about MCP tool discovery — embeddings and LLM routing may be unnecessary complexity for most setups. Builders implementing agent tool selection can adopt weighted BM25 locally without API costs or latency. The benchmark size of 536 tools offers a concrete reference point for comparing their own retrieval setups.
Zed tested 25 approaches to MCP tool search, including decision models, LLMs and embeddings. The benchmark runs 150 search queries against 536 tools. The team settled on weighted BM25 with a few normalization rules. This approach is fast, runs locally, and requires no model or API calls.
We tested 25 approaches to MCP tool search, including decision models, LLMs, and embeddings. Our benchmark runs 150 search queries against 536 tools.
We settled on weighted BM25 with a few normalization rules. Fast, local, no model or API calls. https://t.co/CfITC8lZRl