MCP server testing
Tools that check what an MCP server does: protocol conformance, manual calls, and agent task benchmarks over real servers. Security scanners are under Guardrails and security.
- MCP Conformance - Official test suite that connects to a running MCP server, or drives an MCP client, and checks its tools, resources, prompts, elicitation, transport, and wire messages against a spec version, reporting pass or fail per scenario; pre-1.0, Apache-2.0, with some contributions still MIT.
- MCP Inspector - Official UI for calling an MCP server’s tools by hand and reading the raw JSON-RPC traffic.
- MCPMark - Benchmark of tasks over real Filesystem, GitHub, Notion, PostgreSQL, and Playwright MCP servers, each run from a fresh copy of its environment and checked by a per-task verification script, with models called through LiteLLM; Apache-2.0.
- MCP-Universe - Salesforce framework and benchmark that runs agents against real MCP servers for maps, finance, GitHub, browser, and other tasks and grades their outputs with rule-based evaluators; Apache-2.0.
- MCP-Atlas - Runs an agent against 36 real MCP servers (20 without API keys) in a Docker sandbox on 500 public tasks and scores the final answer with an LLM judge on claim coverage, plus a diagnostic taxonomy of tool-call and cognitive failures; MIT.