mcp-probe
In DevelopmentScripted checks for MCP servers: describe the calls in YAML, get pass/fail in CI
Role: Solo developer
Solo · Since Nov 2026 · Public repo
A command-line tool that starts an MCP server over stdio, talks to it like a client and reports which checks passed. It exits non-zero when something breaks, so it can gate a pull request.
Problem
I kept shipping MCP tools and had no cheap way to notice that a refactor changed a tool's output or stopped rejecting bad input. A hung tool is the worst case: a client that only waits for a failure cannot tell it from a clean error.
Solution
A YAML file names the server command and the calls to make, each with an expectation: it must succeed or fail, and optionally its text must contain a string, match a pattern or equal a golden value. Built-in checks cover the handshake, tool schemas, unknown tools and whether the server still answers at the end. A timeout is its own failure, so an expected error cannot be satisfied by a hang.
Not done yet
stdio only and tools only: no HTTP transport, resources or prompts. It compares the first text block, not structured output. It does not validate every message against the protocol schema and it is not published to npm.
Architecture highlights
- A failed call can be a thrown protocol error or an isError result depending on the server, and the probe accepts both, so it does not fail servers that are correct
- A call that never answers is reported as a timeout and never counts as an expected failure. There is a test with a server that hangs on purpose
- The tests start a small real MCP server and a second one that exits during the handshake, instead of mocking the client
- The example runs against the official reference server pinned to one version with install scripts disabled: 11 checks, all passing. That says the reference server behaves as the probe expects, not that the probe found bugs in it
- An independent audit found false passes in my own tool: a typo in the YAML or in a tool name counted as success, and the server inherited the whole parent environment. All fixed with tests that failed first, and the probe files are now strict