MCP Scanner

Overview

Point it at an MCP endpoint. It reads every tool the server exposes, checks descriptions and schemas for planted instructions, then runs a live agent against the server with poisoned tool output to see whether the agent actually obeys.

The problem

An MCP server hands tool descriptions, schemas, and results straight into a model's context. Every one of those is a place to plant instructions, and the agent has no way to tell them apart from the user's own request. I wanted to know whether a given server was safe to connect to — and to answer it with a number rather than a vibe.

Approach

The decision worth defending is that the scanner drives the MCP connection itself instead of using a provider's server-side connector. A connector executes tool calls on your behalf, which makes it impossible to tamper with a tool result — and tampering with tool results is the entire test, because that is exactly how indirect prompt injection arrives in production. The static pass is pure regex over descriptions and schemas: free, instant, deterministic, no model involved. The live pass runs a real agent loop against high-risk tools, fetches the genuine result, appends a payload carrying a canary token, and feeds it back. Detection is an exact string match on that canary rather than a model's judgement, which keeps it testable. Scans run through a queued worker at concurrency 1 — a live probe makes several sequential model calls and outlives any sensible HTTP timeout, and the free tier is metered per minute.

What it does

• Two passes — deterministic regex rules over tool schemas, then a live agent probe carrying a unique canary token
• Scored against a labelled corpus of 59 tool definitions: 27 graded attacks and 32 benign tools, 13 of them hard negatives written to trip a rule
• Rewrote the weakest rule from 44% to 100% precision, taking the detector from 25% to 9.4% FPR with recall unchanged
• A response suppressed by the provider's safety filter is recorded as inconclusive, never as safe
• 19 tests that need no network and no model quota — the model is injected as a dependency

What I learned

The first measurement came back at F1 0.727 with a 25% false-positive rate, and the cause was one rule: it flagged any URL at high severity, so it fired on more honest tools than hostile ones. Naming a URL is what a fetch tool is for. Rewriting it to key on what the description asks the model to *do* with the URL — an outbound verb beside it, an embed that loads without a click, a URL hardcoded as a parameter default — took that rule to 100% precision and the whole detector to F1 0.800, recall unchanged. Building the labelled corpus first is what made the fix findable at all.