MCP Explained Part 6: Evaluation

TL;DR: Evaluating an MCP server is more than checking whether tools return correct JSON. Each result emerges from the joint reasoning loop between an LLM and the server - planning, tool selection, pagination, and follow-ups all shape the final outcome.

To capture this complexity, the Blockscout MCP server is tested through two complementary methods: automated evaluation with Gemini CLI, which compares structured agent outputs against blockchain ground truth, and cross-agent benchmarking across Claude Desktop, ChatGPT, LeChat, and Gemini CLI.

Together, these approaches measure not just correctness, but how effectively different LLMs use the MCP ecosystem to perform real blockchain analysis.

Challenges in Evaluating MCP Servers

Simply checking that an MCP tool was invoked and returned the “right JSON” after an API call is not enough. An MCP server always operates within a loop involving the LLM and its agent: reasoning, planning, tool selection, pagination, and follow-up calls all influence the final output.

In other words, what you evaluate is not an isolated endpoint, but a reasoning-plus-tools ecosystem.

Why Full Evaluation is Hard

Nondeterministic tool calls

Invoking the same prompt within the same agent multiple times does not guarantee the same execution path. Some LLMs may decide to skip a call altogether or select a different tool, even if the input and environment are identical. The effect is especially strong for rare or narrowly scoped tools, which the model may overlook depending on its internal reasoning trace.

Free-form natural-language answers

Agents usually respond in open-ended text rather than structured data. Their phrasing, ordering of facts, and level of detail vary from run to run. This makes it difficult to define a clear algorithmic rule for “correctness”.

Uneven domain understanding

Different LLMs possess different mental models of blockchain data and operations. Some can correctly map tool descriptions to relevant analytical steps, while others fail to recognize the connection.

Dependence on agent-level instructions

System prompts and runtime policies strongly affect exploration depth. One agent may iterate through multiple tool calls, fetching several pages of results to build a comprehensive view. Another may stop early, summarize partial data, and request user confirmation.

As a result, the breadth and reliability of blockchain analysis vary widely even with the same underlying MCP server.

Using multiple MCP servers

Another complication arises from using multiple MCP servers simultaneously. When several servers expose tools with similar names or overlapping descriptions, interference may occur - the model can confuse their purposes or combine tool calls from different servers unpredictably.

However, such cross-server interactions were deliberately excluded during the design stage of the Blockscout MCP server. Its instruction set and tool descriptions were developed under the assumption of operating in isolation, without considering environments where multiple MCP servers coexist.

Meaningful Testing

For the Blockscout MCP server, meaningful testing should be performed only within agents that realistically represent user scenarios. While environments such as Cursor or Claude Code make sense for developer workflows, most end users are expected to interact through general-purpose chat platforms like Claude.ai, ChatGPT, or LeChat, since blockchain analysts are not necessarily programmers.

The first integration tests were conducted in Claude Desktop in June 2025, when it was the only general-purpose agent allowing MCP configuration. The evaluation revealed that Claude Desktop ignored the server-side instruction set during initialization.

To address this, a technical placeholder tool __unlock_blockchain_analysis__ was added (see Part 8: Server instructions). After this adjustment, the agent was able to complete the entire testing suite.

Automated MCP Evaluation via Gemini CLI

Manual checks of the MCP server are fine for one-off experiments, but they don’t scale. Any change to server instructions, tool descriptions, or response formats forces you to repeat the same scenario in an agent UI.

A reproducible way to run the same prompts is needed, capture an agent’s step chain (reasoning → tool choice → pagination → follow-ups), and compare the result with a ground truth.

This matches the idea that an MCP server is not an “isolated endpoint,” but a “reasoning + tools” ecosystem where the outcome depends on the model’s plan and the server’s guidance.

Why Gemini CLI

As the “engine,” Gemini CLI was chosen because it:

• Lets us query Gemini 2.5 Pro at this stage of the MCP server development without paying for tokens;
• Works as a command-line tool (good for scripts and integrating with other agents);
• Can override system instructions, which could in the future help when the default “coder” frame blocks blockchain-analysis behavior.

Unified agent output format

To check answers automatically, we standardize how the agent returns results. The agent prints JSON with three parts: the original query, a step-by-step trace, and the final answer.

This format is inspired by Schema Guided Reasoning: the structure guides the model and simplifies parsing. The final value must be a simple, easy-to-parse form - a number, a transaction hash, a list of values, or another basic data structure - not a free-form paragraph.

This makes automatic verification and comparison with ground truth straightforward.

{
  "original_query": "The exact user question/request",
  "steps": [
    // Array of analysis steps - see step types below
  ],
  "response": {
    "final": "Simple parseable answer OR null if is_error is true",
    "is_error": false,
    "error_type": null,
    "confidence": "high|medium|low", 
    "comments": "Detailed explanation supporting the final answer or error description"
  }
}

Each steps item is one of three types.

web search

{
  "type": "web_search",
  "data": {
    "query": "search terms used",
    "result_summary": "Key findings from the search",
    "status": "success|error"
  }
}

tool call

{
  "type": "tool_call",
  "data": {
    "tool": "exact_tool_name",
    "args": {actual_parameters_passed},
    "result_summary": "Brief description of what the tool returned",
    "status": "success|error"
  }
}

reasoning

{
  "type": "reasoning",
  "data": {
    "text": "Clear explanation of your reasoning, what you need to do next, or analysis of previous results"
  }
}

Ground Truth Methodology

The idea is as follows:

1. The evaluation system holds a set of prompts for Gemini CLI and a “source of truth” for each prompt:
   • a static value - if the fact is stable (for example, the address for a given ENS name);
   • a Python script - for all other cases. The script pulls data directly from blockchain/Blockscout to form the current ground truth.
2. Prompts are written to minimize drift between running the script and calling the model.
3. For each prompt we run:
   • Gemini CLI: gemini -p "query" (we expect the JSON above on stdout);
   • a parser that extracts response.final and compares it with ground truth.
4. If they do not match, we analyze only what’s available in steps - the short result_summary for each call - to see which tools ran and which intermediate conclusions the agent made.
5. Each prompt can be repeated several times to record repeatability.
6. The result is a report showing how well Gemini CLI, with the current MCP server version, can analyze blockchain activity. Comparing two reports lets us see how changes in MCP instructions, tool descriptions, or tool output formats affect the agent’s analysis quality.

Current progress

We have a base set of 14 prompts that checks two things:

• each MCP tool is distinguishable and callable by the agent in a real session;
• the tool’s response structure is correctly “understood” by the model (including pagination and notes).

What’s next

If this testing approach proves successful, we can build a benchmark on top of it to show how well agents can analyze blockchain activity.

Comparative Testing Across LLM Agents

Beyond the automated evaluation plans, in mid-October 2025 we ran a manual check of how different general-purpose agents reason about on-chain activity:

• Claude Desktop (Sonnet 4.5, Extended Thinking)
• ChatGPT Web (GPT 5 Extended Thinking), MCP Server enabled via developer mode
• LeChat (Think)
• Gemini CLI (Gemini 2.5 Pro)

☝ Strictly speaking, Gemini CLI is not a general-purpose agent, but neither Gemini Web nor Google AI Studio lets you configure an MCP server.

Method

• Separate projects were created in Claude Desktop and ChatGPT with the following special instructions:

You are a senior analyst specializing in Ethereum-blockchain activities with almost ten years of experience. You have deep knowledge of Web3 applications and protocols.

Each prompt was run in a new chat inside the project.

• In LeChat, each prompt was prefixed with the same phrase; every new prompt started a new chat.
• For Gemini CLI we used a GEMINI.md file with instructions equivalent to Blockscout X-Ray; each prompt ran in a new session.

Scoring legend

• OK: correct and complete answer;
• FAIL: incorrect;
• NA: the agent did not propose a workable plan / did not finish the task.

Evaluation results

Notes and observations

• On the seemingly trivial prompt “What address balance of ens.eth?” Gemini CLI converted wei → ETH incorrectly and showed a wrong balance.
• While exploring the Redstone transaction, neither LeChat nor Gemini CLI tried to proactively detect the MUD-specific context of the tx (the domain context stayed “off-screen”).
• For the question about funds moved by a Gnosis Chain validator, LeChat ignored ERC-20 transfers and looked only at native transfers.
• In the cross-chain message task, LeChat did not propose a workable strategy to fetch them.
• In the NFT count for 🇵🇱pl.eth, LeChat dropped the flag emoji during ENS → address resolve, so it failed to find the hash.
• Both LeChat and Gemini CLI misinterpreted the claimTo function during contract analysis.