Updated 2026-07-23. This article was first written against the
2025-11-25Model Context Protocol (MCP) revision. The2026-07-28revision removes protocol-level sessions, theinitializehandshake, and stream resumability. The four-layer architecture below still holds. The transport and idempotency sections have been rewritten for the stateless model, and the earlier WebSocket example has been replaced with Streamable HTTP. If you are migrating an existing server, read MCP Went Stateless: Your 2026-07-28 Migration Guide first, then come back here for the architecture.
Why production MCP servers fail (and demos do not)
Most MCP server implementations die in production not because the protocol is complex, but because engineers treat them like stateless API endpoints. They are not. An MCP server sits between your LLM and your critical infrastructure - databases, APIs, file systems - and must handle three concurrent failure modes: the LLM making nonsensical requests, your backend systems timing out, and the client dropping the request mid-execution.
The naive approach looks like this: wrap your existing REST API in MCP tool definitions, deploy it, watch it handle 50 requests fine, then watch it silently fail at request 51 when the LLM decides to call delete_database with a hallucinated parameter. Or worse, it succeeds but your audit logs show nothing, because you treated tool execution like a fire-and-forget operation.
The real problem is not implementing the protocol specification. That part is straightforward. The problem is that MCP servers operate in a fundamentally adversarial environment. Your LLM will generate invalid inputs. Your backend will fail at unpredictable times. Your network will partition. Traditional API clients fail fast and loud. LLM agents do something worse: they retry with slightly different parameters, creating cascading failures that look like success until you check your database three hours later.
Production MCP servers require thinking about idempotency, partial failure recovery, request validation as a security boundary, and observability that captures both the LLM's intent and the system's actual behavior. The gap between "works in demo" and "works at 3am when your database is slow" is where most implementations fail.
The right mental model: a protocol translator at a trust boundary
Stop thinking of MCP servers as tool registries. They are protocol translators that sit at a trust boundary. On one side you have an LLM that speaks in probabilistic tokens. On the other you have deterministic systems that require exact inputs. The MCP server's job is to maintain invariants across this boundary, and to degrade gracefully when either side violates expectations.
The correct mental model has three layers of responsibility. First, the protocol layer handles MCP message framing, version negotiation, and transport concerns. This layer is stateless and should never make decisions about business logic. It receives JSON-RPC requests and routes them to the appropriate handler.
Second, the validation and transformation layer. This is where you enforce the contract between probabilistic and deterministic systems. Every tool input gets validated against a schema, but not just for type correctness. It gets validated for semantic validity. An LLM might generate syntactically valid JSON that represents an impossible operation. This layer must catch "delete all users where role=admin" before it reaches your database, even if your tool schema technically allows it.
Third, the execution layer, where actual work happens. This layer must be designed for partial failure and idempotency. When an LLM calls a tool that performs multiple operations, you need to know which ones succeeded so you can resume or roll back. Unlike REST APIs where clients handle retries, MCP servers often need to manage this themselves, because the LLM will retry with modified parameters rather than identical requests.
The key invariant: at any point in the execution pipeline, you should be able to reconstruct what the LLM intended, what actually happened, and whether the two diverged. This means logging both the original request and the transformed, validated version that hit your backend. When things go wrong, and they will, this audit trail is the only way to tell whether the failure was in the LLM's reasoning, your validation logic, or the backend system. Debugging context rather than prompts depends entirely on having kept both versions.
Think of an MCP server as a runtime type system for LLM behavior. Static schemas catch obvious errors, but production systems need runtime invariant checking, automatic rollback on constraint violations, and detailed provenance tracking. The server is not just executing tools. It is maintaining the integrity of the boundary between learned and programmed behavior.
Production MCP server architecture: four layers
A production MCP server architecture separates concerns across four primary components: the protocol handler, the tool registry, the execution engine, and the observability layer. Each has distinct responsibilities and failure modes.
flowchart LR
subgraph CLIENT["Client"]
LLM["LLM Client"]
end
subgraph SERVER["MCP Server"]
PH["Protocol Handler"]
TR["Tool Registry"]
VL["Validation Layer"]
EE["Execution Engine"]
OBS["Observability"]
end
subgraph BACKEND["Backend"]
DB["Database"]
API["External APIs"]
FS["File Systems"]
end
LLM -->|"JSON-RPC"| PH
PH -->|"Route"| TR
TR -->|"Schema"| VL
VL -->|"Execute"| EE
EE -->|"Query"| DB
EE -->|"Call"| API
EE -->|"Access"| FS
EE -->|"Results"| PH
PH -->|"Response"| LLM
PH -.->|"Protocol events"| OBS
VL -.->|"Validation events"| OBS
EE -.->|"Metrics"| OBS
style LLM fill:#4A90E2,color:#FFFFFF
style PH fill:#7B68EE,color:#FFFFFF
style TR fill:#98D8C8,color:#2C2C2A
style VL fill:#FFD93D,color:#2C2C2A
style EE fill:#6BCF7F,color:#2C2C2A
style OBS fill:#C2185B,color:#FFFFFF
style DB fill:#95A5A6,color:#2C2C2A
style API fill:#95A5A6,color:#2C2C2A
style FS fill:#95A5A6,color:#2C2C2A
The request path runs left to right. The dotted edges are the point: observability receives events from the protocol, validation, and execution layers separately, rather than sitting underneath them as a single sink.
The protocol handler owns transport concerns. MCP defines two transports: stdio for local processes, and Streamable HTTP for everything else. Its only job is parsing MCP protocol messages, checking the protocol version, and routing requests. It does not interpret tool semantics. When a tools/call request arrives, it extracts the tool name and arguments, then delegates to the registry. Malformed JSON, unsupported protocol versions, and unknown methods get handled here and logged before they propagate.
Under the 2026-07-28 revision this layer holds no per-client state at all. There is no initialize handshake and no Mcp-Session-Id, so any request can land on any instance. Each request carries its own protocol version and client capabilities in _meta. That is a simplification for your infrastructure and a new obligation for your code: the version check that used to happen once per connection now happens on every request.
The tool registry maintains the catalog of available tools and their schemas. In simple implementations this is a static dictionary. In production it is dynamic: tools can be enabled or disabled based on client capabilities, feature flags, or runtime configuration. The registry returns not just tool definitions but also metadata - expected latency, cost estimates, required permissions, and whether the tool is idempotent. This metadata drives decisions in downstream layers. One constraint the stateless revision adds: the catalog must be a pure function of the caller's identity and capabilities. It cannot vary by connection, because there is no connection to vary by.
The validation layer enforces the contract between LLM-generated inputs and your backend systems. Schema validation is table stakes. Production validation includes semantic constraints (date ranges must be positive, user IDs must exist), cross-field dependencies (if action=delete, require confirmation=true), and rate limits per tool per client. This layer also handles parameter coercion. LLMs often generate strings where numbers are expected, or the reverse. Decide explicitly whether to coerce or reject, and log which one you did.
The execution engine is where state management complexity lives. For simple read-only tools this is straightforward: call the backend, return the result. For tools that mutate state you need deduplication keys, transaction boundaries, and partial failure recovery. The engine maintains an execution context that tracks which operations succeeded, the current retry count, and any intermediate state needed to resume or roll back.
sequenceDiagram
participant LLM as LLM Client
participant S as Protocol Handler
participant V as Validator
participant E as Execution Engine
participant B as Backend
LLM->>S: tools/call(name, args)
S->>V: validate(args, schema)
alt validation fails
V-->>S: ValidationError
S-->>LLM: error response
else validation succeeds
V->>E: execute(validated_args)
E->>B: perform operation
alt backend succeeds
B-->>E: result
E-->>S: success
S-->>LLM: tool result
else backend fails
B-->>E: error
E->>E: check if retryable
alt retryable
E->>B: retry with backoff
else not retryable
E-->>S: permanent failure
S-->>LLM: error response
end
end
end
Tracing a single tools/call end to end. The three nested branches are where each guarantee is enforced: reject before execution, retry only what is safe to repeat, and fail permanently rather than loop.
The observability layer sits orthogonal to execution flow. Every request generates structured events: protocol messages received, validation outcomes, execution start and end, backend latencies, and final results. These events flow to your logging infrastructure, not as debug prints, but as structured JSON with a consistent schema. Critical fields include request_id, tool_name, execution_time_ms, validation_errors, backend_status, and result_summary. When debugging production failures you need to correlate LLM behavior with system outcomes across potentially thousands of concurrent requests.
State management deserves explicit architectural attention. Most MCP servers need to track in-flight requests and execution history for deduplication. Choose your state store based on your deployment model. Multi-instance deployments need Redis or similar for shared state, and under the stateless revision that is effectively every deployment, because you can no longer pin a client to an instance. Do not use your application database for this. State store failures should not cascade into backend failures.
Authentication and authorization now happen per request, at the protocol boundary. MCP servers are formally OAuth 2.1 resource servers under the current revision. The protocol handler validates the token and its audience. The execution engine checks whether that identity can invoke the requested tool with the provided arguments. This separation matters because authorization often depends on argument values, not just tool names. A client might be allowed to read user profiles, but only for users in their own organization. Getting that boundary wrong is the most common route to cross-tenant data exposure.
Implementing a stateless MCP server in Python
The protocol handler: Streamable HTTP with no session
The MCP specification defines JSON-RPC 2.0 over stdio and Streamable HTTP. Here is a handler that covers the error cases that actually bite, including per-request version negotiation:
import jsonfrom typing import Any, Dict, Optionalimport structlogfrom fastapi import FastAPI, Request, Responselogger = structlog.get_logger()SUPPORTED_PROTOCOL_VERSIONS = {"2026-07-28", "2025-11-25"}PROTOCOL_VERSION_KEY = "io.modelcontextprotocol/protocolVersion"CLIENT_CAPABILITIES_KEY = "io.modelcontextprotocol/clientCapabilities"class MCPProtocolHandler: """Streamable HTTP transport. Stateless: any request can hit any instance.""" def __init__(self, tool_registry, execution_engine): self.registry = tool_registry self.engine = execution_engine async def handle_post(self, request: Request) -> Dict[str, Any]: try: body = await request.json() except ValueError: return self._error(None, -32700, "Parse error") request_id = body.get("id") method = body.get("method") params = body.get("params") or {} meta = params.get("_meta") or {} log = logger.bind( request_id=request_id, method=method, # Routing headers: a gateway sets these so it can route without # parsing the JSON body. mcp_method=request.headers.get("Mcp-Method"), mcp_name=request.headers.get("Mcp-Name"), ) # There is no handshake, so the version check runs on every request. version = meta.get(PROTOCOL_VERSION_KEY) if version not in SUPPORTED_PROTOCOL_VERSIONS: log.warning("mcp_unsupported_protocol_version", requested=version) return self._error( request_id, -32000, "UnsupportedProtocolVersionError", data={ "requested": version, "supported": sorted(SUPPORTED_PROTOCOL_VERSIONS), }, ) capabilities = meta.get(CLIENT_CAPABILITIES_KEY) or {} log.info("mcp_request_received") if method == "server/discover": return self._discover(request_id) if method == "tools/list": return self._list_tools(request_id, capabilities) if method == "tools/call": return await self._call_tool(request_id, params, request.headers, log) return self._error(request_id, -32601, f"Method not found: {method}") def _discover(self, request_id) -> Dict[str, Any]: return self._result( request_id, { "protocolVersions": sorted(SUPPORTED_PROTOCOL_VERSIONS), "capabilities": {"tools": {"listChanged": False}}, }, ) def _list_tools(self, request_id, capabilities: Dict) -> Dict[str, Any]: # A pure function of the caller's capabilities. It must not vary by # connection, because there is no connection to vary by. return self._result( request_id, {"tools": self.registry.list_tools(capabilities)} ) async def _call_tool(self, request_id, params: Dict, headers, log) -> Dict[str, Any]: result = await self.engine.execute( tool_name=params.get("name"), arguments=params.get("arguments") or {}, # A retried call arrives with a NEW request id, so the dedup key # has to come from the client, not from the envelope. idempotency_key=headers.get("Idempotency-Key"), ) if "error" in result: log.warning("mcp_tool_error", error=result["error"]) return self._error(request_id, -32000, result["error"]) return self._result(request_id, result) @staticmethod def _result(request_id, result: Any) -> Dict[str, Any]: return {"jsonrpc": "2.0", "result": result, "id": request_id} @staticmethod def _error( request_id, code: int, message: str, data: Optional[Dict] = None ) -> Dict[str, Any]: error: Dict[str, Any] = {"code": code, "message": message} if data is not None: error["data"] = data return {"jsonrpc": "2.0", "error": error, "id": request_id}The validation layer: normalize first, then allowlist
This is where most implementations get lazy. Schema validation alone is insufficient, and the order of operations matters more than the rules themselves. Normalization has to run before validation, or the validation is checking a string the backend will never see:
import reimport unicodedataimport urllib.parsefrom typing import Any, Dict, Optional, Tuplefrom pydantic import BaseModel, Field, field_validatordef normalize_string_parameter(value: str) -> str: """Run this BEFORE schema validation, not after. An LLM will encode a path traversal as percent-escapes or as a Unicode compatibility codepoint. A regex that runs on the raw string misses it. The filesystem does not. """ decoded = urllib.parse.unquote(value) return unicodedata.normalize("NFKC", decoded)# Named, reviewed queries. The model picks a name and supplies parameters.# It never composes SQL.## A keyword blocklist (rejecting DROP, DELETE, TRUNCATE) is not a control. It# rejects a legitimate `deleted_at` column and misses anything obfuscated. Do# not accept LLM-authored SQL and then try to sanitize it.QUERY_TEMPLATES = { "orders_by_customer": ( "SELECT id, total, created_at FROM orders " "WHERE customer_id = :customer_id ORDER BY created_at DESC LIMIT :limit" ), "order_detail": "SELECT * FROM order_items WHERE order_id = :order_id",}# In production this comes from your authorization system, not a literal.QUERY_PERMISSIONS = { "reporting-client": {"orders_by_customer", "order_detail"}, "support-client": {"order_detail"},}class QueryDatabaseArgs(BaseModel): query_name: str parameters: Dict[str, Any] = Field(default_factory=dict) limit: int = Field(default=100, le=1000) timeout_seconds: int = Field(default=30, le=300) @field_validator("query_name") @classmethod def known_query(cls, v: str) -> str: if v not in QUERY_TEMPLATES: raise ValueError( f"Unknown query_name: {v}. Allowed: {sorted(QUERY_TEMPLATES)}" ) return vasync def validate_query_tool( args: QueryDatabaseArgs, client_id: str) -> Tuple[bool, Optional[str]]: template = QUERY_TEMPLATES[args.query_name] required = set(re.findall(r":(\w+)", template)) - {"limit"} missing = required - set(args.parameters) if missing: return False, f"Missing parameters for {args.query_name}: {sorted(missing)}" if args.query_name not in QUERY_PERMISSIONS.get(client_id, set()): return False, f"Client is not permitted to run {args.query_name}" return True, Noneclass ToolValidator: def __init__(self, schemas: Dict[str, type]): self.schemas = schemas self.validators = {} def register_tool_validator(self, tool_name: str, validator_func) -> None: self.validators[tool_name] = validator_func async def validate( self, tool_name: str, arguments: Dict, client_id: str ) -> Tuple[bool, Optional[str], Dict]: schema = self.schemas.get(tool_name) if schema is None: return False, f"Unknown tool: {tool_name}", {} # Normalize first. Validating un-normalized input is the bypass. normalized = { key: normalize_string_parameter(value) if isinstance(value, str) else value for key, value in arguments.items() } try: validated = schema(**normalized) except Exception as e: return False, f"Schema validation failed: {e}", {} validator_func = self.validators.get(tool_name) if validator_func is not None: is_valid, error_msg = await validator_func(validated, client_id) if not is_valid: return False, error_msg, {} return True, None, validated.model_dump()The same allowlist discipline applies to any tool that takes a free-text parameter destined for an interpreter, not just SQL. The broader threat model, including what happens when tool output carries instructions back to the model, is covered in designing secure MCP servers.
The execution engine: concurrency, retries, and deduplication
The engine owns three guarantees at once: it never runs more of a tool than the backend can take, it retries only what is safe to retry, and it runs a duplicated call exactly once.
import asyncioimport hashlibimport jsonimport timefrom dataclasses import dataclassfrom typing import Any, Dictclass RetryableError(Exception): """A backend failure that is safe to repeat: timeout, 503, connection reset."""def content_dedup_key(tool_name: str, arguments: Dict[str, Any]) -> str: """Fallback for when the client sends no Idempotency-Key. Weaker than a client-supplied key: two legitimately distinct calls with identical arguments collide. Use a short time-to-live, and only for tools where a repeated identical call is far more likely to be a retry than a real second request. """ stable_args = json.dumps(arguments, sort_keys=True, separators=(",", ":")) return hashlib.sha256(f"{tool_name}:{stable_args}".encode()).hexdigest()@dataclassclass ExecutionContext: tool_name: str arguments: Dict[str, Any] dedup_key: str started_at: float attempt: int = 1 max_attempts: int = 3class ExecutionEngine: def __init__( self, metrics_client, state_store, max_concurrent_per_tool: Dict[str, int] ): self.metrics = metrics_client self.state = state_store self.tool_executors: Dict[str, Any] = {} self.semaphores = { tool: asyncio.Semaphore(limit) for tool, limit in max_concurrent_per_tool.items() } def register_executor(self, tool_name: str, executor_func) -> None: self.tool_executors[tool_name] = executor_func async def execute( self, tool_name: str, arguments: Dict, idempotency_key: str = None ) -> Dict: executor = self.tool_executors.get(tool_name) if executor is None: return {"error": f"No executor registered for {tool_name}"} ctx = ExecutionContext( tool_name=tool_name, arguments=arguments, dedup_key=idempotency_key or content_dedup_key(tool_name, arguments), started_at=time.time(), ) semaphore = self.semaphores.get(tool_name) if semaphore is None: return await self._execute_deduplicated(executor, ctx) async with semaphore: return await self._execute_deduplicated(executor, ctx) async def _execute_deduplicated(self, executor, ctx: ExecutionContext) -> Dict: result_key = f"result:{ctx.dedup_key}" cached = await self.state.get(result_key) if cached is not None: self.metrics.increment("execution.dedup_hit", tags=[f"tool:{ctx.tool_name}"]) return cached # Reserve before executing. Reading the cache and then running is a # check-then-act race: two concurrent duplicates both see an empty # cache, and both execute. lock_ttl = int(self._get_timeout(ctx.tool_name)) + 5 if not await self.state.acquire_lock(result_key, timeout=lock_ttl): return {"error": "Duplicate request already in flight", "retryable": True} try: result = await self._execute_with_retry(executor, ctx) if "error" not in result: await self.state.set(result_key, result, ttl=3600) return result finally: await self.state.release_lock(result_key) self.metrics.histogram( "execution.duration_ms", (time.time() - ctx.started_at) * 1000, tags=[f"tool:{ctx.tool_name}"], ) async def _execute_with_retry(self, executor, ctx: ExecutionContext) -> Dict: last_error = "unknown" for attempt in range(1, ctx.max_attempts + 1): ctx.attempt = attempt try: return await asyncio.wait_for( executor(ctx.arguments), timeout=self._get_timeout(ctx.tool_name), ) except asyncio.TimeoutError: last_error = "Execution timeout" self.metrics.increment( "execution.timeout", tags=[f"tool:{ctx.tool_name}", f"attempt:{attempt}"], ) except RetryableError as e: last_error = str(e) except Exception as e: # Not retryable. Repeating it just burns the budget. return {"error": f"Execution failed: {e}"} if attempt < ctx.max_attempts: await asyncio.sleep(2 ** attempt) return {"error": f"Max retries exceeded. Last error: {last_error}"} def _get_timeout(self, tool_name: str) -> float: timeouts = { "query_database": 30.0, "fetch_url": 10.0, "generate_report": 120.0, } return timeouts.get(tool_name, 60.0)The state store: shared, JSON-serialized, lock-capable
Multi-instance deployments require distributed coordination, and the 2026-07-28 spec removes the in-protocol session model entirely, so a shared state store stops being optional:
import jsonfrom typing import Any, Optionalimport redis.asyncio as redisclass RedisStateStore: """JSON, not pickle. `pickle.loads` on anything that comes back from Redis is remote code execution the moment an attacker can reach Redis or influence a key. The convenience of pickling arbitrary objects is not worth it for a store that holds tool results. """ def __init__(self, redis_url: str): self.redis = redis.from_url(redis_url, decode_responses=True) async def get(self, key: str) -> Optional[Any]: data = await self.redis.get(key) return json.loads(data) if data is not None else None async def set(self, key: str, value: Any, ttl: int = 3600) -> None: await self.redis.set(key, json.dumps(value), ex=ttl) async def incr(self, key: str, ttl: Optional[int] = None) -> int: count = await self.redis.incr(key) if ttl is not None and count == 1: await self.redis.expire(key, ttl) return count async def incr_float(self, key: str, amount: float) -> float: return float(await self.redis.incrbyfloat(key, amount)) async def acquire_lock(self, resource: str, timeout: int = 10) -> bool: """Atomic reserve. SET NX succeeds for exactly one caller.""" return bool(await self.redis.set(f"lock:{resource}", "1", nx=True, ex=timeout)) async def release_lock(self, resource: str) -> None: await self.redis.delete(f"lock:{resource}")Counters belong on the store interface, not on the caller. If CostTracker reaches through to self.state.redis directly, it silently stops working the moment you swap in an in-memory store for local development.
Cost tracking and rate limiting
from typing import Dict, Tupleclass CostTracker: def __init__( self, state_store, budget_per_client: Dict[str, float], default_budget: float = 10.00, ): self.state = state_store self.budget_per_client = budget_per_client self.default_budget = default_budget self.cost_per_tool = { "query_database": 0.01, # US dollars "call_external_api": 0.05, "generate_report": 0.10, } async def check_and_record(self, client_id: str, tool_name: str) -> Tuple[bool, str]: minute_key = f"ratelimit:{client_id}:{tool_name}:minute" minute_count = await self.state.incr(minute_key, ttl=60) if minute_count > self._get_rate_limit(tool_name): return False, "Rate limit exceeded" cost = self.cost_per_tool.get(tool_name, 0.01) daily_cost = await self.state.incr_float(f"cost:{client_id}:daily", cost) if daily_cost > self.budget_per_client.get(client_id, self.default_budget): return False, "Budget exceeded" return True, "" def _get_rate_limit(self, tool_name: str) -> int: limits = { "query_database": 60, # per minute "call_external_api": 30, "generate_report": 10, } return limits.get(tool_name, 100)Wiring it together
# `metrics` is any client exposing increment() and histogram().# `registry` is your tool catalog, exposing list_tools(capabilities).# `run_named_query` is the executor for the query_database tool.state_store = RedisStateStore(REDIS_URL)engine = ExecutionEngine( metrics_client=metrics, state_store=state_store, max_concurrent_per_tool={ "query_database": 20, "call_external_api": 10, "generate_report": 2, },)engine.register_executor("query_database", run_named_query)handler = MCPProtocolHandler(tool_registry=registry, execution_engine=engine)app = FastAPI()@app.post("/mcp")async def mcp_endpoint(request: Request) -> Response: payload = await handler.handle_post(request) return Response(content=json.dumps(payload), media_type="application/json")MCP server failure modes and how to prevent them
Silent failure loops from unhelpful error messages
The silent failure mode is the most dangerous. An LLM calls a tool, the tool returns an error, but the error message is too generic for the LLM to recover from. The agent retries with slightly modified parameters, fails again, and enters a loop. You will not see this in metrics, because each individual request succeeds at the protocol level. It returns valid JSON. Meanwhile your backend is getting hammered with variations of an impossible request.
Prevention requires error messages that are both machine-readable and actionable. Do not return {"error": "Invalid input"}. Return:
{ "error": { "code": "INVALID_DATE_RANGE", "message": "Start date must be before end date", "details": { "provided_start": "2024-03-15", "provided_end": "2024-03-10", "constraint": "start_date < end_date" }, "suggestions": [ "Swap start_date and end_date values", "Verify date format is YYYY-MM-DD" ] }}The suggestions array is not decoration. It is the difference between an agent that corrects itself on the next call and one that loops until it hits your rate limit.
Cost explosions from parallel tool calls
Cost explosions happen when you do not account for LLM behavior patterns. An agent decides to analyze 10,000 documents. Instead of calling your analyze_document tool 10,000 times sequentially, it generates a parallel batch of 500 calls, because that is what it learned is efficient. Your execution engine dutifully spawns 500 concurrent operations and overwhelms your backend. Your database connections saturate, requests start timing out, the LLM sees failures and retries, and now you have 1,000 concurrent requests.
The fix is the max_concurrent_per_tool semaphore map already wired into ExecutionEngine above. Set the limit from what your backend can absorb, not from what the agent asks for. The generate_report limit of 2 in the wiring example is deliberate: expensive tools should have limits that look uncomfortably low.
Duplicate operations when retries carry a new request id
State management failures appear as duplicate operations. An agent calls create_user, the operation succeeds, the response is lost, the LLM retries, and now you have two users.
The original version of this article recommended hashing the tool name and arguments together with the client's request_id. Under the 2026-07-28 revision, that no longer works. Stream resumability is gone, so a dropped response is not resumed. The client re-issues the call as a brand new request with a new request id. Any key derived from the request id therefore changes on every retry, and the deduplication silently stops deduplicating.
Two things fix it, and both are shown in ExecutionEngine above. The key must come from the client as a stable Idempotency-Key that survives the retry, and the reservation must be atomic, taken before execution rather than written after it. A check-then-act cache lets two concurrent duplicates through. The full treatment of this shift, including why a side-effecting tool should reject any call that arrives without a key, is in At-Least-Once Tool Execution.
Validation bypass through encoded parameters
Validation bypass through parameter injection is common. LLMs learn to encode instructions in parameter values. You validate that a filename parameter does not contain path traversal characters, but the LLM generates ../../../../etc/passwd encoded as Unicode escapes or percent-encoding. Your validation regex misses it. Your filesystem does not.
The defense is normalize_string_parameter, and the reason it appears inside ToolValidator.validate above rather than as a standalone utility is the whole point. Normalization that runs after validation, or that a caller can forget to run, is not a control. It has to be on the path every input takes.
Observability gaps that only appear under production load
Observability gaps manifest as "it worked in staging." Staging does not have 50 concurrent clients, does not have clients that hold long-running workflows open for hours, and does not have the variety of inputs production LLMs generate. Your logs show individual requests succeeding but miss the pattern of repeated failures for a specific client and tool combination.
Structured logging with correlation IDs across the entire request lifecycle is non-negotiable:
import structlogdef setup_logging() -> None: structlog.configure( processors=[ structlog.contextvars.merge_contextvars, structlog.processors.add_log_level, structlog.processors.TimeStamper(fmt="iso"), structlog.processors.JSONRenderer(), ] )# Bind once, at the edge of the request. merge_contextvars picks these up on# every later log call in the same task, including inside the execution engine.# A bare ContextVar will not work here: merge_contextvars only reads the# structlog.contextvars registry, not arbitrary context variables.structlog.contextvars.bind_contextvars(request_id=request_id, tool=tool_name)structlog.get_logger().info("tool_executed", duration_ms=duration_ms)Production MCP server checklist
Work down this list in order. Each item assumes the ones above it.
Before you write a single tool executor:
- Structured JSON logging configured with
merge_contextvars, bound once per request - Shared state store (not your application database) reachable from every instance
- Per-request protocol version check against an explicit supported set, failing closed
-
server/discoverimplemented so clients stop probing blindly
Before you expose any read tool:
- Every tool schema defined, with normalization running before validation
- Free-text parameters bound to allowlisted templates, never composed into an interpreter
- Per-tool, per-client rate limits enforced
- Per-tool concurrency semaphores set from backend capacity
Before you expose any write tool:
-
Idempotency-Keyrequired, and calls without one rejected - Reservation taken atomically before execution, not written after
- Retry classification explicit:
RetryableErrorfor safe repeats, everything else fails once - Errors returned with a code, a constraint, and a suggestion the model can act on
- Authorization checked against argument values, not just tool names
Before you call it production:
- Dashboards showing tool success rate segmented by error type, not just aggregate availability
- Both the original and the post-validation arguments recoverable for any request
- Circuit breakers on every backend dependency
- Cost budget enforced per client, with a defined behavior when it is exhausted
If you are building for non-LLM clients as well - CLIs, IDEs, pipelines - the same list applies, minus the assumption that the caller will retry with modified parameters.
Summary
Production MCP servers are protocol translators at a trust boundary, not simple API wrappers. The architecture that works separates protocol handling, validation, execution, and observability into distinct layers with clear failure semantics. Validation must be semantic, not just schematic, and it must normalize before it validates. Execution must handle partial failures, deduplicate atomically, and enforce rate limits. Observability must capture both LLM intent and system behavior.
The failure modes that matter in production are silent failure loops, cost explosions, duplicate operations, validation bypass, and correlation gaps in logs. Each has a specific technical mitigation, but they share a common pattern: treat the LLM as an adversarial client that will eventually explore every edge case in your system.
The stateless revision does not change that conclusion. It sharpens it. Every guarantee the protocol used to provide at connection setup is now a guarantee you provide per request, in your own code, or not at all.
References
- Model Context Protocol. Specification. modelcontextprotocol.io/specification
- Model Context Protocol. Transports. modelcontextprotocol.io/docs/concepts/transports
- Python Software Foundation. pickle - Python object serialization (security warning). docs.python.org/3/library/pickle.html
- OWASP. Input Validation Cheat Sheet. cheatsheetseries.owasp.org
- Unicode Consortium. UAX #15: Unicode Normalization Forms. unicode.org/reports/tr15
- Pydantic. Validators. docs.pydantic.dev/latest/concepts/validators
- structlog. Context Variables. structlog.org/en/stable/contextvars.html
- Redis. SET command (NX, EX options). redis.io/docs/latest/commands/set
Related Articles
More Articles
- Model Context Protocol (MCP): Architecture, Tradeoffs, and Production Realities
- Can MCP Replace Memory Systems? A Critical Analysis
Follow for more technical deep dives on AI/ML systems, production engineering, and building real-world applications:
