Datasetpublic

kshitijthakkar/mcp-server-bench-gradio-optimized

🔬 Gradio vs FastMCP Benchmark Report Generated: 2026-03-02T13:04:10.857215 Total scenarios: 48 Executive Summary echo: Fastmcp wins (96.6 vs 176.5 RPS, 1.83x difference) Gradio best config: concurrency_limit=nan fibonacci: Fastmcp wins (43.5 vs 57.1 RPS, 1.31x difference) Gradio best config: concurrency_limit=nan async_sleep: Gradio wins (93.1 vs 80.2 RPS, 1.16x difference) Gradio best config: concurrency_limit=nan payload_echo: Fastmcp wins (82.5 vs 164.8… See the full description on the dataset page: https://huggingface.co/datasets/kshitijthakkar/mcp-server-bench-gradio-optimized.

sourceHugging Facemitupdated 7mo agoView on Hugging Face

0likes50downloads

Dataset Card

🔬 Gradio vs FastMCP Benchmark Report

Generated: 2026-03-02T13:04:10.857215

Total scenarios: 48

Executive Summary

—echo: Fastmcp wins (96.6 vs 176.5 RPS, 1.83x difference)
—Gradio best config: concurrency_limit=nan
—fibonacci: Fastmcp wins (43.5 vs 57.1 RPS, 1.31x difference)
—Gradio best config: concurrency_limit=nan
—async_sleep: Gradio wins (93.1 vs 80.2 RPS, 1.16x difference)
—Gradio best config: concurrency_limit=nan
—payload_echo: Fastmcp wins (82.5 vs 164.8 RPS, 2.0x difference)
—Gradio best config: concurrency_limit=nan

Throughput (Requests/Second)

	('fastmcp', 'http_api')	('fastmcp', 'mcp_streamable')	('gradio', 'http_api')	('gradio', 'mcp_streamable')
('async_sleep', 1)	15.76	14.41	8.54	14.23
('async_sleep', 10)	80.21	56.47	56.79	93.11
('async_sleep', 25)	42.1	40.13	37.4	56.76
('echo', 1)	176.5	72.37	11.16	72.12
('echo', 10)	79.84	43.33	49.37	96.56
('echo', 25)	62.5	39.44	37.87	70.43
('fibonacci', 1)	45.39	35.94	11.91	35.18
('fibonacci', 10)	57.08	47.93	37.39	43.5
('fibonacci', 25)	51.19	35.94	32.67	43.21
('payload_echo', 1)	164.84	80.1	11.62	70.56
('payload_echo', 10)	91.25	50.53	49.93	82.54
('payload_echo', 25)	66.84	41.57	37.2	61.16

Latency p50 (ms)

	('fastmcp', 'http_api')	('fastmcp', 'mcp_streamable')	('gradio', 'http_api')	('gradio', 'mcp_streamable')
('async_sleep', 1)	62.599	64.32	112.296	75.659
('async_sleep', 10)	105.196	159.981	155.612	102.057
('async_sleep', 25)	430.916	528.933	546.454	337.852
('echo', 1)	5.005	13.005	101.503	13.037
('echo', 10)	102.519	201.01	177.598	95.355
('echo', 25)	298.811	529.175	549.508	275.746
('fibonacci', 1)	20.998	26.535	66.875	27.521
('fibonacci', 10)	166.862	189.789	266.299	226.342
('fibonacci', 25)	463.179	591.419	639.153	573.479
('payload_echo', 1)	5.998	12	81.067	13.662
('payload_echo', 10)	95.531	175.658	175.569	110.116
('payload_echo', 25)	336.98	514.13	557.964	321.014

Gradio concurrency_limit Scaling

How does Gradio's throughput change as concurrency_limit increases?

Protocol Overhead: HTTP API vs MCP

Comparing latency of the same tool called via REST API vs MCP protocol:

	http_api	mcp_streamable
('fastmcp', 'async_sleep')	199.57	251.078
('fastmcp', 'echo')	135.445	247.73
('fastmcp', 'fibonacci')	217.013	269.248
('fastmcp', 'payload_echo')	146.17	233.929
('gradio', 'async_sleep')	271.454	171.856
('gradio', 'echo')	276.203	128.046
('gradio', 'fibonacci')	324.109	275.781
('gradio', 'payload_echo')	271.533	148.264

Error Rates

All scenarios completed with 0% error rate. ✅

Resource Usage

server	('mean', 'avg_cpu_pct')	('mean', 'avg_memory_mb')	('mean', 'peak_memory_mb')	('max', 'avg_cpu_pct')	('max', 'avg_memory_mb')	('max', 'peak_memory_mb')
fastmcp	0	6.14458	6.14458	0	6.15	6.15
gradio	0	6.14417	6.14417	0	6.15	6.15

Benchmark Charts

[image]

Methodology

—Both servers use identical tool implementations (imported from shared_tools.py)
—Each scenario runs in an isolated server subprocess
—Warmup period excluded from measurements
—Load generated by async httpx workers (not external tools)
—MCP tests use full protocol lifecycle (initialize → call_tool)
—System metrics sampled every 1s via psutil

Benchmarks generated by [mcp-server-bench](https://github.com/Mandark-droid/mcp-server-bench)