CoolFace
Datasetpublic

kshitijthakkar/mcp-server-bench-gradio-optimized-full-bench

🔬 Gradio vs FastMCP Benchmark Report Generated: 2026-03-02T21:04:48.460108 Total scenarios: 337 Executive Summary echo: Fastmcp wins (100.9 vs 189.9 RPS, 1.88x difference) Gradio best config: concurrency_limit=1.0 fibonacci: Fastmcp wins (45.5 vs 55.0 RPS, 1.21x difference) Gradio best config: concurrency_limit=nan json_transform: Fastmcp wins (94.5 vs 165.4 RPS, 1.75x difference) Gradio best config: concurrency_limit=5.0 async_sleep: Fastmcp wins (94.0 vs… See the full description on the dataset page: https://huggingface.co/datasets/kshitijthakkar/mcp-server-bench-gradio-optimized-full-bench.

sourceHugging Facemitupdated 7mo agoView on Hugging Face
0likes36downloads
Dataset Card

🔬 Gradio vs FastMCP Benchmark Report

Generated: 2026-03-02T21:04:48.460108

Total scenarios: 337

Executive Summary

  • —echo: Fastmcp wins (100.9 vs 189.9 RPS, 1.88x difference)
  • —Gradio best config: concurrency_limit=1.0
  • —fibonacci: Fastmcp wins (45.5 vs 55.0 RPS, 1.21x difference)
  • —Gradio best config: concurrency_limit=nan
  • —json_transform: Fastmcp wins (94.5 vs 165.4 RPS, 1.75x difference)
  • —Gradio best config: concurrency_limit=5.0
  • —async_sleep: Fastmcp wins (94.0 vs 104.3 RPS, 1.11x difference)
  • —Gradio best config: concurrency_limit=10.0
  • —payload_echo: Fastmcp wins (97.4 vs 172.0 RPS, 1.77x difference)
  • —Gradio best config: concurrency_limit=10.0

Throughput (Requests/Second)

('fastmcp', 'http_api')('fastmcp', 'mcp_streamable')('gradio', 'http_api')('gradio', 'mcp_streamable')
('async_sleep', 1)16.1512.278.7214.2
('async_sleep', 10)104.32060.0794.02
('async_sleep', 25)54.87038.3559.43
('async_sleep', 50)49.0714.6734.1653.27
('echo', 1)189.8780.5311.4577.57
('echo', 10)79.6750.1353.64100.86
('echo', 25)78.6138.4439.1571.79
('echo', 50)63.1532.5334.4866.26
('fibonacci', 1)43.731.0612.0836.39
('fibonacci', 10)54.9943.9138.6545.5
('fibonacci', 25)53.3430.8934.545.02
('fibonacci', 50)45.3626.231.9943.81
('json_transform', 1)165.4261.5411.5876.65
('json_transform', 10)85.9635.9154.9794.48
('json_transform', 25)50.5434.8540.2768.97
('json_transform', 50)60.1229.8934.3265.31
('payload_echo', 1)17241.5811.4573.3
('payload_echo', 10)77.79051.8197.35
('payload_echo', 25)68.7931.1639.171.09
('payload_echo', 50)61.1831.234.2567.32

Latency p50 (ms)

('fastmcp', 'http_api')('fastmcp', 'mcp_streamable')('gradio', 'http_api')('gradio', 'mcp_streamable')
('async_sleep', 1)62.53278.089111.55666.027
('async_sleep', 10)85.0620150.179100.587
('async_sleep', 25)340.3390527.091320.992
('async_sleep', 50)718.0182516.251171.91695.627
('echo', 1)511.97381.0812.232
('echo', 10)86.744174.638162.62291.59
('echo', 25)235.09542.953519.866262.988
('echo', 50)519.9021329.381151.61515.254
('fibonacci', 1)21.50330.18166.3826.53
('fibonacci', 10)171.886205.011258.26215.707
('fibonacci', 25)441.508701.476611.039550.476
('fibonacci', 50)768.8121612.961281.011147.82
('json_transform', 1)5.99615.00479.59112.565
('json_transform', 10)62.526234.142157.18895.055
('json_transform', 25)322.956607.396502.211270.443
('json_transform', 50)536.4651419.951190.15507.173
('payload_echo', 1)5.26622.1180.57313.012
('payload_echo', 10)90.5210167.60194.079
('payload_echo', 25)214.665650.125526.828269.352
('payload_echo', 50)523.9931344.041184.89501.297

Gradio concurrency_limit Scaling

How does Gradio's throughput change as concurrency_limit increases?

1.05.010.0
('async_sleep', 1)10.40511.2711.2825
('async_sleep', 10)30.772.2971.0275
('async_sleep', 25)24.37545.837546.4825
('async_sleep', 50)30.392541.577540.7275
('echo', 1)43.787542.0533.84
('echo', 10)59.017573.087559.8225
('echo', 25)45.057551.4144.3075
('echo', 50)42.337545.2938.755
('fibonacci', 1)23.9523.142520.555
('fibonacci', 10)31.452540.26537.3475
('fibonacci', 25)31.067538.2235.7975
('fibonacci', 50)30.5436.4731.64
('json_transform', 1)43.48542.2636.2125
('json_transform', 10)55.4174.0666.2875
('json_transform', 25)40.02553.847549.5975
('json_transform', 50)34.087549.0845.295
('payload_echo', 1)36.20538.272541.5925
('payload_echo', 10)48.817566.6567.315
('payload_echo', 25)38.59551.0750.155
('payload_echo', 50)39.3846.10545.0975

Protocol Overhead: HTTP API vs MCP

Comparing latency of the same tool called via REST API vs MCP protocol:

http_apimcp_streamable
('fastmcp', 'async_sleep')301.488648.586
('fastmcp', 'echo')211.684514.737
('fastmcp', 'fibonacci')350.927637.407
('fastmcp', 'json_transform')231.986569.122
('fastmcp', 'payload_echo')208.611504.068
('gradio', 'async_sleep')920.038332.935
('gradio', 'echo')693.001246.16
('gradio', 'fibonacci')763.248506.714
('gradio', 'json_transform')690.337234.987
('gradio', 'payload_echo')710.962239.838

Error Rates

scenario_iderror_rate_pct
fastmcp_httpapi_fibonaccivu50_clunlimited0.037
gradio_mcpstreamable_echovu25cl1_noq0.023
gradio_mcpstreamable_payloadecho_vu25cl1_noq0.028
gradio_httpapi_echovu25_cl50.05
gradio_httpapi_fibonaccivu25_cl50.05
gradio_httpapi_payloadecho_vu50cl5_noq0.05
gradio_mcpstreamable_asyncsleep_vu25cl5_noq0.03
gradio_httpapi_echovu25_cl100.048
gradio_httpapi_fibonaccivu25cl10_noq0.056
gradio_httpapi_fibonaccivu50_cl100.066
gradio_httpapi_payloadecho_vu25cl10_noq0.056
gradio_mcpstreamable_asyncsleep_vu25cl10_noq0.028
gradio_httpapi_asyncsleep_vu25clunlimited_noq0.047

Resource Usage

server('mean', 'avg_cpu_pct')('mean', 'avg_memory_mb')('mean', 'peak_memory_mb')('max', 'avg_cpu_pct')('max', 'avg_memory_mb')('max', 'peak_memory_mb')
fastmcp05.6845.68406.156.15
gradio06.149536.1495306.166.16

Benchmark Charts

[image]

[image]

[image]

[image]

Methodology

  • —Both servers use identical tool implementations (imported from shared_tools.py)
  • —Each scenario runs in an isolated server subprocess
  • —Warmup period excluded from measurements
  • —Load generated by async httpx workers (not external tools)
  • —MCP tests use full protocol lifecycle (initialize → call_tool)
  • —System metrics sampled every 1s via psutil

Benchmarks generated by [mcp-server-bench](https://github.com/Mandark-droid/mcp-server-bench)