pxzleo/qwen3.8-27b-inference-benchmark-4090

Qwen3.8-27B Inference Benchmark on RTX 4090 48GB

中文说明 · GitHub benchmark repository

Structured performance and accuracy results for four real Qwen3.8-27B serving configurations on an NVIDIA RTX 4090 48 GB workstation. A dual-GPU llama.cpp BF16 reference additionally used an RTX 3090 24 GB.

This dataset is the analysis-friendly companion to the full benchmark repository. It publishes aggregate tables, 140 normalized per-request performance records, accuracy scores, sanitized environment and deployment metadata, and sample indices or prompt hashes. It does not publish full AIME/GPQA questions, prompts, predictions, chain-of-thought output, model weights, credentials, or personal filesystem paths.

Headline results

Identical code-review workload, approximately 8.4K prompt tokens, 512 generated tokens per request, temperature 0, reasoning enabled, and five measured rounds.

NInfer's four-way aggregate throughput was only 0.16% above q27 in this sample, so they should be treated as the same four-way throughput tier. q27 additionally recorded 278.42 aggregate tok/s at eight-way concurrency.

These small samples do not establish a stable accuracy ranking. Rerun scores used a larger output budget only for responses that were truncated in the original run.

Dataset contents

The canonical downloadable artifacts remain the requested CSV and JSONL files directly under data/. The data/viewer/ JSONL mirrors use one consistent format so Hugging Face can index all four configs with the JSON builder instead of applying one parser across mixed CSV and JSONL inputs.

Additional machine-readable metadata is under metadata/:

• —environment.json: OS, driver, GPU, and software versions, with unknown build fields explicitly null.
• —deployments.json: weight formats, KV types, context limits, concurrency, speculative settings, and project links.
• —benchmark_protocol.json: workload, validity rules, formulas, warm-up difference, and accuracy rerun policy.
• —sources.json: upstream dataset IDs, links, revision warning, and redistribution exclusions.
• —manifest.json: file sizes and SHA-256 checksums for the published data and metadata.

Load with Datasets

from datasets import load_dataset

summary = load_dataset(
    "pxzleo/qwen3.8-27b-inference-benchmark-4090",
    "performance_summary",
)

requests = load_dataset(
    "pxzleo/qwen3.8-27b-inference-benchmark-4090",
    "performance_requests",
)

Methodology notes

• —TTFT is measured from request start to the first non-empty streamed token.
• —Decode throughput is (completion_tokens - 1) / (total_seconds - ttft_seconds).
• —Aggregate throughput is total completion tokens in a concurrent round divided by batch wall time.
• —Every published performance request produced exactly 512 completion tokens with finish_reason=length.
• —q27 tokenized the common prompt as 8,408 tokens; the other three deployments reported 8,401.
• —NInfer concurrent-4 used two complete concurrent warm-up batches; the other headline modes used one. This limits strict TTFT comparability.
• —The long-context quick-suite measurements are total request time, not isolated prefill throughput.
• —Upstream AIME/GPQA dataset revisions were not pinned. Reproductions must verify the published hashes and stop if they differ.

Full commands, methodology, limitations, and reproduction instructions are maintained in the GitHub repository.

Projects tested

• —NInfer for RTX 4090 48GB: the tested mixed Q4/Q5/Q6 deployment, optimized for this hardware/model combination and shipped with a polished WebUI.
• —q27: the tested Q6_K deployment.
• —vLLM: the tested FP8 + FP8 KV MTP2 deployment.
• —llama.cpp: the BF16 dual-GPU reference.

Intended use

Use this dataset to compare the recorded serving configurations, reproduce charts, inspect request-level variance, or establish a transparent baseline for new local inference runs. It is not a general model leaderboard, an official Qwen benchmark, or evidence that one backend will lead on different prompts, machines, concurrency, context lengths, or software revisions.

Licensing and data boundaries

Original benchmark-result data and documentation in this dataset are licensed under CC BY 4.0. Attribute xu and link to this dataset or the GitHub source repository; indicate modifications.

This grant does not relicense third-party software, models, benchmark questions, datasets, or trademarks. No complete AIME or GPQA question text is distributed. Users must review and comply with current upstream terms before downloading or reusing third-party datasets.

中文说明

这是 Qwen3.8-27B 本地推理测试报告 的结构化数据版本，面向 RTX 4090 48GB 本地部署场景。数据集包含 9 行性能汇总、140 条逐请求性能明细、12 行精度汇总、90 条测试样本索引，以及脱敏后的环境、部署和方法配置。

本数据集不公开 AIME/GPQA 完整题面、模型提示词、模型回答、思维链、模型权重、凭据或个人路径。第三方数据集的许可证和使用条款不因本数据集的 CC BY 4.0 许可而改变。

测试中，NInfer mixed Q4/Q5/Q6 在单请求速度、四并发聚合吞吐和四并发 TTFT 上取得本次记录中的最好结果；其四并发聚合吞吐仅比 q27 高 0.16%，两者在该项应视为同一档。精度样本较小，不能据此建立稳定排名。

更完整的部署命令、方法、限制和复现说明请以 GitHub 主仓库 为准。