ShahzebKhoso/local-code-arena-mbpp-qwen3.5_2b

Local Code Arena Telemetry: MBPP Benchmark on Qwen 3.5 2B

This repository hosts the raw evaluation metrics, execution telemetry logs, and structural syntax outputs captured from running the Mostly Basic Python Problems (MBPP) benchmark against the Qwen 3.5 2B base model.

This specific run documents how a tiny, next-generation generalist instruction model handles zero-shot functional programming synthesis under standard local execution bounds without active reasoning tokens.

📊 Core Performance Summary

• —Evaluation Target: qwen3.5:2b (via Ollama Server)
• —Functional Pass@1 Accuracy: 18.2%
• —Average Generation Speed: 144.46 Tokens/Second ⚡
• —Evaluation Window: 500 tasks (Test Split)

📈 Cross-Generation Paradigm Analysis

Placing this dataset into the broader Local Code Arena matrix highlights the massive impact of domain-specific pre-training versus generalized generational updates:

Key Structural Insight: The telemetry demonstrates that generational leaps in base model architectures do not automatically translate to better out-of-the-box performance on strict functional coding benchmarks. Without domain-specific code saturation or active chain-of-thought reasoning steps, small generalist weights face significant accuracy trade-offs.

💻 Baseline Hardware Configuration

All telemetry records inside this dataset matrix were compiled on a singular local environment footprint:

• —Host System: Alienware m18 Performance Notebook
• —GPU Accelerator: NVIDIA GeForce RTX 4090 Laptop GPU (16GB GDDR6 VRAM / 175W TGP Max)
• —Driver / CUDA Stack: NVIDIA Driver 581.95 | CUDA 13.0
• —Isolation Engine: Multi-threaded Python Code Execution Sandbox (2.0s Hard Wall-Clock Timeout Limit)

📂 Dataset Architecture & Feature Schema

Each row within this dataset represents a fully evaluated, structured code generation instance. The table outlines the schemas available in the parquet records:

🛠️ Evaluation Metrics Breakdown

Inside the evaluation_metrics structural child frame, fields map precise tracking criteria:

• —`functional_pass` (bool): Evaluates to true if the code compiled cleanly and completed 100% of the associated test assertion strings.
• —`sandbox_feedback` (string): The precise stdout message or traceback captured by the isolated runtime environment loop (e.g., Execution Timeout, NameError, or Success).
• —`codebleu_overall` (float): An aggregated structural score grading AST matches and data-flow syntax layout configurations against the ground truth target.
• —`generation_speed_tps` (float): The dedicated processing efficiency score capturing exact Tokens per Second generated on the local RTX 4090.
• —`latency_seconds` (float): The absolute round-trip execution latency for model inference response strings.

🚀 How to Utilize This Dataset

You can stream this telemetry dataset into your local evaluation analysis notebooks using the Hugging Face datasets engine:

from datasets import load_dataset

# Stream the local code arena performance log straight into your dataframe
dataset = load_dataset("ShahzebKhoso/local-code-arena-mbpp-qwen3.5-2b")

# Access individual record blocks
first_entry = dataset['train'][0]
print(f"Model Throughput Speed: {first_entry['evaluation_metrics']['generation_speed_tps']} TPS")

📄 Licensing & Citation

This telemetry package is distributed freely under the permissive MIT License. If you adapt this benchmark architecture or build upon these hardware token files in academic workflows, please point back to this Hub repository space.