Datasetpublic

JiaaqiLiu/SkillArena-datasets

SkillArena Offline Datasets Offline evaluation data for SkillArena — a validated automatic benchmark generation framework for AI agent skills, targeting NeurIPS 2026 Datasets & Benchmarks Track. Overview This dataset provides domain-specific task input data for 289 AI agent skills across 13 domains. Each skill has 50 curated data files designed as meaningful agent task inputs — files an agent could receive and act upon (analyze, transform, validate, or generate… See the full description on the dataset page: https://huggingface.co/datasets/JiaaqiLiu/SkillArena-datasets.

sourceHugging Facemitupdated 7mo agoView on Hugging Face

0likes2.4kdownloads

Dataset Card

SkillArena Offline Datasets

Offline evaluation data for SkillArena — a validated automatic benchmark generation framework for AI agent skills, targeting NeurIPS 2026 Datasets & Benchmarks Track.

Overview

This dataset provides domain-specific task input data for 289 AI agent skills across 13 domains. Each skill has 50 curated data files designed as meaningful agent task inputs — files an agent could receive and act upon (analyze, transform, validate, or generate from). The data is used by SkillArena's 10-stage evaluation pipeline to automatically generate validated evaluation tasks, graders, and pilot tests.

Metric	Value
Total skills	289 (16 original + 73 community + 200 new community)
Total files	14,450 (skills) + 160 (original-skills)
Files per skill	50
Dataset size	~126 MB
File formats	5 types: CSV, JSON, YAML, Markdown, Plain Text
Generation method	Deterministic, domain-aware, seed-reproducible

Dataset Structure

skillarena-datasets/
├── skills/                         # 289 skills × 50 files = 14,450 files
│   ├── scikit-learn/               # ML training domain
│   │   ├── ml_training_csv_00.csv  # Classification dataset
│   │   ├── ml_training_csv_01.csv  # Regression dataset
│   │   ├── ...                     # 18 more CSV files
│   │   ├── ml_training_json_00.json # Model config
│   │   ├── ...                     # 11 more JSON files
│   │   ├── ml_training_yaml_00.yaml # Training pipeline config
│   │   ├── ...                     # 9 more YAML files
│   │   ├── ml_training_md_00.md    # Model card
│   │   ├── ...                     # 4 more Markdown files
│   │   ├── ml_training_txt_00.txt  # Training log
│   │   └── ...                     # 2 more text files
│   ├── langchain/                  # NLP/LLM domain
│   ├── grafana-dashboards/         # DevOps/Infra domain
│   ├── rdkit/                      # Chemistry domain
│   └── ...                         # 285 more skill directories
├── original-skills/                # 16 core skills (hand-curated, 10 files each)
│   ├── pdf/                        # PDFs, extraction configs
│   ├── xlsx/                       # Spreadsheets, CSV data
│   ├── frontend-design/            # HTML, CSS, JS, design specs
│   └── ...                         # 13 more original skills
└── index.json                      # Dataset index with file metadata

File Distribution Per Skill

Every skill receives exactly 50 files with a fixed distribution:

Count	Format	Purpose
20	`.csv`	Tabular data for analysis and transformation tasks
12	`.json`	Structured data, configs, and specifications
10	`.yaml`	Configuration and specification files
5	`.md`	Documents for analysis and generation tasks
3	`.txt`	Plain text for processing tasks

Total across all skills: 5,780 CSV + 3,468 JSON + 2,890 YAML + 1,445 MD + 867 TXT = 14,450 files

Domain Classification

Skills are classified into 13 domains, each with a specialized data generator:

Domain	Skills	Example Skills
NLP / LLM	82	langchain, llamaindex, huggingface-tokenizers, dspy, outlines
Frontend / Web	53	react-state-management, tailwind-design-system, nextjs-app-router-patterns
Bioinformatics	27	biopython, scanpy, anndata, pysam, clinical-decision-support
Generic	23	file-organizer, contract-analyzer, financial-calc, create-plan
Backend	21	kafka-producer-consumer, redis-cache-manager, database-schema-designer
DevOps / Infra	18	k8s-manifest-generator, grafana-dashboards, prometheus-configuration
Data Analysis	16	pandas (dask, geopandas), seaborn, matplotlib, statsmodels
ML Training	15	scikit-learn, pytorch-lightning, deepspeed, weights-and-biases
Documentation	12	changelog-generator, architecture-diagram-creator, cli-demo-generator
Chemistry	9	rdkit, deepchem, matchms, molfeat, torchdrug
Quantum	5	qiskit, cirq, pennylane, qutip, sparse-autoencoder-training
Security	5	secret-scanner, jwt-token-validator, sast-configuration
Testing	3	bats-testing-patterns, e2e-testing-patterns, temporal-python-testing

Data Content Examples

CSV — ML Training (scikit-learn)

csv

feature_0,feature_1,feature_2,feature_3,target,split
0.83,0.17,-0.45,1.02,1,train
-0.22,0.91,0.33,-0.67,0,train

JSON — NLP/LLM (langchain)

json

{
  "chain_name": "qa_retrieval_chain",
  "chain_type": "stuff",
  "retriever": {
    "type": "vectorstore",
    "search_type": "similarity",
    "search_kwargs": {"k": 4}
  }
}

YAML — DevOps (k8s-manifest-generator)

yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-gateway
  namespace: production
spec:
  replicas: 3
  strategy:
    type: RollingUpdate

How It's Used

SkillArena's pipeline uses these files in the Data Acquisition stage:

SKILL.md → Analyze → Plan → Synthesize → [Data Acquisition] → Grade → Validate → Meta-Eval → Pilot → Promote
                                                  ↑
                                         This dataset provides
                                         input data for tasks

The OfflineDataProvider is the first source in an 8-provider chain:

Offline Dataset (this) → GitHub → Web Search → Kaggle → HuggingFace → Programmatic → Binary Renderer → LLM Synthesis

When a task is generated, the provider:

1.Looks up skills/{skill_name}/ directory
2.Picks the first unused file (sorted alphabetically)
3.Copies it to task_dir/input/input.{ext} as the agent's input

Original Skills (16)

These hand-curated skills have been validated through the full SkillArena pipeline:

Skill	Domain	Quality Score	Files
frontend-design	Frontend/UI	9.95/10	10
theme-factory	Design Systems	9.95/10	10
web-artifacts-builder	Web Development	9.95/10	10
webapp-testing	Testing	9.84/10	10
mcp-builder	MCP Protocol	9.44/10	10
algorithmic-art	Creative Coding	9.43/10	10
canvas-design	Visual Design	9.42/10	10
internal-comms	Communication	9.41/10	10
slack-gif-creator	Animation	9.41/10	10
brand-guidelines	Brand Design	9.41/10	10
doc-coauthoring	Documentation	9.38/10	10
skill-creator	Meta-Skills	9.37/10	10
pptx	Presentations	8.60/10	10
docx	Documents	7.86/10	10
pdf	PDF Processing	7.36/10	10
xlsx	Spreadsheets	7.17/10	10

Average quality score: 9.12/10 | Validation pass rate: 100% (162/162 tasks)

Generation & Validation

All 14,450 files are deterministically generated using domain-specific generators with per-skill random seeds (hash(skill_name) & 0xFFFFFFFF) for full reproducibility.

Validation checks (all passing):

—File count: exactly 50 per skill (20 CSV + 12 JSON + 10 YAML + 5 MD + 3 TXT)
—Minimum file size: >= 500 bytes
—Format validity: CSV parseable with consistent column counts, JSON/YAML parseable
—CSV integrity: header + data rows, no empty columns, consistent column count
—Markdown: has headers and substantial content
—No source code leakage: no Python/Java/Go/JS imports in data files
—Filename convention: {domain}_{ext}_{NN}.{ext} pattern

Regeneration

bash

# Generate all skills
python -m scripts.generate_task_inputs.main --force

# Single skill
python -m scripts.generate_task_inputs.main --skill scikit-learn --force

# Validate only
python -m scripts.generate_task_inputs.main --validate-only

# Deep validation
python scripts/generate_task_inputs/deep_validate.py

Usage

python

from skillarena.pipeline.orchestrator import PipelineOrchestrator

orchestrator = PipelineOrchestrator(
    output_dir="pipeline_output",
    offline_data_dir="skillarena-datasets",  # Point to this dataset
    seed=42
)
state = await orchestrator.run(skill_path="skills/pdf/SKILL.md")

Citation

bibtex

@inproceedings{liu2026skillarena,
  title={SkillArena: Validated Automatic Benchmark Generation for AI Agent Skills},
  author={Liu, Jiaqi},
  booktitle={NeurIPS 2026 Datasets and Benchmarks Track},
  year={2026}
}

License

MIT License — see the SkillArena repository for details.