Datasetpublic

Artemis0430/skilleval-v1

SkillEval v1 SkillEval v1 is a 100-task benchmark for evaluating whether agents can discover and use local skills to complete deterministic artifact-producing tasks. SkillEval was generated by the skill-use task synthesis pipeline introduced in SKT: Skill-Use Training at Scale via Verified Synthetic Data Generation. Layout Each tasks/<task_id>/ directory retains the template-driven task structure described in SKT, with additional public metadata, gold artifacts… See the full description on the dataset page: https://huggingface.co/datasets/Artemis0430/skilleval-v1.

sourceHugging Faceupdated 2mo agoView on Hugging Face

3likes341downloads

Dataset Card

SkillEval v1

SkillEval v1 is a 100-task benchmark for evaluating whether agents can discover and use local skills to complete deterministic artifact-producing tasks.

SkillEval was generated by the skill-use task synthesis pipeline introduced in **SKT: Skill-Use Training at Scale via Verified Synthetic Data Generation**.

Layout

Each tasks/<task_id>/ directory retains the template-driven task structure described in SKT, with additional public metadata, gold artifacts, and oracle code:

text

tasks/<task_id>/
├── instruction.md       # Task instruction
├── environment/         # Runtime context
│   ├── data/            # Task data and setup files
│   └── skills/          # Task-specific skills
├── task.toml            # Task configuration
├── tests/               # Deterministic evaluator
│   └── test.sh
├── solution/            # Reference solution
│   └── solve.sh
├── task.json            # Open task definition with gold and evaluator spec
├── meta.json            # Open-release metadata
├── reference/           # Canonical output artifacts
└── oracle.py            # Oracle implementation invoked by solve.sh

Evaluation

We recommend materializing each task in an isolated workspace:

bash

python tools/materialize_skillssh_task_workspace.py \
  --task-dir tasks/<task_id> \
  --workspace /tmp/skilleval-workspace \
  --skills-mode with \
  --clean

Use instruction.md as the prompt. Select --skills-mode without for the no-skill control. After the agent exits, score its workspace externally:

bash

WORKSPACE=/tmp/skilleval-workspace tasks/<task_id>/tests/test.sh

Report the arithmetic mean of all 100 task rewards, retaining failed and timed-out tasks in the denominator.

Paper and citation

If you use SkillEval, please cite:

bibtex

@misc{tan2026sktskillusetrainingscale,
      title={SKT: Skill-Use Training at Scale via Verified Synthetic Data Generation},
      author={Zelin Tan and Yiqun Zhang and Hao Li and Zhiyao Cui and Hejia Geng and Shao Zhang and Hangfan Zhang and Yang Chen and Xiaosong Wang and Lilong Wang and Zhenfei Yin and Shuyue Hu and Chen Zhang and Lei Bai},
      year={2026},
      eprint={2608.02287},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2608.02287},
}