datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
Devin-SWE-bench-outputdevin-2024-03-24-17-25-22Research-DevIntent
IntentSpec Benchmark — Data Supplement
This archive contains the benchmark data used to compute Intent Violation
Rate (IVR) in the paper: 49 tasks, each derived from a HumanEval problem and
extended with an ambiguous/gold prompt pair and a decomposed set of
executable constraints.
Files
spec_pairs.jsonl
The benchmark itself — one JSON object per line, one line per task. This is
the file consumed directly by the evaluation pipeline… See the full description on the dataset page: https://huggingface.co/datasets/anote-ai/Research-DevIntent.job-pipeline-data