ds1000
Datasets
All datasets matching “ds1000”DS-1000 DS-1000 in simplified format
🔥 Check the leaderboard from Eval-Arena on our project page.
See testing code and more information (also the original fill-in-the-middle/Insertion format) in the DS-1000 repo.
Reformatting credits: Yuhang Lai, Sida Wang
DS1000DS-1000 is a code generation benchmark with a thousand data science problems spanning seven Python libraries, such as NumPy and Pandas. It employs multi-criteria evaluation metrics, including functional correctness and surface-form constraints, resulting in a high-quality dataset with only 1.8% incorrect solutions among accepted Codex-002 predictions.
Usage
import datasets
# Download the dataset
queries = datasets.load_dataset("embedding-benchmark/DS1000", "queries")
documents =… See the full description on the dataset page: https://huggingface.co/datasets/embedding-benchmark/DS1000.ds1000-test-rollouts
DS-1000 test-suite rollouts
Model-generated test suites for all 1000 DS-1000
problems, for the joint coding-and-testing construction (independent code and test experts;
see the mlcb-ocaml campaign).
The matching code rollouts are the domain=ds1000 cells of
samuki-hf/thinking-rollouts
(join on instance_id = DS-1000 test-split row index).
100 rollouts per problem per model: Qwen3-4B / 8B / 14B, non-thinking
(enable_thinking=False), temperature 0.6, top_p 0.8, top_k 20, min_p 0.0… See the full description on the dataset page: https://huggingface.co/datasets/vxef/ds1000-test-rollouts.rteb-DS1000Retrieval
DS1000Retrieval — RTEB open subset, unified schema
A normalised copy of the dataset behind the mteb task DS1000Retrieval, one of the 17 open tasks in the
RTEB(beta) retrieval benchmark. Same queries, documents and relevance
judgements as the benchmark evaluates — reshaped into one strict schema shared by all 17.
Source
embedding-benchmark/DS1000 @ 25cd4dc8172e (the revision pinned in mteb)
Domain · languages
code · eng
Queries / documents / qrels
1,998 / 1,998 /… See the full description on the dataset page: https://huggingface.co/datasets/Hyukkyu/rteb-DS1000Retrieval.ds1000_pnyx
PNYX - DS-1000
This is a splitted and tested version of DS-1000, based on the reformatted version claudios/ds1000 (extracted metadata as columns). This version is designed to be compatible with the hf_evaluate code_eval package. Also, the code was modified to work with newer versions of the used python packages (numpy, scipy, etc.).
This dataset includes all the original fields and the following ones:
user_chat_prompt: A chat-style prompt for the problem, adapted from the prompt… See the full description on the dataset page: https://huggingface.co/datasets/PNYX/ds1000_pnyx.ds1000DS-1000 dataset annotated with the ground-truth library documentation, to enable evaluations for retrieval and retrieval-augmented code generation.
Please refer to [code-rag-bench] for more details