datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
k-browsecomp
K-BrowseComp
K-BrowseComp is a Korean version of BrowseComp: a web-browsing agent benchmark. Items are grounded in Korean contexts and require retrieving information across multiple Korean websites.
The 300-question verified subset is entirely handcrafted by native Korean speakers and every item underwent thorough manual revision and validation.
📄 Paper: https://arxiv.org/abs/2606.02404
💻 Code: https://github.com/prometheus-eval/K-BrowseComp
Subsets… See the full description on the dataset page: https://huggingface.co/datasets/prometheus-eval/k-browsecomp.Feedback-Collection
Dataset Card
Dataset Summary
The Feedback Collection is a dataset designed to induce fine-grained evaluation capabilities into language models.\
Recently, proprietary LLMs (e.g., GPT-4) have been used to evaluate long-form responses. In our experiments, we found that open-source LMs are not capable of evaluating long-form responses, showing low correlation with both human evaluators and GPT-4.\
In our paper, we found that by (1) fine-tuning feedback generated by GPT-4… See the full description on the dataset page: https://huggingface.co/datasets/prometheus-eval/Feedback-Collection.WebInstructSub-prometheus
Dataset Card for WebInstructSub-prometheus
This dataset has been created with distilabel.
Dataset Summary
TIGER-Lab/WebInstructSub evaluated for logical and effective reasoning using prometheus-7b-v2.0.
This dataset contains a pipeline.yaml which can be used to reproduce the pipeline that generated it in distilabel using the distilabel CLI:
distilabel pipeline run --config… See the full description on the dataset page: https://huggingface.co/datasets/chargoddard/WebInstructSub-prometheus.peerreview-bench
PeerReview Bench
CMU Paper Reviewer:https://prometheus-eval.github.io/cmu-paper-reviewer/
Repository:https://github.com/prometheus-eval/cmu-paper-reviewer
Paper:https://arxiv.org/abs/2605.20668
Point of Contact:seungone@kaist.ac.kr
Expert-annotated review items from scientific papers, organized for three
complementary evaluation tasks. All data in this dataset is intended
for evaluation, not training. All configs reference a shared, deduplicated
file store (submitted_papers)… See the full description on the dataset page: https://huggingface.co/datasets/prometheus-eval/peerreview-bench.BiGGen-Bench
BIGGEN-Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models
Dataset Description
BIGGEN-Bench (BiG Generation Benchmark) is a comprehensive evaluation benchmark designed to assess the capabilities of large language models (LLMs) across a wide range of tasks. This benchmark focuses on free-form text generation and employs fine-grained, instance-specific evaluation criteria.
Key Features:
Purpose: To evaluate LLMs on diverse capabilities… See the full description on the dataset page: https://huggingface.co/datasets/prometheus-eval/BiGGen-Bench.BiGGen-Bench-Results
BIGGEN-Bench Evaluation Results
Dataset Description
This dataset contains the evaluation results for various language models on the BIGGEN-Bench (BiG Generation Benchmark). It provides comprehensive performance assessments across multiple capabilities and tasks.
Key Features
Evaluation results for 103 language models
Scores across 9 different capabilities
Results from multiple evaluator models (GPT-4, Claude-3-Opus, Prometheus-2)
Dataset Statistics… See the full description on the dataset page: https://huggingface.co/datasets/prometheus-eval/BiGGen-Bench-Results.PROMETHEUS
PROMETHEUS artifacts
Heavy artifacts for the PROMETHEUS financial intelligence system: raw price/macro/news parquet, engineered features, model checkpoints, causal graph. Managed by the pipeline — do not edit by hand.
Preference-Collection
Dataset Card
Dataset Summary
The Preference Collection is a dataset designed to induce fine-grained evaluation capabilities into language models.
Recently, proprietary LLMs (e.g., GPT-4) have been used to evaluate long-form responses. In our experiments, we found that open-source LMs are not capable of evaluating long-form responses, showing low correlation with both human evaluators and GPT-4.\
In our paper, we found that by (1) fine-tuning feedback generated by GPT-4… See the full description on the dataset page: https://huggingface.co/datasets/prometheus-eval/Preference-Collection.canva-visual-search-embeddings
Visual Search Embedding Benchmark: Extending Canva's DINOv2 Evaluation
Executive Summary
This benchmark extends Canva's January 2025 engineering evaluation
which chose DINOv2 for production image replacement. We test three newer models released since then
against DINOv2 on 500 design-domain images (advertising posters from the CGL-Dataset).
Key Findings
Metric
Winner
Score
vs DINOv2
Recall@1
facebook/dinov2-base
1.0000
—
Recall@5… See the full description on the dataset page: https://huggingface.co/datasets/prometheus04/canva-visual-search-embeddings.Prometheus2-preference-standardDirectly transformed from prometheus-eval/Preference-Collection.
Here is the data processing code:
ds = load_dataset("prometheus-eval/Preference-Collection",split='train')
new_data = []
for example in tqdm(ds):
prompt = example['instruction']
responses = [example['orig_response_A'], example['orig_response_B']]
scores = [example['orig_score_A'], example['orig_score_B']]
chosen_idx = int(example['orig_preference'] == 'B') # A:0, B:1
chosen_response, rejected_response =… See the full description on the dataset page: https://huggingface.co/datasets/RLHFlow/Prometheus2-preference-standard.Perception-Collection
Dataset Card
Homepage: https://kaistai.github.io/prometheus-vision/
Repository: https://github.com/kaistAI/prometheus-vision
Paper: https://arxiv.org/abs/2401.06591
Point of Contact: seongyun@kaist.ac.kr
Dataset summary
Perception Collection is the first multi-modal feedback dataset that could be used to train an evaluator VLM. Perception Collection includes 15K fine-grained criteria that determine the crucial aspect for each instance.
Languages… See the full description on the dataset page: https://huggingface.co/datasets/prometheus-eval/Perception-Collection.prometheus-llm-as-a-judge-v1qwen3-4b-thinking-microagent-v1-tb2-eval
qwen3-4b-thinking-microagent v1 — terminal-bench 2.0 evaluation
This dataset contains the full evaluation artifacts from running the v1 merged
model (prometheus04/qwen3-4b-thinking-microagent-v1-merged)
against terminal-bench 2.0 (89 tasks, k=1, n_concurrent=1).
Headline result
metric
value
model
prometheus04/qwen3-4b-thinking-microagent-v1-merged
benchmark
terminal-bench@2.0
n_tasks
89
n_attempts (k)
1
pass rate
1 / 89 = 1.12%
errors
0… See the full description on the dataset page: https://huggingface.co/datasets/prometheus04/qwen3-4b-thinking-microagent-v1-tb2-eval.rwp-prometheus
Dataset Card for rwp-prometheus
This dataset has been created with distilabel.
Dataset Summary
This dataset contains a pipeline.yaml which can be used to reproduce the pipeline that generated it in distilabel using the distilabel CLI:
distilabel pipeline run --config "https://huggingface.co/datasets/chargoddard/rwp-prometheus/raw/main/pipeline.yaml"
or explore the configuration:
distilabel pipeline info --config… See the full description on the dataset page: https://huggingface.co/datasets/chargoddard/rwp-prometheus.Perception-Bench
Dataset Card
Homepage: https://kaistai.github.io/prometheus-vision/
Repository: https://github.com/kaistAI/prometheus-vision
Paper: https://arxiv.org/abs/2401.06591
Point of Contact: seongyun@kaist.ac.kr
Dataset summary
Perception-Bench is a benchmark for evaluating the long-form response of a VLM (Vision Language Model) across various domains of images, and it is a held-out test
set of the Perception-Collection
Languages
English
Dataset… See the full description on the dataset page: https://huggingface.co/datasets/prometheus-eval/Perception-Bench.Feedback-Bench
Dataset Card for "Promixtheus-Absolute-Bench"
More Information needed
Prometheusprometheus-eval-distilabel-default
Dataset Card for prometheus-eval-distilabel-default
This dataset has been created with distilabel.
Dataset Summary
This dataset contains a pipeline.yaml which can be used to reproduce the pipeline that generated it in distilabel using the distilabel CLI:
distilabel pipeline run --config "https://huggingface.co/datasets/alvarobartt/prometheus-eval-distilabel-default/raw/main/pipeline.yaml"
or explore the configuration:
distilabel pipeline info --config… See the full description on the dataset page: https://huggingface.co/datasets/alvarobartt/prometheus-eval-distilabel-default.prometheus-eval-distilabel
Dataset Card for prometheus-eval-distilabel
This dataset has been created with distilabel.
Dataset Summary
This dataset contains a pipeline.yaml which can be used to reproduce the pipeline that generated it in distilabel using the distilabel CLI:
distilabel pipeline run --config "https://huggingface.co/datasets/alvarobartt/prometheus-eval-distilabel/raw/main/pipeline.yaml"
or explore the configuration:
distilabel pipeline info --config… See the full description on the dataset page: https://huggingface.co/datasets/alvarobartt/prometheus-eval-distilabel.MM-Eval
Multilingual Meta-EVALuation benchmark (MM-Eval)
👨💻Code
|
📄Paper
|
🤗 MMQA
MM-Eval is a multilingual meta-evaluation benchmark consisting of five core subsets—Chat, Reasoning, Safety, Language Hallucination, and Linguistics—spanning 18 languages and a Language Resource subset spanning 122 languages for a broader analysis of language effects.
Design ChoiceIn this work, we minimize the inclusion of translated samples, as mere translation may alter existing preferences due to… See the full description on the dataset page: https://huggingface.co/datasets/prometheus-eval/MM-Eval.details_AiMavenAi__AiMaven-Prometheus
Dataset Card for Evaluation run of AiMavenAi/AiMaven-Prometheus
Dataset automatically created during the evaluation run of model AiMavenAi/AiMaven-Prometheus on the Open LLM Leaderboard.
The dataset is composed of 63 configuration, each one coresponding to one of the evaluated task.
The dataset has been created from 1 run(s). Each run can be found as a specific split in each configuration, the split being named using the timestamp of the run.The "train" split is always pointing to… See the full description on the dataset page: https://huggingface.co/datasets/open-llm-leaderboard-old/details_AiMavenAi__AiMaven-Prometheus.nvidia-kaggleSWE-Prometheus
SWE-Prometheus Public Tasks
Public question package for SWE-Prometheus from CosmosMind AI Lab.
This release contains the 22 public instances. It includes only the shared task statement and the public repository manifest with fixed base revisions. Verification patches, scores, results, traces, treated evidence, and reference answers are intentionally excluded.
prometheus-abs-eval-distilabel
Dataset Card for prometheus-abs-eval-distilabel
This dataset has been created with distilabel.
Dataset Summary
This dataset contains a pipeline.yaml which can be used to reproduce the pipeline that generated it in distilabel using the distilabel CLI:
distilabel pipeline run --config "https://huggingface.co/datasets/alvarobartt/prometheus-abs-eval-distilabel/raw/main/pipeline.yaml"
or explore the configuration:
distilabel pipeline info --config… See the full description on the dataset page: https://huggingface.co/datasets/alvarobartt/prometheus-abs-eval-distilabel.prometheus-eval-distilabel-ratings
Dataset Card for prometheus-eval-distilabel-ratings
This dataset has been created with distilabel.
Dataset Summary
This dataset contains a pipeline.yaml which can be used to reproduce the pipeline that generated it in distilabel using the distilabel CLI:
distilabel pipeline run --config "https://huggingface.co/datasets/alvarobartt/prometheus-eval-distilabel-ratings/raw/main/pipeline.yaml"
or explore the configuration:
distilabel pipeline info --config… See the full description on the dataset page: https://huggingface.co/datasets/alvarobartt/prometheus-eval-distilabel-ratings.FoundationPose_Datamicroagent-train-v3
microagent-train-v3
SFT corpus for training a 4B-class terminal-agent model (Qwen3-4B-Thinking) in the microagent XML protocol. v3 = the 26,627-trajectory v2 corpus plus 3,951 synthetic failure-recovery trajectories that target the specific execution weaknesses found in v1 evaluation.
Why v3 exists
The v1 model (prometheus04/qwen3-4b-thinking-microagent-v1-merged) scored 1/89 (1.12%) on Terminal-Bench 2.0. Trajectory analysis showed the model reasoned correctly but failed… See the full description on the dataset page: https://huggingface.co/datasets/prometheus04/microagent-train-v3.vicgalle__Merge-Mixtral-Prometheus-8x7B-details
Dataset Card for Evaluation run of vicgalle/Merge-Mixtral-Prometheus-8x7B
Dataset automatically created during the evaluation run of model vicgalle/Merge-Mixtral-Prometheus-8x7B
The dataset is composed of 38 configuration(s), each one corresponding to one of the evaluated task.
The dataset has been created from 1 run(s). Each run can be found as a specific split in each configuration, the split being named using the timestamp of the run.The "train" split is always pointing to the… See the full description on the dataset page: https://huggingface.co/datasets/open-llm-leaderboard/vicgalle__Merge-Mixtral-Prometheus-8x7B-details.chargoddard__prometheus-2-llama-3-8b-details
Dataset Card for Evaluation run of chargoddard/prometheus-2-llama-3-8b
Dataset automatically created during the evaluation run of model chargoddard/prometheus-2-llama-3-8b
The dataset is composed of 38 configuration(s), each one corresponding to one of the evaluated task.
The dataset has been created from 1 run(s). Each run can be found as a specific split in each configuration, the split being named using the timestamp of the run.The "train" split is always pointing to the latest… See the full description on the dataset page: https://huggingface.co/datasets/open-llm-leaderboard/chargoddard__prometheus-2-llama-3-8b-details.details_allknowingroger__PrometheusLaser-7B-slerp
Dataset Card for Evaluation run of allknowingroger/PrometheusLaser-7B-slerp
Dataset automatically created during the evaluation run of model allknowingroger/PrometheusLaser-7B-slerp on the Open LLM Leaderboard.
The dataset is composed of 63 configuration, each one coresponding to one of the evaluated task.
The dataset has been created from 1 run(s). Each run can be found as a specific split in each configuration, the split being named using the timestamp of the run.The "train"… See the full description on the dataset page: https://huggingface.co/datasets/open-llm-leaderboard-old/details_allknowingroger__PrometheusLaser-7B-slerp.