datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
TerraMesh
TerraMesh
A planetary‑scale, multimodal analysis‑ready dataset for Earth‑Observation foundation models: TerraMesh merges data from Sentinel‑1 SAR, Sentinel‑2 optical, Copernicus DEM, NDVI, and land‑cover sources into more than 9 million co‑registered patches ready for large‑scale representation learning.
You find more information about the data sampling and preprocessing in our paper: TerraMesh: A Planetary Mosaic of Multimodal Earth Observation Data.
Samples from the TerraMesh… See the full description on the dataset page: https://huggingface.co/datasets/ibm-esa-geospatial/TerraMesh.acp_bench
ACP Bench
🏠 Homepage •
📄 Paper •
📄 Paper
ACPBench is a benchmark dataset designed to evaluate the reasoning capabilities of large language models (LLMs) in the context of Action, Change, and Planning. It spans 13 diverse domains:
Blocksworld
Logistics
Grippers
Grid
Ferry
FloorTile
Rovers
VisitAll
Depot
Goldminer
Satellite
Swap
Alfworld
Task Types in ACPBench
ACPBench includes the following 8 reasoning tasks:
Action Applicability (app)… See the full description on the dataset page: https://huggingface.co/datasets/ibm-research/acp_bench.data-product-benchmark
DPDisc Dataset
Paper | Code
Dataset Description
This dataset provides a benchmark for automatic data product creation. The task is framed as follows: given a natural language data product request and a corpus of text and tables, the objective is to identify the relevant tables and text documents that should be included in the resulting data product which would useful to the given data product request. The benchmark brings together three variants: HybridQA, TAT-QA, and… See the full description on the dataset page: https://huggingface.co/datasets/ibm-research/data-product-benchmark.argument_quality_ranking_30k
Dataset Card for Argument-Quality-Ranking-30k Dataset
Dataset Summary
Argument Quality Ranking
The dataset contains 30,497 crowd-sourced arguments for 71 debatable topics labeled for quality and stance, split into train, validation and test sets.
The dataset was originally published as part of our paper: A Large-scale Dataset for Argument Quality Ranking: Construction and Analysis.
Argument Topic
This subset contains 9,487 of the arguments only with… See the full description on the dataset page: https://huggingface.co/datasets/ibm-research/argument_quality_ranking_30k.Llama3-SSL4EO-S12-v1.1-captions
Llama3-SSL4EO-S12-Captions
The captions are aligned with the SSL4EO-S12 v1.1 dataset and were automatically generated using the Llama3-LLaVA-Next-8B model.
Please find more information regarding the generation and evaluation in the Llama3-MS-CLIP paper.
Code: https://github.com/IBM/MS-CLIP
Data Structure
We provide the captions in two versions: As a single compressed Parquet file per split and as CSV files with 256 captions each that match the Zarr Zip files of the… See the full description on the dataset page: https://huggingface.co/datasets/ibm-esa-geospatial/Llama3-SSL4EO-S12-v1.1-captions.TerraMesh-Masks
TerraMesh-Masks
TerraMesh-Masks is a dataset for open-vocabulary segmentation of satellite imagery. This dataset provides binary segmentation masks with captions that extend the samples from TerraMesh.
We also provide an human-verfied evaluation benchmark, called TerraMesh-Masks-Eval.
Examples from the training subset:
Usage
Download the data loading code from GitHub and install requirements with pip install -r requirements.txt. For development, you can… See the full description on the dataset page: https://huggingface.co/datasets/ibm-esa-geospatial/TerraMesh-Masks.earnings_callThe dataset reports a collection of earnings call transcripts, the related stock prices, and the sector index In terms of volume, there is a total of 188 transcripts, 11970 stock prices, and 1196 sector index values. Furthermore, all of these data originated in the period 2016-2020 and are related to the NASDAQ stock market. Furthermore, the data collection was made possible by Yahoo Finance and Thomson Reuters Eikon. Specifically, Yahoo Finance enabled the search for stock values and Thomson Reuters Eikon provided the earnings call transcripts. Lastly, the dataset can be used as a benchmark for the evaluation of several NLP techniques to understand their potential for financial applications. Moreover, it is also possible to expand the dataset by extending the period in which the data originated following a similar procedure.claim_stance
Dataset Card for Claim Stance Dataset
Dataset Summary
Claim Stance
This dataset contains 2,394 labeled Wikipedia claims for 55 topics. The dataset includes the stance (Pro/Con) of each claim towards the topic,
as well as fine-grained annotations, based on the semantic model of Stance Classification of Context-Dependent Claims (topic target,
topic sentiment towards its target, claim target, claim sentiment towards its target, and the relation between the… See the full description on the dataset page: https://huggingface.co/datasets/ibm-research/claim_stance.IBM-Transactions-for-Anti-Money-Laundering-HI-Small-TransSombench-pretraining-data
SomBench Pre-training Corpus: Multimodal Lunar Tiles
Dataset Summary
This includes a small sample from SomBench: a corpus of co-registered, multimodal lunar image tiles built for
large-scale self-supervised (foundation-model) pre-training. It contains a subset of modalities from the
low-resolution (WAC-anchored) and high-resolution (NAC-anchored) tracks specifically used in pretraining.
Tiles are anchored to individual LROC Experiment Data Record (EDR) image… See the full description on the dataset page: https://huggingface.co/datasets/nasa-ibm-ai4science/Sombench-pretraining-data.Surya-bench-solarwind
Solar Wind Forecasting Dataset
Dataset Summary
This dataset provides hourly solar wind plasma and interplanetary magnetic field (IMF) parameters at L1, derived from NASA’s OMNI dataset. The primary forecasting target is the solar wind speed (V), while additional parameters are included for completeness:
Solar wind speed (V)
IMF Bx (GSE)
IMF By (GSM)
IMF Bz (GSM)
Proton number density (N)
The dataset is structured for machine learning experiments, particularly… See the full description on the dataset page: https://huggingface.co/datasets/nasa-ibm-ai4science/Surya-bench-solarwind.Sombench-WAC-Crater-Detection
SomBench Benchmark: Robbins Crater Detection, WAC
Science theme: Impact processes
Task: Object detection
Dataset Summary
An impact-crater object-detection benchmark built from the
Robbins (2019) global lunar crater
catalog, a manually compiled, near-complete census of
lunar impact craters (≥ ~1–2 km). Catalog crater centers and diameters are
converted to bounding boxes and packaged over LROC WAC visible tiles drawn
from the pre-training corpus test split, in COCO… See the full description on the dataset page: https://huggingface.co/datasets/nasa-ibm-ai4science/Sombench-WAC-Crater-Detection.REAL-MM-RAG_TechSlides_BEIR
BEIR Version of REAL-MM-RAG_TechSlides
Summary
This dataset is the BEIR-compatible version of the following Hugging Face dataset:
ibm-research/REAL-MM-RAG_TechSlides
It has been reformatted into the BEIR structure for evaluation in retrieval settings.The original dataset is QA-style (each row is a query tied to a document image).Here, queries, qrels, docs, and corpus are separated into BEIR-standard splits.
REAL-MM-RAG_TechSlides
Content: 62 technical… See the full description on the dataset page: https://huggingface.co/datasets/ibm-research/REAL-MM-RAG_TechSlides_BEIR.REAL-MM-RAG_FinReport_BEIR
BEIR Version of REAL-MM-RAG_FinReport
Summary
This dataset is the BEIR-compatible version of the following Hugging Face dataset:
ibm-research/REAL-MM-RAG_FinReport
It has been reformatted into the BEIR structure for evaluation in retrieval settings.The original dataset is QA-style (each row is a query tied to a document image).Here, queries, qrels, docs, and corpus are separated into BEIR-standard splits.
REAL-MM-RAG_FinReport
Content: 19 financial… See the full description on the dataset page: https://huggingface.co/datasets/ibm-research/REAL-MM-RAG_FinReport_BEIR.REAL-MM-RAG_FinSlides_BEIR
BEIR Version of REAL-MM-RAG_FinSlides
Summary
This dataset is the BEIR-compatible version of the following Hugging Face dataset:
ibm-research/REAL-MM-RAG_FinSlides
It has been reformatted into the BEIR structure for evaluation in retrieval settings.The original dataset is QA-style (each row is a query tied to a document image).Here, queries, qrels, docs, and corpus are separated into BEIR-standard splits.
REAL-MM-RAG_FinSlides
Content: 65 quarterly… See the full description on the dataset page: https://huggingface.co/datasets/ibm-research/REAL-MM-RAG_FinSlides_BEIR.euv-spectra
Solar EUV Spectra Modeling Dataset
Dataset Summary
This dataset provides time-aligned Extreme Ultraviolet (EUV) irradiance spectra from NASA’s SDO/EVE (Extreme Ultraviolet Variability Experiment) instrument. This dataset enables machine learning models to learn from and predict EUV spectral behavior driven by solar dynamics. It addresses the need for high-resolution, calibrated spectral data paired with physics-based contextual input. It is designed for image-to-spectra… See the full description on the dataset page: https://huggingface.co/datasets/nasa-ibm-ai4science/euv-spectra.REAL-MM-RAG_TechReport_BEIR
BEIR Version of REAL-MM-RAG_TechReport
Summary
This dataset is the BEIR-compatible version of the following Hugging Face dataset:
ibm-research/REAL-MM-RAG_TechReport
It has been reformatted into the BEIR structure for evaluation in retrieval settings.The original dataset is QA-style (each row is a query tied to a document image).Here, queries, qrels, docs, and corpus are separated into BEIR-standard splits.
REAL-MM-RAG_TechReport
Content: 17 technical… See the full description on the dataset page: https://huggingface.co/datasets/ibm-research/REAL-MM-RAG_TechReport_BEIR.LLMFineTuningBench
Dataset Card for LLMFineTuningBench
A dataset of over 30,000 LLM fine-tuning experiments, capturing detailed performance metrics from jobs run on high-performance computing (HPC) clusters. It spans a wide range of models, fine-tuning methods, and hardware configurations, and is intended to support research on predictive resource allocation, performance optimization, and cost estimation for LLM fine-tuning workloads.
Dataset Details
Dataset Description… See the full description on the dataset page: https://huggingface.co/datasets/ibm-research/LLMFineTuningBench.surya-bench-flare-forecasting
Full-disk Solar Flare Forecasting Dataset
Dataset Summary
This dataset provides labels for solar flare forecasting derived from NOAA GOES flare events from May 2010 to December 2024. Labels are constructed using a 24h rolling prediction window sampled at an hourly cadence. Each window is annotated with both max GOES class (based on peak X-ray flux) and cumulative flare index.
Two derived binary labels are included for forecasting tasks:
label_max: 1 if the maximum… See the full description on the dataset page: https://huggingface.co/datasets/nasa-ibm-ai4science/surya-bench-flare-forecasting.TerraMesh-Masks-Eval
TerraMesh-Masks-Eval
TerraMesh-Masks-Eval is a human-verified benchmark dataset to evaluate open-vocabulary segmentation models on satellite imagery. This dataset provides binary segmentation masks with captions togehther with input samples from TerraMesh.
We also provide a training dataset, called TerraMesh-Masks.
Examples from the evaluation subset:
Usage
Download the data loading code from GitHub and install requirements with pip install -r requirements.txt.… See the full description on the dataset page: https://huggingface.co/datasets/ibm-esa-geospatial/TerraMesh-Masks-Eval.ibm-project-codenet
Project_CodeNet
Overview
This dataset is constructed from the Project CodeNet corpus, consisting of competitive programming submissions collected from online judges.
We extract a large-scale code corpus designed for pretraining language models, with a focus on:
clean executable code
temporal metadata (submission time)
minimal preprocessing to preserve the original distribution
Dataset Statistics
Total samples: ~6.37M
Total tokens: ~3.06B
Average tokens… See the full description on the dataset page: https://huggingface.co/datasets/TheFinAI/ibm-project-codenet.ibm-granite__granite-3.2-8b-instruct-details
Dataset Card for Evaluation run of ibm-granite/granite-3.2-8b-instruct
Dataset automatically created during the evaluation run of model ibm-granite/granite-3.2-8b-instruct
The dataset is composed of 38 configuration(s), each one corresponding to one of the evaluated task.
The dataset has been created from 1 run(s). Each run can be found as a specific split in each configuration, the split being named using the timestamp of the run.The "train" split is always pointing to the latest… See the full description on the dataset page: https://huggingface.co/datasets/open-llm-leaderboard/ibm-granite__granite-3.2-8b-instruct-details.cortex-ibm-tabformer-embeddings
cortex embeddings — IBM TabFormer
Per-transaction hidden states (128-d, float32) produced by Neospace's cortex transaction
foundation model over every transaction in the public IBM TabFormer credit-card dataset
(card_transaction.v1.csv, 24,386,900 transactions, 2,000 cardholders, 1991–2020).
These embeddings let you reproduce the NeoLDM benchmark
without running cortex: feed them to the gradient-boosted-tree fraud classifier in that repo.
Contents
dir… See the full description on the dataset page: https://huggingface.co/datasets/luizcoroo/cortex-ibm-tabformer-embeddings.rfi-rfp-ibmcloud-questionsibm-granite__granite-3.0-2b-base-details
Dataset Card for Evaluation run of ibm-granite/granite-3.0-2b-base
Dataset automatically created during the evaluation run of model ibm-granite/granite-3.0-2b-base
The dataset is composed of 38 configuration(s), each one corresponding to one of the evaluated task.
The dataset has been created from 2 run(s). Each run can be found as a specific split in each configuration, the split being named using the timestamp of the run.The "train" split is always pointing to the latest… See the full description on the dataset page: https://huggingface.co/datasets/open-llm-leaderboard/ibm-granite__granite-3.0-2b-base-details.ibm-granite__granite-3.0-2b-instruct-details
Dataset Card for Evaluation run of ibm-granite/granite-3.0-2b-instruct
Dataset automatically created during the evaluation run of model ibm-granite/granite-3.0-2b-instruct
The dataset is composed of 38 configuration(s), each one corresponding to one of the evaluated task.
The dataset has been created from 2 run(s). Each run can be found as a specific split in each configuration, the split being named using the timestamp of the run.The "train" split is always pointing to the latest… See the full description on the dataset page: https://huggingface.co/datasets/open-llm-leaderboard/ibm-granite__granite-3.0-2b-instruct-details.IBM-Debater-ArgKPpopqa-tp
Dataset Card for "popqa-tp"
Dataset Summary
PopQA-TP (PopQA Templated Paraphrases) is a dataset derived from PopQA (https://huggingface.co/datasets/akariasai/PopQA), created for the paper "Predicting Question-Answering Performance of Large Language Models
through Semantic Consistency". PopQA-TP takes each question in PopQA and paraphrases it using each of several manually-created templates specific to each question category. The paper investigates the relationship… See the full description on the dataset page: https://huggingface.co/datasets/ibm-research/popqa-tp.ibm-granite__granite-3.0-1b-a400m-instruct-details
Dataset Card for Evaluation run of ibm-granite/granite-3.0-1b-a400m-instruct
Dataset automatically created during the evaluation run of model ibm-granite/granite-3.0-1b-a400m-instruct
The dataset is composed of 38 configuration(s), each one corresponding to one of the evaluated task.
The dataset has been created from 2 run(s). Each run can be found as a specific split in each configuration, the split being named using the timestamp of the run.The "train" split is always pointing… See the full description on the dataset page: https://huggingface.co/datasets/open-llm-leaderboard/ibm-granite__granite-3.0-1b-a400m-instruct-details.ibm-granite__granite-3.0-1b-a400m-base-details
Dataset Card for Evaluation run of ibm-granite/granite-3.0-1b-a400m-base
Dataset automatically created during the evaluation run of model ibm-granite/granite-3.0-1b-a400m-base
The dataset is composed of 38 configuration(s), each one corresponding to one of the evaluated task.
The dataset has been created from 2 run(s). Each run can be found as a specific split in each configuration, the split being named using the timestamp of the run.The "train" split is always pointing to the… See the full description on the dataset page: https://huggingface.co/datasets/open-llm-leaderboard/ibm-granite__granite-3.0-1b-a400m-base-details.