datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
deutsche-bahn-data
Deutsche Bahn Train Data
This dataset contains public historical data from Deutsche Bahn, the largest German train company. It includes train schedules, delays, and cancellations from stations across Germany.
For more info visit the project page at GitHub: https://github.com/piebro/deutsche-bahn-data
Dataset Structure
Monthly Processed Data
The monthly processed data is located in monthly_processed_data/ and contains files named… See the full description on the dataset page: https://huggingface.co/datasets/piebro/deutsche-bahn-data.wikidata-extraction
Wikidata Extraction
This dataset contains all RDF triples extracted from the latest Wikidata, converted from the N-Triples format to Parquet.
The data originates from Wikidata, a free and open knowledge base that acts as central storage for structured data used by Wikipedia and other Wikimedia projects. The source file is the "truthy" N-Triples dump (latest-truthy.nt.bz2), which contains only the current, non-deprecated statements.
The code to extract this data is available at… See the full description on the dataset page: https://huggingface.co/datasets/piebro/wikidata-extraction.anchoral-paper-artefactsArtefacts related to the paper AnchorAL: Computationally Efficient Active Learning for Large and Imbalanced Datasets (Lesci and Vlachos, 2024) published at the NAACL 2024 conference.
These artefacts can be reproduced using the code available at github.com/pietrolesci/anchoral.
The outputs/ folder includes the raw files created by the individual experiments.
The results/ folder contains the exported metrics and configurations that are used to complete the analysis and create the tables and… See the full description on the dataset page: https://huggingface.co/datasets/pietrolesci/anchoral-paper-artefacts.nli_fever
Overview
The original dataset can be found here
while the Github repo is here.
This dataset has been proposed in Combining fact extraction and verification with neural semantic matching networks. This dataset has been created as a modification
of FEVER.
In the original FEVER setting, the input is a claim from Wikipedia and the expected output is a label.
However, this is different from the standard NLI formalization which is basically a pair-of-sequence to label problem.
To… See the full description on the dataset page: https://huggingface.co/datasets/pietrolesci/nli_fever.financial-excel-modeling-sftopus-rawExact same data as available at https://github.com/Helsinki-NLP/Tatoeba-Challenge/blob/master/data/README-v2023-09-26.md.
pi-extensions-sessions
Coding agent session traces for thomasmustier/pi-extensions-sessions
This dataset contains redacted coding agent session traces collected while working on tmustier/pi-extensions. The traces were exported with pi-share-hf from a local pi workspace and filtered to keep only sessions that passed deterministic redaction and LLM review.
Data description
Each *.jsonl file is a redacted pi session. Sessions are stored as JSON Lines files where each line is a structured… See the full description on the dataset page: https://huggingface.co/datasets/thomasmustier/pi-extensions-sessions.execcomp-ai
SEC Executive Compensation Dataset
[!CAUTION]
🚧 DATASET UNDER CONSTRUCTION 🚧
This dataset is actively being developed and expanded. The current version contains ~12,000 records out of a target of 100,000+ SEC filings (2005-2022).
What to expect:
Data may contain errors or inconsistencies
Schema and fields may change
More records will be added regularly
Statistics will be updated as processing continues
Use at your own risk for research purposes only.
🔗 Pipeline:… See the full description on the dataset page: https://huggingface.co/datasets/pierjoe/execcomp-ai.irs-990-parsed
IRS 990 Parsed Nonprofit Database
Public relational extract of IRS Form 990 / 990-EZ / 990-PF filings, plus the colocated public files we join for address research: CMS NPPES + T-MSIS Medicare spend, FMCSA DOT carriers, OFAC SDN, FEC committees, and the IRS EO BMF.
Generated: 2026-08-17Tables: 34Rows (sum): 459,069,505License: CC0 / public domain — derived from U.S. government recordsHub: https://huggingface.co/datasets/piercewetter3/irs-990-parsed
Layout
Tables… See the full description on the dataset page: https://huggingface.co/datasets/piercewetter3/irs-990-parsed.PIE_Bench_pp
What is PIE-Bench++?
PIE-Bench++ builds upon the foundation laid by the original PIE-Bench dataset introduced by (Ju et al., 2024), designed to provide a comprehensive benchmark for multi-aspect image editing evaluation. This enhanced dataset contains 700 images and prompts across nine distinct edit categories, encompassing a wide range of manipulations:
Object-Level Manipulations: Additions, removals, and modifications of objects within the image.
Attribute-Level Manipulations:… See the full description on the dataset page: https://huggingface.co/datasets/UB-CVML-Group/PIE_Bench_pp.DocLayNet-baseAccurate document layout analysis is a key requirement for high-quality PDF document conversion. With the recent availability of public, large ground-truth datasets such as PubLayNet and DocBank, deep-learning models have proven to be very effective at layout detection and segmentation. While these datasets are of adequate size to train such models, they severely lack in layout variability since they are sourced from scientific article repositories such as PubMed and arXiv only. Consequently, the accuracy of the layout segmentation drops significantly when these models are applied on more challenging and diverse layouts. In this paper, we present \textit{DocLayNet}, a new, publicly available, document-layout annotation dataset in COCO format. It contains 80863 manually annotated pages from diverse data sources to represent a wide variability in layouts. For each PDF page, the layout annotations provide labelled bounding-boxes with a choice of 11 distinct classes. DocLayNet also provides a subset of double- and triple-annotated pages to determine the inter-annotator agreement. In multiple experiments, we provide smallline accuracy scores (in mAP) for a set of popular object detection models. We also demonstrate that these models fall approximately 10\% behind the inter-annotator agreement. Furthermore, we provide evidence that DocLayNet is of sufficient size. Lastly, we compare models trained on PubLayNet, DocBank and DocLayNet, showing that layout predictions of the DocLayNet-trained models are more robust and thus the preferred choice for general-purpose document-layout analysis.copa_nli
Overview
Original dataset available here.
Current dataset extracted from this repo.
This is the "full" dataset.
Curation
Same curation as the one applied in this repo, that is
from the original COPA format:
premise
choice1
choice2
label
My body cast a shadow over the grass
The sun was rising
The grass was cut
0
to the NLI format:
premise
hypothesis
label
My body cast a shadow over the grass
The sun was rising
entailment
My body cast a shadow… See the full description on the dataset page: https://huggingface.co/datasets/pietrolesci/copa_nli.pie-perf-testcasesosm-data
OpenStreetMap Dataset
This is a parsed and enriched dataset of all OpenStreetMap changesets, changeset comments, notes, and note comments.
The dataset includes 4 different datasets:
changeset_data - All OpenStreetMap changesets with enriched metadata (partitioned by year and month)
changeset_comments_data - Comments on changesets from changeset discussions
notes_data - Notes on the map with their locations and status
notes_comments_data - Comments and actions on notes
This… See the full description on the dataset page: https://huggingface.co/datasets/piebro/osm-data.pubmed-200k-rct
Dataset Card for "pubmed-200k-rct"
More Information needed
amazoncat-13kThis dataset has been created as an artefact of the paper AnchorAL: Computationally Efficient Active Learning for Large and Imbalanced Datasets (Lesci and Vlachos, 2024).
More info about this dataset in the appendix of the paper.
The AmazonCat-13k dataset was released by McAuley and Leskovec (2013)
and is composed of product descriptions and reviews classified into 13k multi-label categories.
The dataset is split into 1.2M train and 300k evaluation instances.
It is commonly used as an extreme… See the full description on the dataset page: https://huggingface.co/datasets/pietrolesci/amazoncat-13k.hyperpartisan_news_detection
Dataset Card for "hyperpartisannewsdetection"
More Information needed
DocLayNet-smallAccurate document layout analysis is a key requirement for high-quality PDF document conversion. With the recent availability of public, large ground-truth datasets such as PubLayNet and DocBank, deep-learning models have proven to be very effective at layout detection and segmentation. While these datasets are of adequate size to train such models, they severely lack in layout variability since they are sourced from scientific article repositories such as PubMed and arXiv only. Consequently, the accuracy of the layout segmentation drops significantly when these models are applied on more challenging and diverse layouts. In this paper, we present \textit{DocLayNet}, a new, publicly available, document-layout annotation dataset in COCO format. It contains 80863 manually annotated pages from diverse data sources to represent a wide variability in layouts. For each PDF page, the layout annotations provide labelled bounding-boxes with a choice of 11 distinct classes. DocLayNet also provides a subset of double- and triple-annotated pages to determine the inter-annotator agreement. In multiple experiments, we provide smallline accuracy scores (in mAP) for a set of popular object detection models. We also demonstrate that these models fall approximately 10\% behind the inter-annotator agreement. Furthermore, we provide evidence that DocLayNet is of sufficient size. Lastly, we compare models trained on PubLayNet, DocBank and DocLayNet, showing that layout predictions of the DocLayNet-trained models are more robust and thus the preferred choice for general-purpose document-layout analysis.finewebedu-20BThis is a subset of the HuggingFaceFW/fineweb-edu/100BT dataset.
I extracted (in order) the initial 20,200,000 rows where, ideally, 20M are meant for training and 200k for validation.
Tokenised configs:
bpe32000minipile: 21.6B tokens
License
For the license, refer to the original dataset (HuggingFaceFW/fineweb-edu).
one-piece-chaptersThis dataset is a collection of all one piece chapter's until today (1157) extracted from the one piece wiki as text.
It includes the following structured data:
interface ChapterData {
coverPage: string;
shortSummary: string;
longSummary: string;
quickReferences: {
chapterNotes: string[];
characters: string; // html table with char references
};
}
pile-deduped-pythia-preshuffledThis dataset contains the fully prepared data, which has been tokenized and pre-shuffled, used to train the Pythia (deduplicated) models.
You can find these models under the EleutherAI organisation, and they are also listed in my Memorisation-Profiles collection.
This data is the same as the one found in EleutherAI/pile-deduped-pythia-preshuffled,
but it is presented in a more manageable format. Instead of using the Megatron format used by the GPT-NeoX library, I have stored the data in a… See the full description on the dataset page: https://huggingface.co/datasets/pietrolesci/pile-deduped-pythia-preshuffled.dialogue_nli
Overview
Original dataset available here.
Dataset curation
Original label column is renamed original_label. The original classes are renamed as follows
{"positive": "entailment", "negative": "contradiction", "neutral": "neutral"})
and encoded with the following mapping
{"entailment": 0, "neutral": 1, "contradiction": 2}
and stored in the newly created column label.
The following splits and the corresponding columns are present in the original files
train {'dtype', 'id'… See the full description on the dataset page: https://huggingface.co/datasets/pietrolesci/dialogue_nli.pile-validationThis dataset has been created as an artefact of the paper Causal Estimation of Memorisation Profiles (Lesci et al., 2024).
More info about this dataset in the related collection Memorisation-Profiles.
The validation data used in our study. The Pythia suite does not have an official validation. However, we confirmed with the authors that the Pile validation split (this one) was not seen during training.
It is still a bit confusing whether the Pile data can be released freely. Thus, we will… See the full description on the dataset page: https://huggingface.co/datasets/pietrolesci/pile-validation.stress_tests_nli
Overview
Original dataset page here and dataset available here.
Dataset curation
Added new column label with encoded labels with the following mapping
{"entailment": 0, "neutral": 1, "contradiction": 2}
and the columns with parse information are dropped as they are not well formatted.
Also, the name of the file from which each instance comes is added in the column dtype.
Code to create the dataset
import pandas as pd
from datasets import Dataset, ClassLabel… See the full description on the dataset page: https://huggingface.co/datasets/pietrolesci/stress_tests_nli.gen_debiased_nli
Overview
Original dataset available here.
@inproceedings{gen-debiased-nli-2022,
title = "Generating Data to Mitigate Spurious Correlations in Natural Language Inference Datasets",
author = "Wu, Yuxiang and
Gardner, Matt and
Stenetorp, Pontus and
Dasigi, Pradeep",
booktitle = "Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics",
month = may,
year = "2022",
publisher = "Association for Computational… See the full description on the dataset page: https://huggingface.co/datasets/pietrolesci/gen_debiased_nli.joci
Overview
Original dataset available here.
This dataset is the "full" JOCI dataset, which is the file named joci.csv.zip.
Dataset curation
The following processing is applied,
label column renamed to original_label
creation of the label column using the following mapping, using common practices (1, 2)
{
0: "contradiction",
1: "contradiction",
2: "neutral",
3: "neutral",
4: "neutral",
5: "entailment",
}
finally, converting this to the usual… See the full description on the dataset page: https://huggingface.co/datasets/pietrolesci/joci.pierogue
Pierogue
Pierogue is a small open-licensed machine-generated dataset that contains fifteen short texts in English covering five topics, provided with the relevance judgements (qrels), designed for educational purposes.
Topics: cosmos, nature, music, technology, fashion
Splits: train (10 documents, 375 qrels) and test (5 documents, 150 qrels)
Texts were generated by ChatGPT 3.5. Queries, qrels, and analogies were generated by GPT-4. Words were provided with Word2Vec embeddings… See the full description on the dataset page: https://huggingface.co/datasets/dustalov/pierogue.civilcomments-wildsThis is the CivilComments datasets available in the wilds library library and downloadable from codalab
whose README.md file is
# Release notes for CivilComments
## v1.0
Dec 9, 2020: First release.
There are two configurations:
default: The processed data (more details below)
raw: The raw datasets as downloaded from the link above
The default configuration has been created using the processing function reported in processing_script.pymultiwoz_all_versionsThis dataset is based on the "cumulative" configuration (i.e., slots previously filled are kept in the state)
of the MultiWoz 2.2 dataset available also at pfb30/multi_woz_v22.
Therefore, the system and user utterances, the active intents, and the services are exactly the same.
In addition to the data present in version 2.2, this dataset contains
the annotations from versions 2.1, 2.3, and 2.4 for each dialogue turn.
This dataset is an artefact of the paper Diable: Efficient Dialogue State… See the full description on the dataset page: https://huggingface.co/datasets/pietrolesci/multiwoz_all_versions.eurlex-57k
Dataset Card for "eurlex-57k"
More Information needed