datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
fineweb-edu
FineWeb-Edu (Lance Format)
A Lance-formatted version of FineWeb-Edu — over 1.5 billion educational web passages with cleaned text, source metadata, language detection signals, and 384-dim text embeddings — available directly from the Hub at hf://datasets/lance-format/fineweb-edu/data/train.lance.
Key features
Cleaned passage text in the text column with the source url and title carried alongside.
Language detection signals (language, language_probability) for filtered… See the full description on the dataset page: https://huggingface.co/datasets/lance-format/fineweb-edu.ms-marco-v2.1-lance
MS MARCO v2.1 QA (Lance Format)
A Lance-formatted version of MS MARCO v2.1 — Microsoft's machine-reading-comprehension benchmark built from anonymized Bing query logs. Each row is one user query, the up-to-10 candidate passages Bing retrieved for it with relevance flags, and the human-written reference answers, with MiniLM query embeddings stored inline and pre-built ANN/FTS indices, available directly from the Hub at hf://datasets/lance-format/ms-marco-v2.1-lance/data.… See the full description on the dataset page: https://huggingface.co/datasets/lance-format/ms-marco-v2.1-lance.hotpotqa-distractor-lance
HotpotQA distractor (Lance Format)
A Lance-formatted version of HotpotQA using the distractor config — multi-hop reading-comprehension questions where each answer requires combining facts from two Wikipedia paragraphs, with 10 candidate paragraphs per question (gold + 8 distractors). The dataset ships with MiniLM question embeddings, flattened context text for full-text search, and pre-built ANN/FTS indices, available directly from the Hub at… See the full description on the dataset page: https://huggingface.co/datasets/lance-format/hotpotqa-distractor-lance.trivia-qa-lance
TriviaQA (Lance Format)
A Lance-formatted version of TriviaQA (rc.nocontext config) — a large reading-comprehension dataset of trivia questions paired with a canonical answer, accepted aliases, and entity-type metadata — with MiniLM question embeddings stored inline and ready for retrieval at hf://datasets/lance-format/trivia-qa-lance/data. The rc.nocontext slice is the standard reading-comprehension form without the multi-gigabyte entity_pages / search_results payloads, which keeps… See the full description on the dataset page: https://huggingface.co/datasets/lance-format/trivia-qa-lance.natural-questions-val-lance
Natural Questions — Validation (Lance Format)
A Lance-formatted version of the Natural Questions validation split — 7,830 real Google search queries paired with the full Wikipedia article a human used to answer them, plus 1–5 annotator labels per question. MiniLM question embeddings are stored inline and the dataset ships with pre-built ANN/FTS indices, all available directly from the Hub at hf://datasets/lance-format/natural-questions-val-lance/data. Sourced from… See the full description on the dataset page: https://huggingface.co/datasets/lance-format/natural-questions-val-lance.DRIFT_QAFT
DRIFT_QAFT Dataset
DRIFT_QAFT is a Question-Answering dataset designed for the DRIFT project, which focuses on decoupling knowledge and reasoning in large language models.
Dataset Summary
This dataset is derived from the Wikipedia dataset released by Wikimedia on Hugging Face:
https://huggingface.co/datasets/wikimedia/wikipedia
The original data comes from Wikipedia snapshots provided by Wikimedia.
Entries are bucketed into long-context intervals based on the… See the full description on the dataset page: https://huggingface.co/datasets/SII-LancelotXie/DRIFT_QAFT.write_doc_sft_v1from datasets import load_dataset
dataset = load_dataset("lancexiao/write_doc_sft_v1")
squad-v2-lance
SQuAD v2 (Lance Format)
A Lance-formatted version of SQuAD v2 — the Stanford Question Answering Dataset with both answerable and deliberately unanswerable questions over Wikipedia passages — with MiniLM question embeddings stored inline and ready for retrieval at hf://datasets/lance-format/squad-v2-lance/data.
Key features
Span-extraction QA over Wikipedia with 130k+ training questions and an is_impossible flag that cleanly separates answerable from unanswerable items.… See the full description on the dataset page: https://huggingface.co/datasets/lance-format/squad-v2-lance.