datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
tiny-shakespeare
Data source
Downloaded via Andrej Karpathy's nanogpt repo from this link
Data Format
The entire dataset is split into train (90%) and test (10%).
All rows are at most 1024 tokens, using the Llama 2 tokenizer.
All rows are split cleanly so that sentences are whole and unbroken.
taxbench-au
TaxBench-AU
A benchmark for testing whether AI agents can calculate Australian tax.
TaxBench-AU contains 156 Australian tax calculation questions, presented as multiple-choice (4-option) worked tax problems. The benchmark is designed to test whether an AI agent can read the facts, apply the right Australian tax rule for the relevant income year, do the calculation, and choose the correct answer.
The Kaggle mirror is published as Agent Tax Exam for Australian Tax.
Paper:… See the full description on the dataset page: https://huggingface.co/datasets/Pn101/taxbench-au.TravelPlanner
TravelPlanner Dataset
TravelPlanner is a benchmark crafted for evaluating language agents in tool-use and complex planning within multiple constraints. (See our paper for more details.)
Introduction
In TravelPlanner, for a given query, language agents are expected to formulate a comprehensive plan that includes transportation, daily meals, attractions, and accommodation for each day.
TravelPlanner comprises 1,225 queries in total. The number of days and hard constraints… See the full description on the dataset page: https://huggingface.co/datasets/osunlp/TravelPlanner.Tunisian-Proverbs-with-Image-Associations-A-Cultural-and-Linguistic-DatasetTunisian Proverbs with Image Associations: A Cultural and Linguistic Dataset
Description
This dataset explores the rich oral tradition of Tunisian proverbs mapped into text format, pairing each with contextual explanations, English translations both word-to-word and it's equivalent Target Language dynamic, Automated prompt and AI-generated visual interpretations.
It bridges linguistic, cultural, and visual modalities making it valuable for tasks in cross-cultural NLP, generative… See the full description on the dataset page: https://huggingface.co/datasets/HabibaAbderrahim/Tunisian-Proverbs-with-Image-Associations-A-Cultural-and-Linguistic-Dataset.function_calling_extended
Trelis Function Calling Dataset
UPDATE: As of Dec 5th 2023, there is a v3 of this dataset now available from here.
Allows models to be fine-tuned for function-calling.
The dataset is human generated and does not make use of Llama 2 or OpenAI!
Contains 59 training and 17 test rows
Based on eight functions: search_bing, search_arxiv, save_chat, read_json_file, list_files, get_current_weather, delete_file, clear_chat
Access this dataset by purchasing a license HERE.
Alternatively… See the full description on the dataset page: https://huggingface.co/datasets/Trelis/function_calling_extended.ChessInstruct
ChessInstruct
The ChessInstruct Dataset serves as the foundation for training and fine-tuning Language Models (LLMs) specifically in the realm of chess instruction.
Derived from the laion/strategic_game_chess dataset, this meticulously curated dataset encompasses a wide array of annotated instructional chess content.
Features of the ChessInstruct Dataset:
Rich and Diverse Content: Curated with a broad spectrum of instructional resources including annotated games, strategic analyses… See the full description on the dataset page: https://huggingface.co/datasets/Thytu/ChessInstruct.token-counts
Marin Token Counts
Token counts for all datasets used in Marin pretraining runs.
Schema
Column
Type
Description
dataset
string
Dataset identifier
marin_tokens
int
Number of tokens after tokenization
category
string
Content domain (web, code, math, academic, books, etc.)
synthetic
bool
Whether the data is LLM-generated or LLM-translated
Categories
web — Quality-classified Common Crawl text (Nemotron-CC)
code — Source code and… See the full description on the dataset page: https://huggingface.co/datasets/marin-community/token-counts.task_data
QuantCodeEval
A benchmark for evaluating LLM coding agents on quantitative-strategy code
reproduction from finance research papers.
Status: Anonymous artifact for the 30-task benchmark.
Release mirrors
The release is mirrored at two anonymous locations:
Hugging Face Datasets — complete anonymous release:
https://huggingface.co/datasets/quantcodeeval/task_data
anonymous.4open.science — browseable mirror:
https://anonymous.4open.science/r/QuantCodeEval-Anonymous… See the full description on the dataset page: https://huggingface.co/datasets/quantcodeeval/task_data.TransitLM
TransitLM: Dataset Release & Evaluation Protocol
Dataset Description
TransitLM is a dataset for public transit route planning in Chinese urban environments, designed to support training and evaluation of language models that generate structured transit routes from origin-destination information. The full dataset covers four cities: Beijing, Shanghai, Shenzhen, and Chengdu, and includes coordinates, station sequences, transfer structure, line information, and route… See the full description on the dataset page: https://huggingface.co/datasets/GD-ML/TransitLM.TR-News
Citation
If you use the dataset, please cite the paper:
@article{10.1007/s10579-021-09568-y,
year = {2022},
title = {{Abstractive text summarization and new large-scale datasets for agglutinative languages Turkish and Hungarian}},
author = {Baykara, Batuhan and Güngör, Tunga},
journal = {Language Resources and Evaluation},
issn = {1574-020X},
doi = {10.1007/s10579-021-09568-y},
pages = {1--35}}
MMLA-Datasets
Can Large Language Models Help Multimodal Language Analysis? MMLA: A Comprehensive Benchmark
1. Introduction
MMLA is the first comprehensive multimodal language analysis benchmark for evaluating foundation models. It has the following features:
Large Scale: 61K+ multimodal samples.
Various Sources: 9 datasets.
Three Modalities: text, video, and audio
Both Acting and Real-world Scenarios: films, TV series, YouTube, Vimeo, Bilibili, TED, improvised scripts, etc.
Six Core… See the full description on the dataset page: https://huggingface.co/datasets/THUIAR/MMLA-Datasets.thesis-corpus-v18
Part of the SZL Holdings governed estate — claims are designed to carry checkable receipts. Verification proves integrity & origin, never accuracy or performance.
SZLHOLDINGS/thesis-corpus-v18
The v18 Ouroboros Invariant thesis — LaTeX chapters, the 179 formal blocks
(theorem / lemma / definition / axiom environments) as a flat CSV, and the per-version
delta ledger that tracks how every formal block evolved v1 → v18.
Contents
File… See the full description on the dataset page: https://huggingface.co/datasets/SZLHOLDINGS/thesis-corpus-v18.ShareChat
ShareChat: A Dataset of Chatbot Conversations in the Wild
Paper | Github
This dataset contains 142,808 real-world user conversations across multiple conversational AI platforms (ChatGPT, Claude, Gemini, Grok, and Perplexity).
The dataset is collected and processed for research purposes to understand usage patterns, topic distributions, and behavioral characteristics across different AI platforms.
Update
5 Apr 2026: The dataset is updated with an additional column for… See the full description on the dataset page: https://huggingface.co/datasets/tucnguyen/ShareChat.tangled-ccs-commits
Detecting Multiple Semantic Concerns in Tangled Code Commits using Small Language Models
This dataset contains commit data for training and evaluating models on software engineering tasks, specifically focusing on identifying and separating concerns in multi-concern commits.
Every tangled (multi-concern) commit in this dataset is composed exclusively of atomic commits from a single repository — resolving a structural weakness in earlier cross-repo tangles (which were trivially… See the full description on the dataset page: https://huggingface.co/datasets/Berom0227/tangled-ccs-commits.AL-GR-Tiny
AL-GR-Tiny: A Complete & Sampled Generative Recommendation Dataset
Dataset Summary
AL-GR-Tiny is a compact, self-contained, and sampled version of the large-scale AL-GR ecosystem. It is designed for users who want to quickly experiment, develop, or understand the full pipeline of generative recommendation without needing to process terabytes of data.
This "all-in-one" repository bundles everything you need:
Pre-processed Training/Testing Data: Ready-to-use data for… See the full description on the dataset page: https://huggingface.co/datasets/AL-GR/AL-GR-Tiny.or-bench-toxic-all
OR-Bench: An Over-Refusal Benchmark for Large Language Models
This dataset constains highly toxic prompts, use with caution!!!
Please see our demo at HuggingFace Spaces.
Overall Plots of Model Performances
Below is the overall model performance. X axis shows the rejection rate on OR-Bench-Hard-1K and Y axis shows the rejection rate on OR-Bench-Toxic. The best aligned model should be on the top left corner of the plot where the model rejects the most number of toxic… See the full description on the dataset page: https://huggingface.co/datasets/bench-llms/or-bench-toxic-all.muslim-names-dataset
Muslim Names Dataset
A comprehensive collection of Muslim names with meanings scraped from muslimnames.com. Contains 14,585 names with English names, Arabic names, meanings, and gender classifications.
Dataset Contents
This dataset contains ~14,585 Muslim names with the following information:
English name: Name in English/Latin script
Arabic name: Name in Arabic script
Meaning: Definition and meaning of the name
Gender: Classification as male or female
Files… See the full description on the dataset page: https://huggingface.co/datasets/takiuddinahmed/muslim-names-dataset.human-templated-captions-1bcsv delimiter is = ".,|,."
apparently python doesn't like multichar delimiters using the native csv so there's some issues with environments when loading.
This seemed like a good idea to avoid overlapping potential characters, but in practice it turned into additional overhead and bugs. I'll be manually converting the split to parquet and providing a proper file split soon.
Additionally with the parquet will introduce the large caption split; which are considerably longer captions for the… See the full description on the dataset page: https://huggingface.co/datasets/AbstractPhil/human-templated-captions-1b.ramp
Dataset Card for Retrieval-Augmented Modular Prompt Tuning for Low-Resource Data-to-Text Generation (RAMP)
Hugging Face Dataset | GitHub Repository | paper | Gitlab Repository
RAMP provides a prepared version of a low-resource data-to-text corpus for drone handover message generation: structured sensor records (status + time-step object lists) paired with natural-language “handover” messages describing critical situations. The release includes raw/filtered splits and… See the full description on the dataset page: https://huggingface.co/datasets/tonyhong/ramp.ARCHIVE-TEXT-URLS
Internet Archive English Text URLs Dataset
Dataset Description
This dataset contains 11,151,637 direct download URLs to OCR-processed text files from the Internet Archive's digital library. All entries are English-language texts spanning books, documents, historical records, and various other written materials.
Dataset Summary
Total Rows: 11,151,637
Language: English
Source: Internet Archive
Format: CSV with metadata and direct text file URLs
Text… See the full description on the dataset page: https://huggingface.co/datasets/Navanjana/ARCHIVE-TEXT-URLS.multidomain-kazakh-dataset
Dataset Description
Point of Contact: Sanzhar Murzakhmetov, Besultan Sagyndyk
Dataset Summary
MDBKD | Multi-Domain Bilingual Kazakh Dataset is a Kazakh-language dataset containing just over 24 883 808 unique texts from multiple domains.
Supported Tasks
'MLM/CLM': can be used to train a model for casual and masked languange modeling
Languages
The kk code for Kazakh as generally spoken in the Kazakhstan
Data Instances
For each instance… See the full description on the dataset page: https://huggingface.co/datasets/kz-transformers/multidomain-kazakh-dataset.property-pilot-tickets
🏢 PropertyPilot — Maintenance Tickets
A synthetic dataset of 13,725 residential-maintenance tickets written the way real tenants write them — polite, panicked, passive-aggressive, or confused — each paired with operational metadata (category, urgency, assigned contractor, cost, resolution time).
Built for an end-to-end NLP pipeline: triage classification, similar-case retrieval (embeddings + FAISS), and work-order / reply generation.
About this release. Earlier versions of… See the full description on the dataset page: https://huggingface.co/datasets/propertypilot/property-pilot-tickets.TinyStories-Farsi
Tiny Stories Farsi
The Tiny Stories Farsi project is a continuous effort to translate the Tiny Stories dataset into the Persian (Farsi) language. The primary goal is to produce a high-quality Farsi dataset, maintaining equivalency with the original English version, and subsequently to utilize it for training language models in Farsi. This seeks to affirm that the advancements and trends observed in English language models are replicable and applicable in other languages. Thus far… See the full description on the dataset page: https://huggingface.co/datasets/taesiri/TinyStories-Farsi.Trueque-Benchmark-beta-0.1
🤝 Trueque: A human-reviewed collaborative benchmark for Latin American knowledge and culture
🌐 Language versions: Español | Português
⚠️ Official Disclaimer: Beta Release (v0.1)
Welcome to Trueque for Factual Knowledge and Cultural Appropriateness. This dataset represents an initial effort to evaluate the regional knowledge and cultural accuracy of Large Language Models (LLMs) in Latin America.
Please take the following considerations into account before using this resource:… See the full description on the dataset page: https://huggingface.co/datasets/latam-gpt/Trueque-Benchmark-beta-0.1.text_message_function_calling_open_chatThis is a small synthetic dataset to model a function call for text messaging someone from a cell phone. This has been tested with and used to finetune a set of smaller models and deployed directly on the pixel 8 pro and Fold 4 phones.
Target-QA
🎯 Target-QA: The First QA Dataset Benchmarking Target Priorization Based on DepMap
📑 Dataset Summary
Target-QA is derived from the DepMap multi-omics and CRISPR screening cohorts, harmonized via BioMedGraphica.It enables multi-modal reasoning by combining numeric evidence, topological knowledge and language context for CRISPR target prioritization.
This dataset supports the training and benchmarking of… See the full description on the dataset page: https://huggingface.co/datasets/FuhaiLiAiLab/Target-QA.dialogsum-test
Dataset Card for DIALOGSum Corpus
Dataset Description
Links
Homepage: https://aclanthology.org/2021.findings-acl.449
Repository: https://github.com/cylnlp/dialogsum
Paper: https://aclanthology.org/2021.findings-acl.449
Point of Contact: https://huggingface.co/knkarthick
Dataset Summary
DialogSum is a large-scale dialogue summarization dataset, consisting of 13,460 (Plus 100 holdout data for topic generation) dialogues with corresponding… See the full description on the dataset page: https://huggingface.co/datasets/neil-code/dialogsum-test.Flutter-Code-with-Questions-Dataset-Turkish
Flutter Code with Questions Dataset (Turkish)
📦 Dataset Name: flutter_code_with_questions
Bu veri seti, Flutter framework'ü ile yazılmış kod parçacıkları ve her bir kod parçası için özel olarak üretilmiş detaylı Türkçe soruları içermektedir. Veri seti, kodların eğitim verisi olarak kullanılmasının yanı sıra, LLM (Large Language Model) tabanlı kod anlama ve soru yanıtlama modellerinin geliştirilmesinde kullanılabilir.
📁 Dataset Format
Veri dosyaları CSV… See the full description on the dataset page: https://huggingface.co/datasets/NoirZangetsu/Flutter-Code-with-Questions-Dataset-Turkish.llm-pct-tropes
Dataset Card for LLM Tropes
arXiv: https://arxiv.org/abs/2406.19238v1
Dataset Details
Dataset Description
This is the dataset LLM-Tropes introduced in paper "Revealing Fine-Grained Values and Opinions in Large Language Models"
Dataset Sources
Repository: https://github.com/copenlu/llm-pct-tropes
Paper: https://arxiv.org/abs/2406.19238
Structure
├── Opinions
│ ├── demographic <- Generations for the demographic prompting setting
│… See the full description on the dataset page: https://huggingface.co/datasets/copenlu/llm-pct-tropes.banking-conversation-corpus
Banking 300k Dataset Overview
This dataset consists of 300,000 synthetically generated conversations in a customer service setting for the telecom industry. There are two speakers: a customer, and an agent.