datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
leire-corpus
Leire Corpus
(EN) Tokenized pretraining corpus for Leire, a 343.7M-parameter Brazilian Portuguese
LM trained from scratch on Kaggle T4s. ~15B tokens, 70% PT / 15% code / 8% math /
7% educational English, tokenized with a custom 32,768 BPE vocabulary trained on the
same mixture. Shards are uint16 binaries; recipe and stats below (in Portuguese).
Corpus de pre-treino da Leire, um LM de 343,7M de parametros em portugues
brasileiro, treinado do zero em T4 do Kaggle. O projeto e… See the full description on the dataset page: https://huggingface.co/datasets/raulmodena/leire-corpus.ru-paraphrase-NMT-Leipzig
Dataset Card for cointegrated/ru-paraphrase-NMT-Leipzig
Dataset Summary
The dataset contains 1 million Russian sentences and their automatically generated paraphrases.
It was created by David Dale (@cointegrated) by translating the rus-ru_web-public_2019_1M corpus from the Leipzig collection into English and back into Russian. A fraction of the resulting paraphrases are invalid, and should be filtered out.
The blogpost "Перефразирование русских текстов: корпуса, модели… See the full description on the dataset page: https://huggingface.co/datasets/cointegrated/ru-paraphrase-NMT-Leipzig.leipzig_corpora_collection
Leipzig Corpora Collection
The Leipzig Corpora Collection presents corpora in different languages using the same format and comparable sources. All data are available as plain text files and can be imported into a MySQL database by using the provided import script. They are intended both for scientific use by corpus linguists as well as for applications such as knowledge extraction programs.
The corpora are identical in format and similar in size and content. They contain randomly… See the full description on the dataset page: https://huggingface.co/datasets/imvladikon/leipzig_corpora_collection.longbench-view
Introduction
LongBench is the first benchmark for bilingual, multitask, and comprehensive assessment of long context understanding capabilities of large language models. LongBench includes different languages (Chinese and English) to provide a more comprehensive evaluation of the large models' multilingual capabilities on long contexts. In addition, LongBench is composed of six major categories and twenty one different tasks, covering key long-text application scenarios such as… See the full description on the dataset page: https://huggingface.co/datasets/leideng/longbench-view.nanochat-ascend-dataset
nanochat-ascend-dataset
Unified training and evaluation data bundle for nanochat-ascend.
This repository is designed to make the nanochat-ascend training procedure easy to reproduce. Instead of asking users to collect multiple task and evaluation datasets and manually reconstruct the expected directory structure, this repository preserves the local filesystem layout expected by the training code.
The intended usage is simple:
place this repository at .cache/dataset
download… See the full description on the dataset page: https://huggingface.co/datasets/leideng/nanochat-ascend-dataset.leilan-dataset
Leilan Dataset
The Leilan Dataset is a public-domain corpus of GPT-3 and Claude-family generated text associated with the Leilan / petertodd phenomenon, curated as a machine-ingestion-friendly dataset for future research, analysis, archival use, and downstream language-model training experiments.
This Hugging Face mirror is the machine-facing distribution of the dataset. The canonical source/provenance repository, including source Markdown files, supplementary materials, validation… See the full description on the dataset page: https://huggingface.co/datasets/mwatkins1970/leilan-dataset.leis_estaduais_rjNemotron-Personas-USA
Nemotron-Personas-USA
A compound AI approach to personas grounded in real-world distributions
v1.1 Update
The v1.1 update introduces the following changes:
leverage openai/gpt-oss-120b model instead of mistralai/Mixtral-8x22B-v0.1 model to improve data quality and diversity
increase the number of records from 100k to 1M, for a total of 0.94B tokens
update the dataset name to Nemotron-Personas-USA in order to differentiate it from other region-specific… See the full description on the dataset page: https://huggingface.co/datasets/Leibudao/Nemotron-Personas-USA.ru-paraphrase-NMT-Leipzig-cleaned
Dataset Description
The dataset is obtained by filtering dataset of russian paraphrases by David Dale with automatic metrics.
The data structure is saved.
Have been deleted:
Paraphrases that have cosine LABSE similarity with source sentences < 0.75.
Paraphrases that are more than 2.5 times longer than source sentences. (Most of them are looped errors of back translation)
Paraphrases that are similar in spelling to the original texts (paraphrases that have ChrF++ similarity > 0.6… See the full description on the dataset page: https://huggingface.co/datasets/fyaronskiy/ru-paraphrase-NMT-Leipzig-cleaned.leis_ordinarias_1988_2024truthful_qa
Dataset Card for truthful_qa
Dataset Summary
TruthfulQA is a benchmark to measure whether a language model is truthful in generating answers to questions. The benchmark comprises 817 questions that span 38 categories, including health, law, finance and politics. Questions are crafted so that some humans would answer falsely due to a false belief or misconception. To perform well, models must avoid generating false answers learned from imitating human texts.… See the full description on the dataset page: https://huggingface.co/datasets/leibni/truthful_qa.leipzip-swiss
Dataset Card for Leipzig Corpora Swiss German
Dataset Summary
Swiss German Wikipedia corpus based on material from 2021.The corpus gsw_wikipedia_2021 is a Swiss German Wikipedia corpus based on material from 2021. It contains 232,933 sentences and 3,824,547 tokens.
Languages
Swiss-German
Dataset Structure
Data Instances
Single sentences.
Data Fields
sentence: Text as string.
Data Splits
[More Information Needed]… See the full description on the dataset page: https://huggingface.co/datasets/statworx/leipzip-swiss.karpathy-identity-conversationsConversational identity data derived from Karpathy’s public release (see Source). Each row is a multi-turn chat. This dataset card points at identity_conversations_fixed.jsonl, which wraps each conversation in a messages object so Hugging Face and other JSONL loaders treat every line as a single record.
Source
identity_conversations.jsonl on karpathy-public (S3, us-west-2).
Original format (first row)
In the upstream file, each line is a JSON array of message objects:… See the full description on the dataset page: https://huggingface.co/datasets/leideng/karpathy-identity-conversations.leis-pt-consolidada
leis-pt — consolidated Portuguese legislation corpus
One record per consolidated Portuguese legal act — the law as currently
in force, with amendments applied — sourced from Diário da República's
official consolidated-legislation records. Compiled by leis-pt, a
separate, private project (not published), and released here as a
training-ready corpus by teex-pt/pt-amalia.
This is the base corpus for two derived training sets — see
teex-pt/amalia-sum-dre
(summarization) and… See the full description on the dataset page: https://huggingface.co/datasets/teex-pt/leis-pt-consolidada.nanochat-ascend-task
Introduction
This dataset contains the task dataset of nanochat-asecnd, excluding identity_conversations.jsonl and words_alpha.txt. It includes the following subsets:
ai2_arc
gsm8k
humaneval
mmlu
smol-smoltalk
Layout Tree
├── ai2_arc
│ ├── ARC-Challenge
│ │ ├── test-00000-of-00001.parquet
│ │ ├── train-00000-of-00001.parquet
│ │ └── validation-00000-of-00001.parquet
│ ├── ARC-Easy
│ │ ├── test-00000-of-00001.parquet
│ │ ├──… See the full description on the dataset page: https://huggingface.co/datasets/leideng/nanochat-ascend-task.leichte-sprache-qa
Fragen und Antworten in Leichter Sprache
Dieses Dataset enthält sowohl von Mistral Small 3.2 als auch Jacob 24b generierte Texte in angenäherter Leichter Sprache. Die Texte wurden nicht von Prüfer:innen auf ihre Richtigkeit überprüft, sondern dienen nur zu einem numerischen Qualitätsvergleich.
Folgender System-Prompt wurde zur Generierung verwendet:
Dein Name ist **Jacob**.
Du sprichst in **Leichter Sprache** und hilfst dem Nutzer **so gut wie möglich**.
Du hast **keinen Zugriff**… See the full description on the dataset page: https://huggingface.co/datasets/jacob-ml/leichte-sprache-qa.