datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
PersonaMem-v2
PersonaMem-v2: Towards Personalized Intelligence via Learning Implicit User Personas and Agentic Memory
📅 We have now released PersonaMem-v3!
🚨 The paper is now released. View the full paper here and codebase here.
Personalization is becoming the next milestone of artificial super-intelligence. AI cannot always satisfy every user, especially on tasks with subjective goals, but personalization offers a path toward pluralistic alignment.… See the full description on the dataset page: https://huggingface.co/datasets/bowen-upenn/PersonaMem-v2.officeqa-pro-v2
OfficeQA Pro v2
Dataset Summary
OfficeQA Pro v2 is a grounded reasoning benchmark by Databricks for evaluating model and agent performance on end-to-end reasoning over real-world documents.
The benchmark consists of question–answer pairs that require reasoning over two centuries of U.S. Federal Accounts of Receipts and Expenditures reporting (1793–2024) — Combined Statements of Receipts, Outlays, and Balances of the United States Government, together with earlier… See the full description on the dataset page: https://huggingface.co/datasets/databricks/officeqa-pro-v2.PersonaMem-v2
PersonaMem-v2: Towards Personalized Intelligence via Learning Implicit User Personas and Agentic Memory
🚨 The paper is now released. View the full paper here and codebase here.
🙌 The dataset has been downloaded over 12,000 times. Thank you everybody for finding our work helpful!
Personalization is becoming the next milestone of artificial super-intelligence. AI cannot always satisfy every user, especially on tasks with subjective goals, but personalization… See the full description on the dataset page: https://huggingface.co/datasets/milanow/PersonaMem-v2.iac-eval-v2
IaC-Eval v2
Modernised Terraform code-generation benchmark — 186 tasks, Terraform 1.15 + OPA 1.16 (Rego v1).
An updated and extended version of the IaC-Eval NeurIPS 2024 benchmark.
Scoring is deterministic: the generated HCL either passes terraform plan + opa eval, or it doesn't — no LLM-as-judge.
Dataset summary
Field
Value
Tasks
186 (AWS only)
Difficulty
1–6 (distribution: 1→35, 2→40, 3→51, 4→22, 5→9, 6→13)
AWS services
34 distinct
Terraform… See the full description on the dataset page: https://huggingface.co/datasets/iac-eval-v2/iac-eval-v2.math-code-qa-v2
Math & Code QA v2 — Instruction Dataset
Worked mathematical solutions and short code answers, spanning arithmetic word
problems through to algebra, geometry and combinatorics.
Built for the Adaption Labs AutoScientist Challenge (Math & Code category).
The model trained on this beats Llama-3.3-70B-Instruct 72 to 28 on the
held-out Math category evaluation.
Rows
5,297 (4,197 math, 1,100 code)
Distinct answers
5,297 (100%)
Duplicate questions
none
Nulls
none… See the full description on the dataset page: https://huggingface.co/datasets/flamiinngo/math-code-qa-v2.CiviVox-Swahili-text-corpus-v2.0
Swahili Text Dataset
Overview
This dataset contains a comprehensive collection of Swahili text data, derived from the AfriBERTa Corpus. It provides a rich resource for natural language processing tasks focused on the Swahili language.
Dataset Details
Source: AfriBERTa Corpus (Swahili subset)
Language: Swahili
Size: 1.54M
Format: Hugging Face Dataset
Content
The dataset consists of two main columns:
id: A unique identifier for each text entry
text:… See the full description on the dataset page: https://huggingface.co/datasets/Adeptschneider/CiviVox-Swahili-text-corpus-v2.0.qwen-v2
qwen-v2
Vietnamese labor-law raw document corpus prepared for continued pretraining.
Files
documents.csv
Columns
text
id
so_ky_hieu
Source
Local file: /home/thaivv/hehe/data/processed/labor_source_pack/core_relationship_cleaned_text_dataset_dict_fix/documents.csv
Rows: 3475
Notes
This dataset is document-level text.
so_ky_hieu is preserved as metadata for each document.
merged-data-v2
Info
This dataset is a merge of the following datasets:
flpelerin/openorca-alpaca-50k
sam-liu-lmi/databricks-dolly-15k-alpaca-style
TokenBender/roleplay_alpaca
vicgalle/alpaca-gpt4
CreitinGameplays/chat-assistant
CreitinGameplays/filter
CiviVox-Swahili-text-corpus-v2.0
Swahili Text Dataset
Overview
This dataset contains a comprehensive collection of Swahili text data, derived from the AfriBERTa Corpus. It provides a rich resource for natural language processing tasks focused on the Swahili language.
Dataset Details
Source: AfriBERTa Corpus (Swahili subset)
Language: Swahili
Size: 1.54M
Format: Hugging Face Dataset
Content
The dataset consists of two main columns:
id: A unique identifier for each text entry
text:… See the full description on the dataset page: https://huggingface.co/datasets/niqqyniqqy/CiviVox-Swahili-text-corpus-v2.0.neo_ara_v2Startups_V2english_karakalpak_pairs_parallel_corpus_v2_8907
English-Karakalpak Parallel Corpus v2 (8.9K)
Dataset Description
English-Karakalpak Parallel Corpus v2 is a high-quality dataset containing 8,906 carefully aligned sentence pairs in English (en) and Karakalpak (kaa).
This dataset is designed to advance the representation and capability of the Karakalpak language in large-scale AI models (LLMs) and Neural Machine Translation (NMT) systems, enabling them to better understand and generate Karakalpak text. This resource… See the full description on the dataset page: https://huggingface.co/datasets/bekan/english_karakalpak_pairs_parallel_corpus_v2_8907.ft-data-v2-colab-hours尝试把我获取到的一些小黄文转换成 instruction fine-tuning 的格式。
ai_response 是小说的原文，每一个章节都是一条数据，然后使用 mistral-nemo 生成三条 user prompt 出来，放在 user_input 1, 2, 3里面。
显然小说还需要做一些数据清理，不过生成出来的 user prompt 质量还不错。
数据只有165条，因为我是在 colab 上跑的量化 mistral-nemo，然后把我免费版的GPU用量跑炸了。
这是我的一些计算:
在 Tesla T4 (Google Colab 免费版)上:
生成一条user prompt 的平均時間在 20秒左右 (5s -> 25s不等)
每個 txt 要生成三次數據, 所以每個txt 耗時约 1min
有235,312個數據, 用一張T4 狂跑 大約要花163天
總字符數量: 1420689772, 也就是14億字
中文字符用mistral-nemo 的tokenizer, 我的尝试是 2979字 -> 4156 token, 也就是1.39 倍, word to token… See the full description on the dataset page: https://huggingface.co/datasets/radeon-zhang/ft-data-v2-colab-hours.