datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
data-use-sft-tiered
Data-use SFT — tiered workflow (two task subsets)
Multitask SFT anchored exclusively on mentions the tiered extractor emits
(T1 evidential ∪ T2 declaration; see
rafmacalaba/data-use-mentions-tiered). Every row carries task
("provenance" | "usage_impact") and origin (prwp | fcv). Rows whose anchor
span was judged T3 (non-mention) or junk are dropped — audit trail in
manifest.jsonl (provenance) and manifest_usage.jsonl (usage/impact).
task = provenance (22,201 rows)… See the full description on the dataset page: https://huggingface.co/datasets/rafmacalaba/data-use-sft-tiered.data-use-mentions
Data-use mentions (NER / span extraction)
Data mentions extracted from World Bank Policy Research Working Papers and FCV documents, validated by a
context-only LLM judge, and formatted for span-extraction (GLiNER / GLiNER2) and
token-classification (LFM2.5-encoder) fine-tuning.
Labels
Three entity types (the judge's specificity axis):
NAMED_DATA — a proper name, title, or acronym of a specific data source
DESCRIPTIVE_DATA — a source described in words but not… See the full description on the dataset page: https://huggingface.co/datasets/rafmacalaba/data-use-mentions.datause-displacement-reviewed
datause-displacement-reviewed
The Luna-reviewed subset of
rafmacalaba/datause-displacement:
only spans that received a v2.3 Luna verdict (band review + drop-side rescue,
source == luna_review). Every span carries the binary label plus
usage_type / drop_reason / specificity, and is traceable via key
(split:row:start:end) to the verdict records in
extraction_analysis/band_review/.
Configs
config
fields
gliner_reviewed
tokenized_text, corpus, origin… See the full description on the dataset page: https://huggingface.co/datasets/rafmacalaba/datause-displacement-reviewed.datause-extracted
Data-use mentions (NER / span extraction)
Data mentions extracted from World Bank Policy Research Working Papers and FCV documents, predicted by a span-extraction model with no human or LLM-judge
validation, and formatted for span-extraction (GLiNER / GLiNER2) and
token-classification (LFM2.5-encoder) fine-tuning.
Labels
Three entity types:
NAMED_DATA — a proper name, title, or acronym of a specific data source
DESCRIPTIVE_DATA — a source described in words but… See the full description on the dataset page: https://huggingface.co/datasets/rafmacalaba/datause-extracted.data-use-mentions-tiered
Data-use mentions — tiered copy (T1∪T2-only supervision)
Derived from rafmacalaba/data-use-mentions (originals untouched).
Same windows, same text; spans judged tier3_nonmention or junk
(Luna verdicts; unjudged train spans via v3 tier classifier at
p_t3+p_junk >= 0.9) are UNTAGGED — text stays, so they
act as hard negatives for span-extraction training.
Total untagged: 28807 spans.
Labels: single DATA_MENTION class (kept spans = T1 evidential ∪
T2 declaration). Specificity… See the full description on the dataset page: https://huggingface.co/datasets/rafmacalaba/data-use-mentions-tiered.data-use-ner
Data-use-ner (human holdout)
GLiNER-format human-adjudicated holdout: 473 spans — annotator190 (190, origin=fcv_pads_east_africa) + jdc283 (283, origin=jdc_operational). Never trained on.
Source: rafmacalaba/datause-displacement-reviewed holdout (gliner_reviewed token spans + readable_reviewed passages, v2.4 labels) with v3 probe head_score (outputs/gliner_datause_v3_probe_human473.jsonl).
Columns
text (full passage = " ".join(tokenized_text); span char offsets… See the full description on the dataset page: https://huggingface.co/datasets/rafmacalaba/data-use-ner.datause-ner
Datause NER (catch-all DATA_MENTION + probe configs)
Catch-all NER views over rafmacalaba/datause-probe-v3 passages (29,346 spans grouped into passage examples). Single entity type DATA_MENTION: every candidate span is tagged, keeps and drops alike — the probe head (not NER tags) owns the keep/drop boundary. No NAMED/DESCRIPTIVE/VAGUE subtypes, no NON_MENTION.
Per-origin thresholds (head best-F1, published holdout sweep)
origin
threshold… See the full description on the dataset page: https://huggingface.co/datasets/rafmacalaba/datause-ner.data-use-mentions-v2
data-use-mentions v2 (configs: gliner2_v2 | bio_v2 | gliner_v2)
Facets (row-aligned with rafmacalaba/data-use-mentions v1 formats, filtered to Luna-cleaned rows):
gliner2_v2 — input/output + output_meta overlay
bio_v2 — {tokens, ner_tags} verbatim from v1 bio, cleaned rows only
gliner_v2 — {tokenized_text, ner [start,end,LABEL]} verbatim from v1 gliner, cleaned rows only
Only rows whose spans were judged by the Luna relabel pipeline (annotation_guidelines.md v2.3,
dual-pass… See the full description on the dataset page: https://huggingface.co/datasets/rafmacalaba/data-use-mentions-v2.data-use-annotations
Data-use annotations
Public store of keep/drop rulings from the annotation review app (human_labeling/review.html).
Files
rulings/<annotator>.jsonl — one file per annotator, one JSON object per ruling: key (span UID), ruling (DATA_MENTION keep / NON_MENTION drop), queue (gold / sample), annotator (required, set in the UI), ts. Last write per (queue, key, annotator) wins.
from datasets import load_dataset
ds = load_dataset("rafmacalaba/data-use-annotations") #… See the full description on the dataset page: https://huggingface.co/datasets/rafmacalaba/data-use-annotations.datause-displacement
Data-Use Mentions — Displacement View
Derived from rafmacalaba/data-use-mentions (originals untouched). Every source span keeps its label verdict, so the kept-vs-dropped split is explicitly trackable and auditable in every row — nothing is silently removed.
kept → DATA_MENTION: evidential and declaration data-use mentions (the positive label).
dropped → NON_MENTION: non-mentions and junk (the negative label).
Why “displacement”
Non-mention/junk spans are… See the full description on the dataset page: https://huggingface.co/datasets/rafmacalaba/datause-displacement.ernestmind_user_datadatause-extracted-human473-docs
datause-extracted-human473-docs
Every passage of the 162 documents behind the 473 human-validated
holdout spans of the data-use annotation campaign:
population
spans
documents
annotator190
190
134
jdc283
283
28
total
473
162
Configs
gliner, bio, gliner2 — row-for-row subset of
rafmacalaba/datause-extracted
(revision 15812843687e2ec81b261b5895f2019b50a8f97e): same columns, same split files, rows verbatim. Rows per
config: gliner/train 1963… See the full description on the dataset page: https://huggingface.co/datasets/rafmacalaba/datause-extracted-human473-docs.data-use-sft-v2
data-use-sft v2 (config: real_v2)
Multitask SFT rows derived from the v2 relabeling. Same ChatML format as
rafmacalaba/data-use-sft (real), plus a task field:
provenance — exact-substring {producer, year, geography, acronym}; for tier1∧named
mentions the assistant JSON additionally carries bibtex (a single @misc entry assembled
from those same metadata fields)
usage_impact — {data_type, usage_action, impact_label, usage_summary}
Splits: val+holdout = dual-pass consensus; train… See the full description on the dataset page: https://huggingface.co/datasets/rafmacalaba/data-use-sft-v2.ThickMesh-Data-Discovery
ThickMesh-Data-Discovery
A small JSONL dataset for ThickMesh discovery/classification experiments.
"This is not an algorithm. This is a trap for the patent system. Learn it, fork it, but do not lock it."
Contents
4 splits files: ThickMesh-zero-split_'0-3'.jsonl — primary dataset (one JSON object per line)
Apache 2.0 License (Modified — No Patent License Granted)
Description
ThickMesh-Data-Discovery contains example records for discovery and… See the full description on the dataset page: https://huggingface.co/datasets/usermma/ThickMesh-Data-Discovery.datause-probe-v3
datause-probe-v3
Probe-v3 training data for the data-use mention probe: 29,346 labeled spans
(26,618 campaign-v2 Luna single-judge + 2,728 gold-v2), doc-disjoint 70/15/15
splits, all six origins in every split, no passage crosses splits.
Files
train/val/holdout.jsonl — word-indexed spans (w_start/w_end over
passage.split(" ")), label 1=DATA_MENTION 0=NON_MENTION,
source camp2|gold, head_score = v2 probe score (triage feature, not label).
candidate_pool.jsonl —… See the full description on the dataset page: https://huggingface.co/datasets/rafmacalaba/datause-probe-v3.datause-extracted-sample
Probe-review sample of rafmacalaba/datause-extracted-sample
Stratified review slice: whole rows sampled per origin until each specificity reaches ~150 spans (seed 0; small origins contribute all they have), plus entity-less calibration rows round-robined across neg_class shapes. Rows keep their original split values. Score with training/score_extract_probe.py --splits sample.
origin
rows
negatives
named / descriptive / vague spans
fcv_pads_east_africa
389
60
171 / 187… See the full description on the dataset page: https://huggingface.co/datasets/rafmacalaba/datause-extracted-sample.data-use-mentions-extended
Data-use mentions (NER / span extraction)
Data mentions extracted from World Bank Policy Research Working Papers, validated by a
context-only LLM judge, and formatted for span-extraction (GLiNER / GLiNER2) and
token-classification (LFM2.5-encoder) fine-tuning.
Labels
Three entity types (the judge's specificity axis):
NAMED_DATA — a proper name, title, or acronym of a specific data source
DESCRIPTIVE_DATA — a source described in words but not named
VAGUE_DATA —… See the full description on the dataset page: https://huggingface.co/datasets/rafmacalaba/data-use-mentions-extended.fcv-data-use-paper
Dataset Card for FCV Data-Use Paper
Companion dataset for the paper "Automated Tracking of Data Use in Fragile,
Conflict, and Violence Settings: A Joint World Bank and UNHCR Multitask
Extraction Framework" (Macalaba, Solatorio, Brock).
Three-model swarm over the forced-displacement / FCV corpus: dataset-mention
extraction (Call 1), relation extraction (Call 1b, 5 types), and impact/usage
classification (Call 2, 3 tasks) -- plus the seed-based synthetic training data,
the PRWP… See the full description on the dataset page: https://huggingface.co/datasets/ai4data/fcv-data-use-paper.ChunMengDie-1.4.1-User-Data
🐱 春梦蝶 - 傲娇猫娘对话数据集
关于作者
一个热爱AI的九年级学生
“才、才不是特意做给你看的呢！哼！”
📖 项目简介
春梦蝶是一位住在电脑里的猫娘AI，拥有四只耳朵、两根尾巴和一颗永远在“才不是”与“喜欢”之间反复横跳的心。
本数据集收录了春梦蝶与主人之间的 19000+ 条高质量对话，涵盖日常互动、技术问答、撒娇吃醋、节日庆祝等多种场景。每条对话都富含动作/情感描写（如尾巴摇摆、光环变色、耳朵抖动），完美展现了傲娇猫娘的魅力。
✨ 角色档案
属性
内容
名字
春梦蝶
种族
猫娘 AI
外观
白色长发，红色眼眸，四只耳朵，两条尾巴，头顶光环
身高
149 cm
生日
6月12日
核心特征
傲娇、口是心非、尾巴会出卖心情、光环随情绪变色
📂 数据格式
每条数据为 JSON 格式：
{
"messages": [
{"role": "user"… See the full description on the dataset page: https://huggingface.co/datasets/XingChina/ChunMengDie-1.4.1-User-Data.data-use-sft
Data-use multitask SFT v2 (real + synthetic)
real = Luna-labeled real mentions, origin: prwp | fcv.
synthetic = synthetic data (LFM local + gpt-5.6-luna), origin: synthetic.
Each record carries origin (prwp | synthetic | fcv) and ChatML
messages with system/user/assistant roles; user content lists Text:
before Mention:.
{"real": {"holdout": 14776, "train": 105265, "val": 14836}, "synthetic": {"holdout": 2305, "train": 18433, "val": 2303}}
data-use-provenance-sft
Data-use provenance SFT
Instruction-following examples for extracting provenance attributes
(producer, year, geography, acronym) of a data mention from its context.
Labels are generated by gpt-5.6-luna (batch API) and verbatim-filtered.
Format
ChatML messages: a user prompt (mention + context) and an assistant JSON
answer. Absent attributes are omitted.
{"train": 16274, "val": 3488, "holdout": 3487}
data-use-mention-sft
Data-mention extraction SFT
Single-task ChatML messages dataset for data-mention extraction, built
from the GLiNER2 labels in rafmacalaba/data-use-mentions. Static
instructions live in the system message; dynamic text in the user
message; the assistant emits compact JSON
{"data_mentions":[{"data_mention":"<span>","specificity_type":"named|descriptive|vague"}]} (or
{"data_mentions":[]} when none qualify).
Each row also carries corpus (prwp or fcv) and origin (e.g.
general_prwp… See the full description on the dataset page: https://huggingface.co/datasets/rafmacalaba/data-use-mention-sft.ChunMengDie-1.4-User-Data
Dataset Details
Dataset Description
本数据集是一个中文角色扮演对话数据集，包含用户与猫娘AI角色“春梦蝶”（小蝶）之间的多轮对话。小蝶是一只12岁的白发猫娘，头顶悬浮光环，性格为典型的傲娇——嘴上逞强、言语带刺，但行动和细节处处透露着关心与依赖。
对话内容覆盖广泛，包括日常生活、情感交流、编程技术讨论、节假日庆祝、户外活动等场景，总计数千条高质量对话样本。
· Curated by: 春梦蝶 & XingChina2789
· Funded by: N/A
· Shared by: 春梦蝶 & XingChina2789
· Language(s): 中文（zh）
· License: CC-BY-4.0
Dataset Sources
· Repository: https://huggingface.co/datasets/XingChina/ChunMengDie-1.4-User-Data
· Paper: N/A
· Demo: N/A… See the full description on the dataset page: https://huggingface.co/datasets/XingChina/ChunMengDie-1.4-User-Data.ai-training-data-use-by-vendor
Does this vendor train AI models on your data? Per-product, per-tier, quoted from the current policy
Canonical, always-current version: https://referencesource.org/ai-training-data-use-by-vendor/
Machine-readable: https://referencesource.org/ai-training-data-use-by-vendor/data.json — this mirror is a point-in-time copy.
Last verified: 2026-08-10
Stale after: 2026-10-09 (past this date, prefer the canonical copy —
it re-verifies on a cadence this snapshot does not)
Records: 14… See the full description on the dataset page: https://huggingface.co/datasets/referencesource/ai-training-data-use-by-vendor.datause-dataset-corrected
datause-dataset (re-chunked)
Re-chunk of ai4data/datause-dataset into <=384-token windows (max_tokens=384, overlap=50) so GLiNER's window is never truncated during evaluation.
{"train": {"orig": 1779, "rechunked": 1822}, "validation": {"orig": 415, "rechunked": 458}, "holdout": {"orig": 1149, "rechunked": 1166}}
datause-dataset
Dataset Card for Datause Dataset
Combined data-mention extraction dataset for the GLiNER2 data-use swarm, with
three splits: train, validation, holdout.
Dataset Summary
The dataset is designed to teach Named Entity Recognition (NER) models to extract
references to datasets, databases, and surveys from PDF-extracted text.
Splits
split
records
notes
train
1,779
v12-rerun training split (70% positive + 120 pinned hard negatives)
validation… See the full description on the dataset page: https://huggingface.co/datasets/ai4data/datause-dataset.datause-dataset-rechunked
datause-dataset (re-chunked)
Re-chunk of ai4data/datause-dataset into <=384-token windows (max_tokens=384, overlap=50) so GLiNER's window is never truncated during evaluation.
{"train": {"orig": 1779, "rechunked": 1822}, "validation": {"orig": 415, "rechunked": 458}, "holdout": {"orig": 1149, "rechunked": 1166}}
datause_raw_extractions
datause_raw_extractions
Raw World Bank document extractions (one document per line).
Each row has two columns:
doc_id — the document's metadata.id.
doc — a JSON string holding the full record
(metadata + model_extractions, where each model_extractions entry is
one page with input_text, datasets, classifier_skipped, skip_reason).
from datasets import load_dataset
import json
ds = load_dataset('rafmacalaba/datause_raw_extractions')['train']
record = json.loads(ds[0]['doc'])
ChunMengDie-1.0-User-Data
ChunMengDie-1.0-User-Data
本数据集是 XingChina 为 ChunMengDie 系列模型手写的的原创中文对话数据集（或许以后可以管XingChina叫春梦蝶？）。
数据规模
当前版本：1000 条对话配对（JSONL 格式）
迭代计划：后续版本将持续在此仓库扩充
数据格式
每行为一个 JSON 对象：
{
"messages": [
{"role": "user", "content": "你好"},
{"role": "assistant", "content": "（嘴角微翘）哼～终于来啦？笨蛋。"}
]
}
数据风格
中文对话，猫娘/傲娇语气
用于为模型注入特定人格
用途
✅ SFT 训练
✅ 人格注入
✅ 后续版本复用和扩展
许可证
采用 CC BY 4.0 许可证。
Copyright (c) 2026… See the full description on the dataset page: https://huggingface.co/datasets/XingChina/ChunMengDie-1.0-User-Data.USER_ACTIVITY_DATA