Datasetpublic

rafmacalaba/datause-displacement

Data-Use Mentions — Displacement View Derived from rafmacalaba/data-use-mentions (originals untouched). Every source span keeps its label verdict, so the kept-vs-dropped split is explicitly trackable and auditable in every row — nothing is silently removed. kept → DATA_MENTION: evidential and declaration data-use mentions (the positive label). dropped → NON_MENTION: non-mentions and junk (the negative label). Why “displacement” Non-mention/junk spans are… See the full description on the dataset page: https://huggingface.co/datasets/rafmacalaba/datause-displacement.

sourceHugging Facecc-by-4.0updated 19d agoView on Hugging Face

0likes131downloads

Dataset Card

Data-Use Mentions — Displacement View

Derived from `rafmacalaba/data-use-mentions` (originals untouched). Every source span keeps its label verdict, so the kept-vs-dropped split is explicitly trackable and auditable in every row — nothing is silently removed.

—kept → `DATA_MENTION`: evidential and declaration data-use mentions (the positive label).
—dropped → `NON_MENTION`: non-mentions and junk (the negative label).

Why “displacement”

Non-mention/junk spans are displaced from the positive label set — so training sees them as negatives — but stay in the data with their verdict attached, rather than being deleted from the record.

Configurations

config	fields	purpose
`gliner_displacement`	`tokenized_text`, `corpus`, `origin`, `language`, `ner`, `spans`	GLiNER-style span extraction; `ner` lists only kept `DATA_MENTION` spans, `spans` carries the per-span `uid`, `tier`, and `source` for full auditability
`bio_displacement`	`tokens`, `ner_tags`, `spans`, `corpus`, `origin`, `language`	token-level BIO with two explicit classes (`DATA_MENTION`, `NON_MENTION`), overlap-safe longest-first; `spans` is the readable reconstruction of the tags
`readable_displacement`	`text`, `spans`, `corpus`, `origin`, `origin_model`, `language`	human-readable: full window `text` with character-offset `spans` (`{start, end, text, label, tier, source}`)

Label scheme

label	meaning
`DATA_MENTION`	kept span — evidential or declaration data-use mention (positive)
`NON_MENTION`	dropped span — non-mention or junk (negative)

Verdict provenance

—Luna judgments win for any span they cover (val/holdout, plus the judged train subset).
—Unjudged train spans use the v3 classifier's predicted class (train_v3_remaining_scores.jsonl).
—A span with no resolvable id stays tagged DATA_MENTION (asymmetric keep: never risk dropping a positive).
—Same-surface conflicts keep when any positive verdict is present.
—A Luna review pass (refine_band.py, label-blind) adjudicated the ~6.3k highest-risk kept spans across all splits; spans it confirmed T3/junk flipped to NON_MENTION (source luna_review), confirmed T1/T2 stayed.

Splits and sizes

Total: 122,864 rows, 148,584 spans (117,380 kept, 31,204 dropped, 1,149 same-surface conflicts).

split	rows	spans	kept (`DATA_MENTION`)	dropped (`NON_MENTION`)	conflicts
train	85,003	103,230	81,995	21,235	758
val	19,152	23,302	18,141	5,161	229
holdout	18,709	22,052	17,244	4,808	162

Source corpora

Rows come from the fcv and prwp corpora with five source origins: fcv_pads_east_africa, general_prwp, reliefweb, refugee_pads, jdc_operational. This dataset contains only real (non-synthetic) rows; origin_model is "" for every row.

Usage

python

from datasets import load_dataset

ds = load_dataset("rafmacalaba/datause-displacement",
                  "readable_displacement", split="train")

The origin field is normalized (legacy suffixes stripped).

{"holdout": {"by_source": {"luna": 21409}, "conflict_spans": 162, "dropped_T3junk": 4808, "kept_T1T2": 17244, "review_drop_flips": 159, "review_keep_flips": 0, "rows": 18709, "spans": 22052, "tiers": {"junk": 299, "tier1_evidential": 13326, "tier2_declaration": 3918, "tier3_nonmention": 4509}}, "train": {"by_source": {"classifier": 70385, "luna": 26779}, "conflict_spans": 758, "dropped_T3junk": 21235, "kept_T1T2": 81995, "review_drop_flips": 1983, "review_keep_flips": 0, "rows": 85003, "spans": 103230, "tiers": {"junk": 1758, "tier1_evidential": 63880, "tier2_declaration": 18115, "tier3_nonmention": 19477}}, "val": {"by_source": {"luna": 22552}, "conflict_spans": 229, "dropped_T3junk": 5161, "kept_T1T2": 18141, "review_drop_flips": 172, "review_keep_flips": 0, "rows": 19152, "spans": 23302, "tiers": {"junk": 299, "tier1_evidential": 14169, "tier2_declaration": 3972, "tier3_nonmention": 4862}}}