CoolFace
Datasetpublic

rafmacalaba/datause-ner

Datause NER (catch-all DATA_MENTION + probe configs) Catch-all NER views over rafmacalaba/datause-probe-v3 passages (29,346 spans grouped into passage examples). Single entity type DATA_MENTION: every candidate span is tagged, keeps and drops alike — the probe head (not NER tags) owns the keep/drop boundary. No NAMED/DESCRIPTIVE/VAGUE subtypes, no NON_MENTION. Per-origin thresholds (head best-F1, published holdout sweep) origin threshold… See the full description on the dataset page: https://huggingface.co/datasets/rafmacalaba/datause-ner.

sourceHugging Faceapache-2.0updated 13d agoView on Hugging Face
0likes179downloads
Dataset Card

Datause NER (catch-all DATA_MENTION + probe configs)

Catch-all NER views over rafmacalaba/datause-probe-v3 passages (29,346 spans grouped into passage examples). Single entity type DATA_MENTION: every candidate span is tagged, keeps and drops alike — the probe head (not NER tags) owns the keep/drop boundary. No NAMED/DESCRIPTIVE/VAGUE subtypes, no NON_MENTION.

Per-origin thresholds (head best-F1, published holdout sweep)

originthreshold
fcv_pads_east_africa0.5
general_prwp0.3
jad_paddy_docs0.3
jdc_operational0.5
refugee_pads0.6
reliefweb0.3

Columns

  • —gliner: tokenized_text + ner — every candidate span tagged DATA_MENTION (catch-all; keeps and drops alike).
  • —gliner2: input + output (data_mention = all span strings).
  • —Keep/drop lives in spans[].luna_label + head_score/threshold traceability, not in tags — the probe head owns that boundary.
  • —Traceability on every row: corpus_id, page, chunk, split, spans (per-span text, pred, lunalabel, headscore, threshold, char offsets, key, source). Luna keep/drop verdicts are preserved in spans[].luna_label for audit, but tagging follows the head-threshold operating point.

Splits

Doc-disjoint train / val / holdout, inherited from rafmacalaba/datause-probe-v3 (outputs/probe_v3_data), so NER splits match probe splits. See split_stats.json.

Usage

python
from datasets import load_dataset
gliner  = load_dataset("rafmacalaba/datause-ner", "gliner")
gliner2 = load_dataset("rafmacalaba/datause-ner", "gliner2")
# catch-all supervision: every span is DATA_MENTION:
#   gliner['train']['ner'] / gliner2['train']['output']

Files

  • —gliner_{train,val,holdout}.jsonl, gliner2_{train,val,holdout}.jsonl
  • —labels.json, thresholds.json, split_stats.json
  • —probe_{train,val,holdout}.jsonl, probe_candidate_pool.jsonl, probe_split_stats.json — mirror of rafmacalaba/datause-probe-v3 (same data, probe_* configs)

Probe configs (same data as rafmacalaba/datause-probe-v3)

python
splits = load_dataset("rafmacalaba/datause-ner", "probe_splits")
cands  = load_dataset("rafmacalaba/datause-ner", "probe_candidates")