rafmacalaba/datause-ner

1---2task_categories:3  - token-classification4tags:5  - ner6  - span-extraction7  - data-mentions8license: apache-2.09configs:10  - config_name: gliner11    data_files:12      - split: train13        path: gliner_train.jsonl14      - split: val15        path: gliner_val.jsonl16      - split: holdout17        path: gliner_holdout.jsonl18  - config_name: gliner219    data_files:20      - split: train21        path: gliner2_train.jsonl22      - split: val23        path: gliner2_val.jsonl24      - split: holdout25        path: gliner2_holdout.jsonl26  - config_name: probe_splits27    data_files:28      - split: train29        path: probe_train.jsonl30      - split: val31        path: probe_val.jsonl32      - split: holdout33        path: probe_holdout.jsonl34  - config_name: probe_candidates35    data_files:36      - split: pool37        path: probe_candidate_pool.jsonl38---39 40# Datause NER (catch-all DATA_MENTION + probe configs)41 42Catch-all NER views over `rafmacalaba/datause-probe-v3` passages (29,346 spans grouped into passage examples). Single entity type **DATA_MENTION**: every candidate span is tagged, keeps and drops alike — the probe head (not NER tags) owns the keep/drop boundary. No NAMED/DESCRIPTIVE/VAGUE subtypes, no NON_MENTION.43 44## Per-origin thresholds (head best-F1, published holdout sweep)45 46| origin | threshold |47|---|---|48| `fcv_pads_east_africa` | 0.5 |49| `general_prwp` | 0.3 |50| `jad_paddy_docs` | 0.3 |51| `jdc_operational` | 0.5 |52| `refugee_pads` | 0.6 |53| `reliefweb` | 0.3 |54 55## Columns56 57- `gliner`: `tokenized_text` + `ner` — every candidate span tagged `DATA_MENTION` (catch-all; keeps and drops alike).58- `gliner2`: `input` + `output` (`data_mention` = all span strings).59- Keep/drop lives in `spans[].luna_label` + `head_score`/`threshold` traceability, not in tags — the probe head owns that boundary.60- Traceability on every row: `corpus_id`, `page`, `chunk`, `split`, `spans` (per-span text, pred, luna_label, head_score, threshold, char offsets, key, source). Luna keep/drop verdicts are preserved in `spans[].luna_label` for audit, but tagging follows the head-threshold operating point.61 62## Splits63 64Doc-disjoint `train` / `val` / `holdout`, inherited from `rafmacalaba/datause-probe-v3` (`outputs/probe_v3_data`), so NER splits match probe splits. See `split_stats.json`.65 66## Usage67 68```python69from datasets import load_dataset70gliner  = load_dataset("rafmacalaba/datause-ner", "gliner")71gliner2 = load_dataset("rafmacalaba/datause-ner", "gliner2")72# catch-all supervision: every span is DATA_MENTION:73#   gliner['train']['ner'] / gliner2['train']['output']74```75 76## Files77 78- `gliner_{train,val,holdout}.jsonl`, `gliner2_{train,val,holdout}.jsonl`79- `labels.json`, `thresholds.json`, `split_stats.json`80- `probe_{train,val,holdout}.jsonl`, `probe_candidate_pool.jsonl`, `probe_split_stats.json` — mirror of `rafmacalaba/datause-probe-v3` (same data, `probe_*` configs)81 82## Probe configs (same data as `rafmacalaba/datause-probe-v3`)83 84```python85splits = load_dataset("rafmacalaba/datause-ner", "probe_splits")86cands  = load_dataset("rafmacalaba/datause-ner", "probe_candidates")87```88