rafmacalaba/datause-extracted

Data-use mentions (NER / span extraction)

Data mentions extracted from World Bank Policy Research Working Papers and FCV documents, predicted by a span-extraction model with no human or LLM-judge validation, and formatted for span-extraction (GLiNER / GLiNER2) and token-classification (LFM2.5-encoder) fine-tuning.

Labels

Three entity types:

• —NAMED_DATA — a proper name, title, or acronym of a specific data source
• —DESCRIPTIVE_DATA — a source described in words but not named
• —VAGUE_DATA — generic data wording with no identifiable source

O (BIO) = background, including hard negatives.

Negative strategy

• —No hard negatives — every predicted span is kept as a positive.
• —Negative-only chunks (no predicted spans) are sampled at a controlled ratio per split.

Re-chunking (why windows are <= 384 tokens)

The upstream input_text is already an extractor chunk of at most max_tokens (default 384) whitespace tokens, so it fits GLiNER/GLiNER2's context window as-is; a sliding-window re-chunk pass runs only as a safety net.

Provenance

Every row contains corpus (prwp or fcv) and origin (the extraction config, e.g. general_prwp, fcv_pads_east_africa, jdc_operational, refugee_pads, reliefweb). See split_stats.json for per-split provenance counts.

Configs

• —gliner — {"tokenized_text": [...], "ner": [[start, end, label], ...]} (word-level spans)
• —bio — {"tokens": [...], "ner_tags": ["O", "B-NAMED_DATA", ...]}
• —gliner2 — {"input": "...", "output": {"entities": {...}, "entity_descriptions": {...}}} (GLiNER2 flat-NER, span strings + descriptions)

Source (raw extraction, no judge)

Unlike the judged build, every span here is a raw prediction of rafmacalaba/gliner_datause (labels NAMED_DATA, DESCRIPTIVE_DATA, VAGUE_DATA, footnotelink=True, dedupeoverlap=True). There are no hard negatives; negatives are entity-less chunks sampled at ~0.25 per positive per split. Treat spans as noisy positives, not gold labels.

Traceability columns

Every config carries the same flat provenance columns alongside its training keys, so any row joins back to the parsed .md (<origin>/<corpus_id with ':' -> '_'>.md), the source PDF, and the extraction run:

• —corpus_id — document id (<origin>:<NNNNNN>); page, chunk — position within the document
• —title, pdf_url — document title and source PDF URL
• —extractor — model id that produced the spans; footnote_link, dedupe_overlap — pipeline flags baked into input/text
• —has_data_score — chunk-level data score (1.0 iff spans present)
• —split — which file the row came from (train / val / holdout)
• —spans — full span detail the training keys collapse: [{text, label, score, start, end}] with char offsets into input/text

Splits

Document-disjoint train / val / holdout (70% / 15% / 15%). See split_stats.json.

Usage

from datasets import load_dataset
gliner  = load_dataset("rafmacalaba/datause-extracted", "gliner")
bio     = load_dataset("rafmacalaba/datause-extracted", "bio")
gliner2 = load_dataset("rafmacalaba/datause-extracted", "gliner2")

Files

• —gliner_{train,val,holdout}.jsonl, bio_{train,val,holdout}.jsonl, gliner2_{train,val,holdout}.jsonl
• —labels.json (label lists), split_stats.json (per-split counts)