rafmacalaba/data-use-annotate

human_labeling

Fork of analysis/v24_sample_review/audit_deck.html as a hostable annotation pipeline. Same blind KEEP/DROP interaction, but the queue comes from the Hub instead of an 11MB embedded ROWS array.

• —Source: rafmacalaba/datause-ner (viewer) for passages; scores are singlepass only — the infer head in rafmacalaba/gliner-datause-catchall-singlepass (sigmoid(head([start; end; mean; ±64 window])) on inference-native features). No v3/encode-head scores anywhere in this tool.
• —Default queue: probe_candidates/pool — the only config with an embedded passage. probe_splits train/val/holdout rows have no passage (surface + word offsets only), so they annotate surface-only unless joined to a chunk corpus (see caveat below).
• —Annotations land in human_labeling/annotations/rulings.jsonl (one JSON object per ruling). On static hosting (github.io) there is no writer: the app exports a JSON file the annotator drops into that folder (PR). On Spaces the bundled app.py accepts POST /api/rulings and appends server-side.

Quick start (local)

uv run python human_labeling/build_gliner_queue.py            # Luna spans + singlepass scores
uv run python human_labeling/app.py  # serves index.html + /api/queue + POST /api/rulings

Data source: the gliner config of rafmacalaba/datause-ner (tokenized passages + catch-all entity spans with camp2 Luna traceability). The queue maps each Luna verdict onto its span, rescores with the singlepass head (probe score) and the GLiNER proposer (extractor score @0.1), and the UI reviews keep/drop with both signals. Retired experiments (pool, passage grouping, gliner2/human473 queues) live only in git history.

Hosting

• —github.io (static): publish human_labeling/ as-is. index.html fetches ./queue_gliner.json (built by build_gliner_queue.py, committed). Rulings persist in localStorage; Export downloads rulings-<annotator>.json, which is committed under annotations/ and reconciled with merge.py.
• —Hugging Face Spaces (SDK: Docker, port 7860): same folder plus app.py. The frontend feature-detects POST /api/rulings: if it 200s, rulings stream to annotations/rulings.jsonl on disk; if not, it falls back to export. Dockerfile provided.

Files

Caveat: probe_splits has no context

probe_train/val/holdout.jsonl rows are key, origin, corpus_id, w_start, w_end, surface, label, source, head_score, split — no passage text. The underlying chunk text lives in per-corpus repos (rafmacalaba/fcv-extractions-meta[-tiered]), and general_prwp (the majority origin) has no published chunk source at all (same gap as demo/server.py: HOLDOUT_META_SOURCES). So:

• —build_pool_queue.py --config probe_candidates (default): full passage, blind ruling quality = audit_deck.
• —build_pool_queue.py --config probe_splits: surface-only triage; items are flagged "ctx_missing": true and the UI shows the surface + scores without a context card. Fine for obvious keeps/drops, not for adjudication.

Bands (decision labels)

Every queue item carries band, tagged from the singlepass score with the bundle's per-origin best-F1 thresholds (probe_labels.py):

Per-origin thresholds (published thresholds.json): fcv_pads_east_africa 0.5 · general_prwp 0.4 · jad_paddy_docs 0.1 · jdc_operational 0.6 · refugee_pads 0.5 · reliefweb 0.4; fallback global_best 0.5.

Ruling schema

{"key": "sample:fcv_pads_east_africa:002082:7:1:0", "ruling": "DATA_MENTION",
 "annotator": "rm", "ts": "2026-09-08T12:00:00Z", "note": "",
 "head_score": 0.61, "origin": "fcv_pads_east_africa", "queue": "probe_candidates/pool"}
`ruling` is `DATA_MENTION` (keep) or `NON_MENTION` (drop) — same labels as the deck export.
## review.html (current)

Multi-span review over `rafmacalaba/data-use-ner`: `gliner` holdout (gold 473),
`to_annotate` pool, per-user `annotate_{paddy,aj,aivin,rafael}[_part2]` sets
(240 passages each, mutually exclusive, stratified by origin × probe tag).
Assignment manifests live in `assignment_manifest.json` (verified equal to
the Hub configs — queue files are append-only; never resampled in place).

### For annotators

1. Open your link (or pick your name on the landing page) — your file loads
   automatically. Type your name if asked; it locks on first ruling.
2. Judge each highlighted mention in its window: `→` next mention,
   `k` keep, `d` drop, `space` skip, `[`/`]` slide the window,
   `-`/`+` resize it. Optional freeform note per ruling.
3. Before ruling you see only `probe: easy` or `probe: confusion`. After
   ruling, Luna's verdict + reasoning appear, with a DISAGREE flag (also a
   toast) when you differ.
4. Every 10% you'll be nudged to **export** a checkpoint (`e` or the button,
   header and footer). Send `rulings-<name>-<queue>.jsonl` to rafael/aivin.
   To resume later, **import** the file back — you jump to the first
   unreviewed mention.

### For coordinators

Drop returned files into `human_labeling/annotations/` and run `merge.py` →
majority verdict per span `key` (ties → REVIEW) in `annotations/adjudicated.jsonl`.
Public app: https://rafmacalaba-data-use-annotate.static.hf.space/