fromhope/acubench

AcuBench

Indication-based acupoint-set recommendation. Given an indication / symptom string (e.g. "Headache"), predict the set of WHO-standard acupoints indicated for it, grounded in AcuKG's Indication table, over a fixed 361-point label space. AcuBench is a small (446-sample) benchmark with a dedicated conformal-prediction calibration split.

Not a clinical prescription benchmark. A row's acupoint set is "acupoints indicated for this symptom in AcuKG", i.e. a candidate pool, not a set a practitioner would prescribe together in one session. Large sets (e.g. 63 points for "Headache") are a symptom of pooling, not a 63-point prescription. See the datasheet caveat below.

Why the labels are not shipped here (BUILD-SCRIPT pattern)

AcuBench's labels are derived from AcuKG Indication.csv, and AcuKG has no license file (confirmed via GitHub API: "license": null). Under default copyright we do not have redistribution rights to AcuKG's raw files or to a substantially complete derived copy of them.

So this repo does not contain the assembled labels (acubench.jsonl) or the raw AcuKG clone. Instead it ships a deterministic build script that regenerates the exact same labels + splits locally on your machine from a copy of AcuKG that you clone yourself. What ships here:

Reproduce the full benchmark

# 1. Clone AcuKG yourself (under its terms, not ours):
git clone https://github.com/<acukg-owner>/AcuKG.git /path/to/acukg

# 2. Regenerate the exact 446-row benchmark + splits locally:
python3 build_acubench.py --acukg /path/to/acukg --out ./build

This writes build/acubench.jsonl (446 rows) and build/splits/{train,val,calib,test}_ids.txt. The script is byte-for-byte equivalent to the reference research pipeline (seed=42); it prints per-file SHA-256 checksums and asserts the 446 / 266-70-65-45 counts so you can confirm you regenerated the canonical dataset.

Each acubench.jsonl row:

{"id": 0, "indication": "Abdomen Skin Itching", "acupoints": ["CV15"],
 "meridians_present": ["CV"], "n_points": 1, "split": "train"}

Splits

60/15/15/10 train/val/calib/test, stratified by target-set-size bin ({1, 2, 3-5, 6-8, 9-12, 13-20, 21-40, 41+}) with deterministic largest-remainder allocation (seed=42). Realized sizes (446 total):

Target-set-size distribution is heavily right-skewed: min 1, median 2, mean ~5.3, max 63; 88% of indications have ≤12 points. 360 of the 361 points appear as a label (ST17 never does).

Metric suite (eval.py)

Self-contained (numpy + scikit-learn only). Scores a predictions JSONL against a gold JSONL over the 361-point space:

• —Set metrics: jaccard_mean, f1_micro, f1_macro (example-based)
• —Ranking metrics (need per-point scores): precision_at_k, recall_at_k, ndcg_at_k for k∈{5,10,20}, prauc_mean
• —`invalid_combination_rate`: a structural meridian-scatter PROXY for prescription plausibility (analogous in spirit to a DDI-rate), not a clinical-safety number.

Prediction format (JSONL, one object per line):

{"id": 0, "acupoints": ["CV15"], "scores": {"CV15": 0.9, "LU1": 0.1}}

• —acupoints drives the set metrics + validity proxy.
• —scores (optional but recommended) drives the ranking metrics; without it, ranking falls back to alphabetical order of the predicted set.

# score the test split of your locally-built gold:
python3 eval.py --pred preds.jsonl --gold build/acubench.jsonl \
    --split test --who who_acupoints.csv
# score against the shipped 20-row sample (keyed by symptom string):
python3 eval.py --pred preds.jsonl --gold sample_labels.jsonl \
    --gold-key symptoms --who who_acupoints.csv

Reference baselines (from the AcuBench paper, TEST n=45)

Absolute numbers are modest by design (the task has ~50% single-point targets over 361 classes). Learned baselines beat popularity by ~2.5-5x on most metrics.

Limitations (honest)

• —Single label source: all labels derive from one structured source (AcuKG Indication.csv); no independent source cross-validates it.
• —Small scale: 446 samples / 361 classes — underpowered for strong coverage guarantees. Conformal results in the paper are a small-scale pilot (strict full-containment conformal is data-sparsity-bound at this scale; it degenerates to ~100% abstention at a 30-point cap).
• —Indication strings are not deduplicated / normalized (near-synonyms are distinct rows).
• —Random split only (stratified by set size); no temporal/population split, so it tests interpolation within AcuKG's vocabulary, not novel indications.
• —`invalid_combination_rate` is a structural proxy, not clinical validity.

Citation

@misc{acubench2026,
  title  = {AcuBench: A Benchmark for Indication-based Acupoint-Set Recommendation},
  author = {You, Taewan},
  year   = {2026},
  note   = {Labels derived from AcuKG via a local build script; see NOTICE.}
}

Please also cite AcuKG (the upstream source of the derived labels) per its authors' request.