CoolFace
Datasetpublic

ggranberry/lemmanaid-afp-reruns

Lemmanaid AFP-pool Reproducibility Reruns Reproducibility study for claude-opus-4-5 on the yalhessi/lemexp-commerical-llm-experiment benchmark, using an AFP demo pool (honest eval — no train/test theory leakage). Companion to ggranberry/lemmanaid-commercial-results, which holds the earlier shot-count + retrieval sweeps under test-LOO. Configs Two configs, one per benchmark domain: Config Source HF config Test rows octonions template_octonions_2026… See the full description on the dataset page: https://huggingface.co/datasets/ggranberry/lemmanaid-afp-reruns.

sourceHugging Facemitupdated 5mo agoView on Hugging Face
0likes684downloads
Dataset Card

Lemmanaid AFP-pool Reproducibility Reruns

Reproducibility study for claude-opus-4-5 on the `yalhessi/lemexp-commerical-llm-experiment` benchmark, using an AFP demo pool (honest eval — no train/test theory leakage).

Companion to `ggranberry/lemmanaid-commercial-results`, which holds the earlier shot-count + retrieval sweeps under test-LOO.

Configs

Two configs, one per benchmark domain:

ConfigSource HF configTest rows
octonionstemplate_octonions_2026, lemma_object_octonions_2026350 each
rankingstemplate_rankings_swap_distance_2026, lemma_object_rankings_swap_distance_2026139 each

Load with:

python
from datasets import load_dataset
ds = load_dataset("ggranberry/lemmanaid-afp-reruns", "octonions", split="template_t1_r1")

Splits

Each config has 10 splits = 2 tasks × 5 runs.

SplitTaskTemperatureNotes
template_t1_r1 … template_t1_r4template1.04 independent stochastic samples
template_t0template0.0Greedy decode
lemma_object_t1_r1 … lemma_object_t1_r4lemma_object1.04 independent stochastic samples
lemma_object_t0lemma_object0.0Greedy decode

All runs share identical prompts (5-shot nearest-by-symbols retrieval, seed=0, demos drawn from yalhessi/lemexp-task1-v3/{template_afp,lemma_object_afp} train split). Only API sampling differs.

Row schema

json
{"custom_id": "...", "match": bool, "prediction": "...", "gold_output": "...", "lemma_name": "..."}

Raw results

For each split there's also a results_ext_*.jsonl file at repo root (not mounted as a config) containing the unscored predictions plus stop_reason and token usage metadata.

Results

5-shot nearest, AFP-pool, opus-4-5, seed=0:

DatasetnT=1 mean ± std (n=4)T=0pass@4 (T=1)
template_octonions_202635041.7% ± 1.2%42.3%46.0%
lemma_object_octonions_202635018.8% ± 0.4%18.6%19.7%
template_rankings_swap_distance_202613914.4% ± 1.0%14.4%17.3%
lemma_object_rankings_swap_distance_202613922.8% ± 0.7%21.6%25.2%
  • —Sampling variance is small: std across 4 T=1 reruns ≤1.2pp; T=0 falls inside the T=1 band.
  • —pass@4 headroom: gap pass@4 − pass@1 ranges 0.9–4.3pp — recoverable with a verifier/reranker.
  • —Note on pass@k: 4 independent T=1 samples per query; "pass" = any match.

Method

  • —5-shot prompt; demos picked by Jaccard symbol-overlap from AFP train pool (no test theory overlap)
  • —Same seed=0 across all 5 runs → identical demos, identical prompts; only API sampling differs
  • —Anthropic Message Batch API (50% discount, ≤24h SLA)
  • —Exact-match scoring against gold

Pipeline source: https://github.com/... (private)