suchirsalhan/beetle-eval-results

BEETLE evaluation results

Every evaluation number behind the BEETLE curriculum-learning models, on one schema. Produced by `beetle-analyze`; each row traces to a completed job, and a model that could not be evaluated gets a row with status != "ok" and the error rather than an interpolated value.

How it is organised

from datasets import load_dataset

meco  = load_dataset("suchirsalhan/beetle-eval-results", "view_meco", split="train")   # the MECO table
bliss = load_dataset("suchirsalhan/beetle-eval-results", "view_bliss", split="train")  # the BLiSS table
long  = load_dataset("suchirsalhan/beetle-eval-results", "meco", split="final")        # every MECO row, long
words = load_dataset("suchirsalhan/beetle-eval-results", "meco_items", split="train")  # per-word surprisals

MECO: the value is delta log L, and the measure is always named

view_meco is the headline table -- one row per (model, reader cohort), one column per eye-tracking measure, and every value is delta log L, not surprisal. Word-level surprisal is an intermediate: it lives in meco_items, one row per word, and is never reported as a result.

In the long form (meco, split final), metric is delta_logl and subset names the reading-time measure the fit was on. loglik_full and loglik_baseline are carried alongside so the subtraction can be checked, and subset="coverage" marks the scoring-diagnostic row rather than a result.

There is no refix.dur. The release has refix and firstrun.refix, which are refixation counts, not durations.

BLiSS: all six metrics travel together

view_bliss gives one row per (model, L1 cohort) with rp_at_0, rp_at_tau, ngs, cps, lp and so as columns. RP@0 and RP@tau answer different questions, and CPS is the sanity check that says whether the rest mean anything for that model, so reporting one without the others is misleading. matched marks the cohort whose L1 is the model's own -- the cell the paper's claim is about.

Splits inside one config share a schema, which is why Tier 2 and each view are their own config rather than another split -- load_dataset will not mix them.

Searchable by construction

Registry facets are denormalised onto every row of every config, so one filter works everywhere -- no joins:

meco.filter(lambda r: r["scale_tokens"] == "24B"
                      and r["kind"] == "bilingual"
                      and r["curriculum"] in ("B2", "B3"))

Tiers

• —Tier 1 (results/, and the per-benchmark configs) -- one row per (model, checkpoint, benchmark, subset, eval_lang, cohort, condition, metric).
• —Tier 2 (items/, the items split) -- per-item and per-participant scores, keyed by run_id. item_id lives here and never in metric.
• —Tier 3 (views/) -- generated wide tables. Never hand-edited.

Conventions

• —Results are append-and-supersede: a rerun writes a new run_id; readers resolve to the newest run_id per key. Nothing is overwritten.
• —checkpoint = "final" means the model's released head. For repos whose main branch carries no weights, "final" resolves to the highest step-* branch, and the resolved commit is in revision.
• —MECO word surprisal is the sum of subword surprisals; delta log L is logLik(full) - logLik(baseline) from lmer(..., REML = FALSE) with uncorrelated by-subject random slopes. Comparable only within a reader cohort -- never average across cohorts.
• —Minimal-pair accuracy is the fraction of pairs where the grammatical sentence has the higher summed sentence log-probability.
• —Scored in float32 (the evaluation host has no bfloat16); the dtype is in each row's condition.

Schema version 2.0.0. Licence: CC-BY-4.0.