lumetix-ai/itorgov-sn97-albedo-eval-traces-v6

Albedo Eval Traces — Turns

One row per scored challenger turn in the Albedo subnet's published eval traces. Scoring is BINARY yes/no-question scoring: an evaluator writes a flat set of yes/no questions per task, and each judge answers them with 1/0 for the king and the challenger independently. A judge's yes_rate is the mean of its 1/0 answers; a side's score is the mean of its per-judge yes-rates. Every per-judge, per-side record that scored a turn is folded into the row's judges list.

Only evals from the validator-computed response-measurements rollout are included. The inclusive production cutoff is 2026-07-17T08:20:09.716775+00:00 (first verified eval 2cb6b088-257b-41d7-9033-74933454e78b). Every retained turn also has judge-explanation evidence that the authoritative measurements were received.

Columns

• —date, eval_id, challenger_hotkey, king_hotkey — eval identity.
• —global_idx, turn_idx, instance_id — the turn's coordinate. global_idx is the eval-wide turn counter (the sample's position within its eval); turn_idx is the index within an instance. Both are constant across the turn's judges.
• —prompt — the exact rendered generation prompt; the question evaluator receives its rstripped contents inside the production TASK wrapper. prompt_messages is its convenient parsed role/content view.
• —king_reply, chal_reply — the reigning king's and the challenger's responses.
• —questions — the exact ordered yes/no checklist used for this turn. Each entry preserves id, category, text, and example_bad.
• —king_score, challenger_score — the turn's authoritative absolute yes-rates in [0, 1] (each the mean yes-rate across that side's judges). They do NOT sum to 1. The eval verdict averages each side across valid turns and crowns the challenger when the gap between those eval-level means is at least 0.02. Only fully scored rows are included.
• —judge_models — the exact judge-model set expected for the turn, retained from the run's dashboard score provenance.
• —judges — list of per-judge, per-side scorings, each with side (previous_king or challenger), judge_model, provider, yes_rate (that judge's mean 1/0 answer for that side), parse_ok, error, and the per-question answers and explanations (each a list of {key, value} entries keyed by the evaluator's per-sample question ids q_01, q_02, ...; answers values are "1"/"0"). Every included judge model has exactly one king result and one challenger result.

Source: every eval run's canonical generated-samples.jsonl and scoring-results.jsonl artifacts, published under the validator's per-run S3 prefix and discovered via the public dashboard.json.

Generated by scripts/training/01_upload_eval_traces.py — re-running the script first removes legacy rows from the existing dataset and then merges newly mirrored traces, with local rows overriding remote ones on the primary key (eval_id, global_idx).