t2ance/mat-01-luna-teacher-search

01 Luna teacher search

Does a much larger black-box model, reached through a local proxy with no code change to the campaign's own tree-search harness, sample solution families other than the Qwen3.6-27B student, and how strong are its scores against the tasks' medal lines and against the student's recorded archives, and is its trajectory archive worth distilling into the student? This repository is the data root of that question: everything its ten standalone harness.run trees wrote, minus the exclusion list in section 5.

1. Question and links

• —Question and card: personal/experiments/distillation/01-luna-teacher-search/ in t2ance/ml-agent-training-personal. All ten runs below belong to the question's one leaf, luna_api_search_5tasks/card.md (its rollouts: header names only the pattern of the five tasks launched on 2026-07-18; the three tasks added on 2026-07-19 and their reruns are recorded in the card's append-only check log instead); luna_api_search_5tasks/configs/run_search_<task>.sh are the eight launcher scripts.
• —Report: personal/experiments/distillation/01-luna-teacher-search/report/main.tex in t2ance/ml-agent-training-personal, "01. How strong is the black-box teacher on our tasks, and is it a distillation target?" Quoting its abstract, recomputed from the rollout archives: "the best node reaches gold on aerial-cactus (AUC 1.0) and denoising-dirty-documents (RMSE 0.01371), silver on nomad2018 (RMSLE 0.06219) and text-normalization English (accuracy 0.99271), bronze on random-acts-of-pizza (AUC 0.6994) and text-normalization Russian (0.97892), and only the median tier on dogs-vs-cats (log loss 0.0918) and spooky-author-identification (0.3641)"; on the one same-day, same-prompt comparison with the student (spooky), the teacher samples the same solution family (100% tf-idf, 100% linear, 0/96 out-of-family) and scores worse on every statistic recorded for both, so the diversity question is answered negative for that gate and "whether the archive is a distillation target was not decided here" (report's Conclusion); the readout fed the three-step roadmap (HumanRank alone, then a teacher-SFT cold start, then the teacher as an in-RL scorer) carried forward by report 02 and rl-training 09.
• —Code: no branch or PR of its own; the report's Implementation subsection names the product files read (ml-agent-training/harness/run.py, search.py, llm.py, reward.py, tasks.py, executor/k8s.py) and states the work of 2026-07-14 to 07-21, including the results-cap fix of report Sec. 4.3, reached Git only in the squashed commits 4c0ed5c, 899d203, eb74aa0 of 2026-08-10.
• —Issue: none is named in the card or the report; the report's own header lists only "Related: 02; rl-training 05, 08, 09", each a report path, not a tracked issue.
• —W&B: none. The report states directly that "No W&B run exists for any search here (standalone harness.run; no luna_search row in personal/docs/wandb_index.csv)"; personal/docs/wandb_index.csv confirms it (its one luna-matching row, sft_27b_luna, belongs to a different question, distillation/02-luna-sft-before-rl, not this one); none of the ten run directories below holds a wandb/ subdirectory.
• —Status (2026-09-05): finished. All ten runs ran between 2026-07-18 22:53:20Z (smoke_luna_spooky start) and 2026-07-19 04:15:11Z (the last summary file, luna_tn_en_0719b); none is a training run (reward mode task_score, no verl process, no weight update) and none is live. No other question's roots.json entry lists a run of this root as an input (this root has no consumers).

2. Directory tree

One subdirectory per run, named by its run id (the same string as the card's evidence paths and the report's Evidence blocks). Each run is one tree over one MLE-bench task, driven by the stock harness.run entrypoint with only model.name, model.base_url and the API key swapped for the proxied teacher gpt-5.6-luna in place of the locally served student; 96 nodes per full tree (12 drafts, the rest improve or debug), 2 for the smoke. Every run directory has the same shape: round_0/<exp_id>__<ts>.jsonl (the raw node records, section 4), round_0/<exp_id>__<ts>__transcript.md (a human-readable per-node summary written alongside), round_0/<exp_id>__<ts>__tree.txt (an ASCII parent-child dump of the same tree), round_0/<exp_id>__summary_<ts>.json (the run-level counters, section 4), and two top-level images, viz_dashboard_<exp_id>.png (the harness's six-panel tree-GRPO diagnostics figure) and viz_tree_<exp_id>.png (the rendered search tree); no run has a round_1 (each is one standalone round). None of these files is cited by any other question's report.

README.md                                    this front page (FILES.txt is added by personal/tools/publish_experiment_repo.py before each upload; not on disk yet)
luna_search_cactus_20260718/                 aerial-cactus-identification, 2026-07-18 23:02:54Z-23:48:05Z, GPU pods (pq-mle-exec-gpu:20260703, declare.enabled); report Sec.4.2 row "cactus": 81/12/3 ok/crash/timeout, best AUC 1.0 (gold), 97% neural-network code by keyword count
luna_search_denoising_20260719/              denoising-dirty-documents, first attempt, 2026-07-19 00:57:40Z-01:26:59Z; 0/96 valid, every no-submission node's term_tail carries PQ_EXEC_RC=0 plus PQ_RESULTS_TOOBIG (report Sec.4.3): the k8s executor's 4 MiB results cap dropped every submission; superseded by the fixedcap rerun below, kept as the failure's own evidence
luna_search_denoising_20260719_fixedcap/     denoising-dirty-documents, rerun after the 4 MiB -> 64 MiB results-cap fix (ml-agent-training/harness/executor/k8s.py); report Sec.4.2 row "denoising": 70/19/7 ok/crash/timeout, best RMSE 0.013712 (gold)
luna_search_dogscats_20260718/               dogs-vs-cats-redux-kernels-edition, 2026-07-18 23:02:54Z-2026-07-19 00:59:35Z, GPU pods; report Sec.4.2 row "dogscats": 39/10/47 ok/crash/timeout (47 timeouts at the 600s cap, execution busy 52871s, more than twice any other tree), best log loss 0.09182 (median)
luna_search_nomad_20260718/                  nomad2018-predict-transparent-conductors, 2026-07-18 23:02:54Z-23:50:54Z, CPU pods, 300s timeout; report Sec.4.2 row "nomad": 81/4/11, best RMSLE 0.06219 (silver)
luna_search_pizza_20260718/                  random-acts-of-pizza, 2026-07-18 23:02:54Z-23:38:35Z, CPU pods, 300s timeout; report Sec.4.2 row "pizza": 95/1/0, best AUC 0.6994 (bronze)
luna_search_spooky_20260718/                 spooky-author-identification, 2026-07-18 23:02:54Z-23:27:34Z, CPU pods, 300s timeout; report Sec.4.2 row "spooky" and the Sec.4.4 same-day student comparison: 90/6/0, best log loss 0.3641 (median), 100% tf-idf + 100% linear code (0/96 out-of-family)
luna_tn_en_0719b/                             text-normalization-challenge-english-language, third attempt for this task (the first attempt and a "_fixedcap" retry both failed before any code ran, an undocumented pod-label-length rejection; neither left a rollout directory under this root), 2026-07-19 03:13:11Z-04:15:11Z; report Sec.4.2 row "textnorm-en": 71/25/0, best accuracy 0.99271 (silver)
luna_tn_ru_0719b/                             text-normalization-challenge-russian-language, third attempt, same fix, 2026-07-19 03:13:12Z-04:01:48Z; report Sec.4.2 row "textnorm-ru": 86/10/0, best accuracy 0.97892 (bronze)
smoke_luna_spooky/                            spooky-author-identification, wiring-only smoke (report Sec.4.1), 2026-07-18 22:53:20Z-23:01:28Z, 2 draft nodes, 2/2 valid (log loss 0.4056, 0.4512); proves the proxy and the k8s executor only, no diversity or strength information

Two-node probe trees the report's Sec.4.3 and Sec.4.5 tables also quote (probe_bigresults_denoising, probe_bigresults_textnorm_en, smoke_hr_denoising, smoke_hr_textnorm) are not under this root: their run ids belong to rl-training/09-multitask-humanrank's own root, even though their summaries record model: gpt-5.6-luna; a reader who wants them opens that question's repository, t2ance/mat-09-multitask-humanrank.

3. How to read each kind of file

• —Small text, JSON and log files (a run's tree.txt and summary_<ts>.json): https://huggingface.co/datasets/t2ance/mat-01-luna-teacher-search/raw/main/<path>, for example raw/main/smoke_luna_spooky/round_0/spooky-author-identification__summary_1784415688.json or raw/main/luna_search_spooky_20260718/round_0/spooky-author-identification__1784415774__tree.txt.
• —Every node-record .jsonl and its paired _transcript.md (each several MB) and the two viz_*.png images per run: resolve/main/<path> (raw/ returns only the LFS pointer), for example resolve/main/luna_search_spooky_20260718/round_0/spooky-author-identification__1784415774.jsonl, resolve/main/luna_search_spooky_20260718/round_0/spooky-author-identification__1784415774__transcript.md, or resolve/main/luna_search_spooky_20260718/viz_dashboard_spooky-author-identification.png.
• —The metrics a row carries: test_metric (the task's own raw metric), oriented (its GRPO sign, always improved-is-larger), reward (derived from oriented), and medal (none, median, bronze, silver or gold, computed against the thresholds in ml-agent-training/harness/tasks.py, independent of reward mode); a summary's valid/nodes is the tree's valid rate and best_test_metric/best_node_id name its best node.
• —The status invariant the report's Sec.4.3 observation rests on: every no_submission node's term_tail in luna_search_denoising_20260719 ends PQ_EXEC_RC=0 and a "Wrote submission" line, followed by PQ_RESULTS_TOOBIG size=<bytes> (27.6 MB on this run) — the executor's real 4 MiB results cap dropping a real, successfully written submission, not a code or task failure; the six possible status values across these runs, from the dashboard's legend, are ok, crashed, timeout, no_submission, invalid_submission, no_code.
• —The prompt-identity invariant the report's Sec.4.4 checked by byte comparison: luna_search_spooky_20260718's node records all carry a prompt_messages of two entries, role: system at 2635 characters and role: user at 8221 characters, byte-identical to the concurrent 27B run's prompt (report 08).
• —No token log-probabilities were requested from the teacher (return_logprobs defaults to False in ml-agent-training/harness/llm.py): every row's response_token_ids and response_logprobs is an empty list; only the text (code, plan, thinking, raw_completion) is archived.
• —The family-mix percentages the report quotes (spooky, pizza, nomad, cactus) came from a session script, candidate_quality, that was not retained; it cannot be re-run from anything in this repository, and the report's own keyword-count check (also not scripted here) is the closest a reader can get without rewriting the classifier.

4. Row fields

round_0/<exp_id>__<ts>.jsonl, one row per tree node: id, step, round, exp_id, parent_id, sibling_group; mode (draft, improve or debug); plan, code, raw_completion, thinking (the teacher's free text); prompt_messages (a list of {role, content}, system then user); response_token_ids, response_logprobs (both always empty here, see section 3); exc_type, exec_time, term_tail (the pod's captured stdout/stderr tail), where (the k8s pod name), exec_timings (pending, stage, exec, retrieve seconds); resources (the declared {gpu, cpu, mem, ephemeral} on GPU-declared nodes, null on CPU-only tasks), decl_malformed, decl_deps_ignored, decl_miss_reason; gpu_util, gpu_util_peak, gpu_mem_used_gi, gpu_name (null throughout except gpu_name, which the k8s scheduler fills even for these CPU-bound jobs); cpu_util_mean, cpu_util_peak, mem_peak_gi, kube_node, returncode, attempt; parent_P, parent_P_evidence (the parent's learnability prior and whether it was evidence-based, improve/debug nodes only); is_buggy, status (section 3), grade_error, test_metric, oriented, reward, source (grade); n_children, depth, medal, parent_oriented, improve_delta (the child's gain over its parent, improve nodes only), group_best.

round_0/<exp_id>__summary_<ts>.json, one per run: exp_id, round, model (always gpt-5.6-luna), exec_backend (k8s), adapter_path (always empty, no local adapter), nodes, valid, buffer_nodes, expansions, phase (train), best_test_metric, best_node_id, honest_test_metric (always null, no held-out split beyond the search's own valid split), rollout_file, workspace, gen_llm_seconds, gen_exec_seconds.

round_0/<exp_id>__<ts>__tree.txt: one line per node, [<id short>] <mode> <status> test=<test_metric> sib_grp=<sibling_group> | <where> <exec_time>s, indented under its parent by tree depth; a footer line gives nodes=, sibling_groups= and the best valid node.

round_0/<exp_id>__<ts>__transcript.md: a header line (nodes, valid, best oriented metric) and the sorted list of every valid node's oriented metric, followed by one prose section per node (its plan, code and outcome); written for a human reading pass, not the file the report's numbers were recomputed from (that was the .jsonl and the summary).

5. Exclusions and inputs

This question ran no training at all: reward mode task_score, no verl process, no optimizer, no critic, and no checkpoint of any kind was ever written by these ten runs (confirmed by the directory tree above: every run holds only round_0/, a summary and two images, no checkpoints/). None of the rule's five standard exclusions removes anything from this root, because none of the five kinds of file they name — optimizer/RNG/extra training state, full-model shards of a LoRA step, a critic or value model, a resume pointer, an intermediate full-model dump — was ever produced here; every file the ten runs wrote goes up.

Inputs from other experiments (not copied):

• —The teacher: gpt-5.6-luna, a model alias served by a local OpenAI-compatible reverse proxy (CLIProxyAPI, 127.0.0.1:8377); the repository knows it only as a billing-catalog entry in ml-agent-training/training_verl/judge_client.py (input \$1.0, output \$6.0, cached input \$0.1 per million tokens), and neither the vendor nor the underlying model is established by anything here; no weights or adapter exist locally for it.
• —The student comparison on spooky (report Sec.4.4): run 20260718_231413_27b_bond_judgew1_t10 of rl-training/08-bond-blind-judge, published (once that root is published) as t2ance/mat-08-bond-blind-judge at path 20260718_231413_27b_bond_judgew1_t10/; card personal/experiments/rl-training/08-bond-blind-judge/tree_grpo_27b_spooky_bond_judgew1_t12/card.md.
• —The student archives cited only as numbers, not as run directories (report Sec.4.5): the pre-RL archive of 2026-06-30 (report 01, mined into a Claude memory note this workspace's own rules forbid reading or writing) and the audit table personal/experiments/rl-training/09-multitask-humanrank/results/preaudit_multitask_20260719_final.txt; both stay outside this repository as the report cites them, by document path, not by run id.
• —The tasks: eight MLE-bench competitions (aerial-cactus-identification, dogs-vs-cats-redux-kernels-edition, nomad2018-predict-transparent-conductors, random-acts-of-pizza, spooky-author-identification, denoising-dirty-documents, text-normalization-challenge-english-language, text-normalization-challenge-russian-language) as the harness's packed task caches and the MLE-bench private answers under datasets/mlebench (MLEBENCH_DATA_ROOT); benchmark data, stays out of this repository. The competitions' real leaderboard files (MLEBENCH_LB_ROOT, needed to recompute the report's HumanRank figures) are absent from the workspace host today and are not part of this input either; the report's HumanRank numbers are quoted from the audit table above, never recomputed.