t2ance/mat-01-lr-versus-learning-signal

01 Learning rate versus the learning signal

When the round-best score on a CPU-only Kaggle task stops rising during GRPO training, is the binding constraint the learning rate (too small to move the policy, or too large to keep it stable), or the learning signal itself (what the search samples and how the reward separates it)? This repository is the data root of that question: every training run's tree-search rollout archive it produced between 2026-06-29 and 2026-07-03, minus the exclusion list in section 5.

1. Question and links

• —Report source: personal/experiments/rl-training/01-lr-versus-learning-signal/report/main.tex in t2ance/ml-agent-training-personal; its launcher configs (edited in place per launch, so each keeps only the last boot's settings) are personal/experiments/rl-training/01-lr-versus-learning-signal/tree_grpo_27b_{pizza,nomad,spooky}/configs/run_tree_grpo_27b_*.sh in the same repository.
• —Issue: none. No card exists for this window either (the card mechanism began 2026-07-18, after every run below) and the report names no issue number.
• —Answer, as the report states it: at every stable learning rate tried on any of the three tasks (random-acts-of-pizza, spooky-author-identification, nomad2018), the round-best score did not move whether the policy stayed still (actor/kl_loss 0.0007-0.0008 at 3e-5) or moved far (actor/kl_loss up to 0.41 at 1e-4 over ten steps, roughly 500-600 times the 3e-5 level, with rollout_corr/ppl_ratio never above 1.150); unguarded 1e-4 (no KL leash, no entropy floor) diverged instead, ppl_ratio reaching 2258.06 by step 14. The report calls this evidence for "the learning signal binds" over "the learning rate binds", but states plainly that the question stays open: no run pushed a guarded rate past 1e-4, no same-task pair isolates the advantage estimator from the learning rate, and the causal reading behind the contemporaneous "strategy monoculture on a flat reward" diagnosis rests on a per-node analysis whose report file was never located.
• —The report's own numbers for how tightly the round-best score stayed put are inconsistent between its per-run tables and its rounded conclusion-section prose, and this repository's own summary.json files side with the per-run tables. For nomad: the per-run tables (main.tex lines 417-423 and 532-542) give nomad1e4 step 3 as 0.0654 (confirmed in `20260630204814/round3/nomad2018-predict-transparent-conductorssummary1782859083.json's besttestmetric 0.06538962579529436) and nomad_1e4_0703 step 7 as 0.0670 (confirmed in 20260703154220/round7/'s besttestmetric 0.06704117456406464), so the true span across the three runs' own tables is 0.0654 to 0.0670; the report's conclusion instead rounds the same three runs to "all inside 0.0655 to 0.0670" (main.tex lines 628-629), a band its own per-run tables already sit outside. For spooky: the per-run table (main.tex line 477) gives spooky_1e4's own range as 0.347 to 0.384 (confirmed in 20260701052951/round1/'s besttestmetric 0.3474613421126908 and round_2's 0.3842695564177869), while the report's conclusion rounds the two hinted runs to "stayed inside 0.353 to 0.384" (main.tex` lines 489-490), a band that same table entry already sits outside. Quote either figure with this discrepancy noted; do not repeat the narrower conclusion-section band as if it were the report's only claim.
• —W&B project: pqin/tree-grpo. Runs (report's short names in parentheses): tree_grpo_27b_pizza (pizza), tree_grpo_27b_pizza_fix (pizza_fix), tree_grpo_27b_spooky (spooky), tree_grpo_27b_nomad_3e5 (nomad_3e5), tree_grpo_27b_nomad_1e4 (nomad_1e4), tree_grpo_27b_spooky_ranked (spooky_ranked), tree_grpo_27b_spooky_1e4 (spooky_1e4), tree_grpo_27b_nomad_1e4_0703 (nomad_1e4_0703). Three run directories in this root have no W&B run at all (section 2): the nomad attempt at an effective 1e-5 (20260630_005254, its W&B id likely lost in a W&B incident the memory notes date to 2026-06-30), an unmatched nomad attempt the report does not use (20260630_153215), and the 9B eval-phase wiring check (run_1783012800).
• —Status: finished. Every run below is stopped, diverged, or collapsed; none is live. The runs span 2026-06-29 04:00 UTC (pizza's first boot) to 2026-07-03 (nomad_1e4_0703's last recorded round), the window the report's abstract states. The report's own header marks the research question itself, not the data collection, as still "Status: Open": the runs settle several individual comparisons (section 1's answer above) but not the user's 2026-07-03 question of whether a rate above 1e-4 or the pre-rank estimator would have behaved differently.

2. Directory tree

Every run directory is named by its run id, one line each below; every top-level file follows. Inside each run directory, one round_N/ holds one training step's tree-search rollout: <task>__<epoch>.jsonl (one row per tree node, the model's full prompt, completion, tokens, logprobs, executed code, and grade), <task>__<epoch>__transcript.md (a human-readable rendering of the same round, with the tree and the valid nodes' scores), <task>__<epoch>__tree.txt (a compact ASCII rendering of the same tree, no prose), and <task>__summary_<epoch>.json (the round's own counts and best scores, section 4). A run's last round directory shows one of three disk states, and only timing.jsonl (below), never the mere presence of a summary.json, says whether a round became a trained step: (1) a 0-byte jsonl with no transcript, tree, or summary at all, when the process died before any node finished, for example 20260630_005254/round_6; (2) a partial jsonl (a handful of the round's 32 rows) with a fully rendered transcript.md and tree.txt for that partial node set but no summary.json, when the process died after rendering some nodes but before the round's own summary was written, for example 20260701_052951/round_5; (3) a round that is fully complete on disk, jsonl, transcript, tree, and a real summary.json all present, that timing.jsonl still never logs as a trained step, because the rollout finished and was summarized but the process was stopped before that step's own timing line was written, for example 20260630_181003/round_5 and 20260630_204814/round_6 (both below). None of these three states counts as one of the report's counted steps. One root-level timing.jsonl per run holds one line per completed step's wall-clock phase breakdown (section 4); its line count, not the round directory count, is the number of trained steps the report's tables use. A run's own round number inside summary.json restarts at 1 with every process boot rather than continuing the previous boot's count (verified on spooky's two boots below), so match a round to the report's tables by its best_test_metric/valid-node count and its position in timing.jsonl, not by directory round number alone.

README.md            this front page
20260629_033731/      pizza boot 1 (leaf tree_grpo_27b_pizza): rounds 1-10, random-acts-of-pizza, launched 06-29 04:00 UTC; config asked for 3e-5 but actor/lr logged 1e-6 because verl restored the scheduler from the 1e-6-era checkpoint on resume; W&B tree_grpo_27b_pizza; report section "The two ends of the ladder on pizza", table row pizza
20260629_095651/      pizza boot 2: rounds 11-16 complete, round_17 incomplete (jsonl only, the boot died mid-round); same W&B id tree_grpo_27b_pizza and report row
20260629_141502/      pizza boot 3: round_17 redone and complete, round_18 incomplete (the run's final manual stop); same W&B id and report row; the console log surviving for this run, analysis/logs/tree_grpo_27b_pizza.log, is outside this root
20260629_153429/      pizza_fix (leaf tree_grpo_27b_pizza): rounds 1-15 on disk, fresh start at 1e-4 with no KL leash and no entropy floor, grpo_valid_only z-score, crash advantage -1.0; diverged (ppl_ratio to 2258.06 by step 14 in the 14-row W&B history); W&B tree_grpo_27b_pizza_fix, report row pizza_fix; cited as an input by rl-training/04-noise-floor-and-snr-fixes
20260630_005254/      nomad at an effective 1e-5 (leaf tree_grpo_27b_nomad): rounds 1-5 complete, round_6 incomplete; no config.json or W&B export survives for this attempt; report section "The nomad attempt at 1e-5 (rollout archive only)"
20260630_034901/      spooky boot 1 (leaf tree_grpo_27b_spooky): round_1 complete (32 nodes, 12 valid), round_2 incomplete (17 of 32 nodes before the boot died); 3e-5, grpo_valid_only z-score, no hint; W&B tree_grpo_27b_spooky, report row spooky
20260630_072028/      spooky boot 2: round_1 and round_2 complete, round_3 complete with 0 of 32 valid nodes (the run's "collapsed" step), round_4 incomplete; same W&B id tree_grpo_27b_spooky and report row
20260630_153215/      nomad, unused: round_1 complete (32 nodes), round_2 incomplete (8 of 32 nodes); the report states this archive "matches no run or note and is not used"; leaf tree_grpo_27b_nomad, no W&B export
20260630_181003/      nomad_3e5 (leaf tree_grpo_27b_nomad): rounds 1-4 trained (timing.jsonl logs steps 1-4); round_5 is also fully complete on disk (32 nodes, 21 valid, best_test_metric 0.06602093601171502) but timing.jsonl never logs a step 5, so its rollout was generated and summarized but never trained; guarded, grpo_valid_only z-score, no hint; W&B tree_grpo_27b_nomad_3e5, report row nomad_3e5
20260630_204814/      nomad_1e4 (leaf tree_grpo_27b_nomad): rounds 1-5 trained (timing.jsonl logs steps 1-5); round_6 is also fully complete on disk (32 nodes, 12 valid, best_test_metric 0.0659388893779089) but timing.jsonl never logs a step 6, so its rollout was generated and summarized but never trained; same estimator and guards as nomad_3e5 at 1e-4; W&B tree_grpo_27b_nomad_1e4, report row nomad_1e4
20260701_003841/      spooky_ranked (leaf tree_grpo_27b_spooky): rounds 1-11 complete, round_12 incomplete; grpo_valid_only rank estimator, the gradient-boosting-banning hint, and the prompt fixes all entered together with this run; W&B tree_grpo_27b_spooky_ranked, report row spooky_ranked; cited as an input by rl-training/02-9b-capability-ceiling and rl-training/04-noise-floor-and-snr-fixes
20260701_052951/      spooky_1e4 (leaf tree_grpo_27b_spooky): rounds 1-4 complete, round_5 incomplete; same estimator, hint and prompt fixes as spooky_ranked, at 1e-4; W&B tree_grpo_27b_spooky_1e4, report row spooky_1e4; cited as an input by rl-training/02-9b-capability-ceiling
20260703_154220/      nomad_1e4_0703 (leaf tree_grpo_27b_nomad): rounds 1-10 complete, round_11 incomplete (configured for 30 steps, stopped by hand at 10); grpo_valid_only rank estimator, no hint, and the first run in this root under the post-external-split prompt (no self-reported score shown, summary.json carries honest_test_metric instead of best_validation_metric); W&B tree_grpo_27b_nomad_1e4_0703, report row nomad_1e4_0703; cited as an input by rl-training/04-noise-floor-and-snr-fixes
run_1783012800/        the external split's eval-phase wiring check named "Block B's eval phase" in the report: one round (round_0), 4 nodes, 0 valid, student switched to Qwen3.5-9B, exec_backend "k8s" (2026-07-02); not one of the three named leaves above and has no W&B export; report calls it "a wiring check only", no honest number produced

This root holds only the harness's own tree-search rollout archive: round_N/ directories and timing.jsonl per run, as listed above. Unlike a verl-trained question published later in this project, nothing else lives here: no checkpoints/ directory, no wandb/ client directory, and no exported W&B history table exist anywhere under this root. Every summary.json's adapter_path field is empty (every run below starts a fresh LoRA on the base model rather than loading a saved one), this project's checkpoint store /data2/peijia/ml-agent-checkpoints/rl-training/ does not exist, and the runs' own W&B history and config exports that the report cites live outside this repository, in personal/experiments/rl-training/01-lr-versus-learning-signal/results/wandb_exports/<id>/ (config.json, history.jsonl, summary.json) in the t2ance/ml-agent-training-personal repository, not in this Hugging Face dataset. So no trained weights and no exported learning-curve table are published here; the round-level facts this repository can support directly are covered in sections 3 and 4.

3. How to read each kind of file

• —Text, JSON and JSONL files at https://huggingface.co/datasets/t2ance/mat-01-lr-versus-learning-signal/raw/main/<path>: a round's summary, for example raw/main/20260629_033731/round_1/random-acts-of-pizza__summary_1782706700.json; a round's transcript, raw/main/20260629_033731/round_1/random-acts-of-pizza__1782705606__transcript.md; a round's tree rendering, raw/main/20260629_033731/round_1/random-acts-of-pizza__1782705606__tree.txt; a run's step timing, raw/main/20260629_033731/timing.jsonl. These are all well under a megabyte.
• —The round's main rollout file, round_N/<task>__<epoch>.jsonl, at resolve/main/<path> instead (raw/ on a file this size returns only the LFS pointer): for example resolve/main/20260629_033731/round_1/random-acts-of-pizza__1782705606.jsonl (7.5 MB; the largest in this root is 8.1 MB, in 20260629_153429/round_9/). It is large because it carries the full prompt_messages, raw_completion, thinking, response_token_ids, and response_logprobs of all 32 nodes.
• —What this repository can support directly: the round statistics the report calls valid_auc_max/critic/score/max (renamed here best_test_metric) and the valid share of a round (valid/nodes in summary.json), and every node's own reward. The reward-map invariant the report's Verification section checks is reproducible from a real record in this root: the step-1 best node of 20260703_154220 (id 142e88244232, matching that round's summary.json.best_node_id) carries test_metric 0.06570251263152488 and reward 0.9364093928262915, and 0.9364093928262915 == exp(-0.06570251263152488), the error-metric reward map (r = exp(-m)) the report's Preliminary section states for nomad and spooky. The AUC map (r = max(0, 2*AUC-1)) checks the same way on 20260629_033731/round_1's best node (id bbfacf745b7e): test_metric 0.7850193012579306, reward 0.5700386025158612 == max(0, 2*0.7850193012579306-1). A failed node's reward is the fixed sentinel -0.5 regardless of task (for example 20260629_033731/round_1's first row, id 9e7b14a8a087, status no_code).
• —What this repository cannot support: the optimizer-side instruments the report's tables also cite (actor/kl_loss, rollout_corr/ppl_ratio, actor/entropy, actor/grad_norm, actor/lr) are verl's own step metrics and are not written into any file in this root; they exist only on the W&B run pages linked in section 1 (and, as a static export outside this repository, under personal/experiments/rl-training/01-lr-versus-learning-signal/results/wandb_exports/<id>/history.jsonl in the GitHub repository named above).
• —A run's local round number restarts at 1 on every process boot (section 2); to align a round directory with a row of the report's per-run tables, match on best_test_metric and the valid/nodes counts, not on the directory's round index.

4. Row fields

round_N/<task>__<epoch>.jsonl, one row per tree-search node (32 per full round, fewer in an incomplete one): id (this node's short hex id); step, round (training step; round restarts per boot, section 2); exp_id (the competition slug); parent_id (the parent node's id, or null for a fresh draft); sibling_group (the id shared by every sibling sampled together, i.e. one GRPO group); mode (draft writes a fresh script, improve edits a valid parent, debug fixes a failed parent); plan, raw_completion, thinking (the model's plan text, full raw completion, and thinking trace); code (the extracted Python script, empty when none was extracted); prompt_messages (the exact chat messages sent to the model); response_token_ids, response_logprobs (the sampled tokens and the rollout engine's own log-probabilities for them, the values the trainer's rollout_corr/ppl_ratio compares against); exc_type, term_tail (the exception class and the tail of stdout/stderr, if the node crashed); exec_time (seconds the script ran); is_buggy, status (whether and how the node failed, e.g. no_code, crashed, ok); grade_error (the grader's error message, if scoring failed); test_metric (the held-out competition metric actually achieved, null for a failed node); oriented (the metric oriented so higher is always better, for cross-task round-best comparisons); reward (this node's GRPO reward: exp(-test_metric) on nomad/spooky, max(0, 2*AUC-1) on pizza, or the fixed sentinel -0.5, verified in section 3); source (how this node's code originated, none when no code exists); n_children (child nodes later expanded from this one). Runs before the 2026-07-01 external-split redesign (every run except run_1783012800 and 20260703_154220) also carry validation_metric (the model's own self-reported validation score, removed by commit c33db14). Runs from the redesign onward instead carry decl_deps_ignored, decl_malformed, decl_miss_reason (the harness's parsing of the script's declared dependencies), exec_timings (a finer breakdown of exec_time), gpu_util (GPU utilization sampled during execution, present even for these CPU tasks), resources, where (the resource limits applied and the backend/location the node executed on).

round_N/<task>__summary_<epoch>.json, one row per round: exp_id, round; model (the base model's absolute path, e.g. /data2/peijia/models/qwen3_5_27b, or a short label such as qwen3_5_9b in run_1783012800); adapter_path (a starting adapter to load; empty in every run in this root); nodes (tree nodes generated this round, 32 unless the round was cut short); valid (nodes that executed and produced a gradeable submission); expansions (parent expansions configured for the round, 4 throughout); best_test_metric (the best held-out metric among valid nodes); best_node_id (the id of that node, joining to the round's jsonl); rollout_file, workspace (the harness's own relative paths for the round's jsonl and execution workspace); gen_llm_seconds, gen_exec_seconds (wall-clock seconds spent generating completions versus executing code). Runs before the external-split redesign also carry best_validation_metric (the best node's self-reported validation score); runs from the redesign onward instead carry exec_backend (local or k8s), phase (train or eval), and honest_test_metric (the harness-owned held-out evaluation score, null when the eval phase produced no valid node, as in run_1783012800).

timing.jsonl (one file per run, one row per completed training step): step; total_s (total wall-clock seconds for the step); phases_s, a nested object with update_weights (loading the trained weights into the rollout engine), gen (generation), old_log_prob (log-probs under the pre-update policy), ref (reference-policy log-probs for the KL leash), adv (advantage computation), update_actor (the optimizer step), and, only on a step that saved a checkpoint, save_checkpoint (for example the last row of 20260703_154220/timing.jsonl); no checkpoint files from any such save survive in this root (section 2).

round_N/<task>__<epoch>__transcript.md and round_N/<task>__<epoch>__tree.txt are not row files: the transcript is a human-readable rendering of the same round (a one-line header with the node and valid counts and the valid nodes' scores, then an indented parent-to-child rendering of the tree with each node's mode, outcome, and sibling group), and the tree file is a shorter ASCII-only rendering of the same structure with each node's mode, ok/BUGGY status, and test value.

5. Exclusions and inputs

The rule's exclusion list applies to nothing that actually exists in this root. Optimizer state, RNG state, other extra training-step state, full-model shards of a LoRA step, a critic or value model, a resume pointer, and an intermediate full-model dump: none of these files exist anywhere under this root, because verl's own checkpoint tree for these runs was never copied here. Every adapter_path in every summary.json is empty (each run starts a fresh LoRA rank 32, alpha 32, all linear modules, merged into the weights after every step, per the report's "Held constant" section, rather than loading a saved one), and this project's own checkpoint store for the line, /data2/peijia/ml-agent-checkpoints/rl-training/, does not exist. No critic exists in any case (GRPO). So no trained weights from this window are published here, not because they were excluded, but because none survive outside the training node's own local disk.

Inputs from other sources (not copied):

• —The base model: the runs' own model field records /data2/peijia/models/qwen3_5_27b in every summary.json except run_1783012800 (which switched the student to qwen3_5_9b); this is the public Qwen3.5-27B (or 9B) weights. The report's own title and background section name the 27B model "Qwen3.6-27B"; this repository quotes both spellings as each source states them.
• —The starting adapter: none. Every run in this root trains a fresh LoRA directly on the base model (section 4's adapter_path is empty throughout); no other question's SFT or RL output is loaded here.
• —The tasks: three Kaggle competitions run as MLE-bench-style harness tasks, random-acts-of-pizza, spooky-author-identification, and nomad2018-predict-transparent-conductors, as the harness's packed task caches under the shared, read-only datasets/mlebench root (MLEBENCH_DATA_ROOT, /data3/ruiyi/mlebench-playground/data) and each competition's held-out labels; this is benchmark data and stays out of this repository.
• —No run of any other question is cited as an input to this one. Four runs of this root are themselves cited as inputs by later questions (rl-training/02-9b-capability-ceiling and rl-training/04-noise-floor-and-snr-fixes); section 2 names each one where it appears in the tree.