SLM-Lab/benchmark

1# Phase 5.1 PPO — Operations Tracker2 3Single source of truth for in-flight work. Resume from here.4 5---6 7## Principles8 91. **Two canonical specs**: `ppo_playground` (DM Control) and `ppo_playground_loco` (Loco). Per-env variants only when officially required: `ppo_playground_fingerspin` (gamma=0.95), `ppo_playground_pendulum` (training_epoch=4, action_repeat=4 via code).102. **100M frames hard cap** — no extended runs. If an env doesn't hit target at 100M, fix the spec.113. **Strategic reruns**: only rerun failing/⚠️ envs. Already-✅ envs skip revalidation.124. **Score metric**: use `total_reward_ma` (final moving average of total reward) — measures end-of-training performance and matches mujoco_playground reference scores.135. **Official reference**: check `~/.cache/uv/archive-v0/ON8dY3irQZTYI3Bok0SlC/mujoco_playground/config/dm_control_suite_params.py` for per-env overrides.14 15---16 17## Wave 3 (2026-03-16)18 19**Fixes applied:**20- stderr suppression: MuJoCo C-level warnings (ccd_iterations, nefc overflow, broadphase overflow) silenced in playground.py21- obs fix: _get_obs now passes only "state" key for dict-obs envs (was incorrectly concatenating privileged_state+state)22 23**Envs graduated to ✅ (close enough):**24FishSwim, PointMass, ReacherHard, WalkerStand, WalkerWalk, SpotGetup, SpotJoystickGaitTracking, AlohaHandOver25 26**Failing envs by root cause:**27- Humanoid double-norm (rs10 fix): HumanoidStand (114→700), HumanoidWalk (47→500), HumanoidRun (18→130)28- Dict obs fix (now fixed): Go1Flat/Rough/Getup/Handstand, G1Flat/Rough, T1Flat/Rough29- Unknown: BarkourJoystick (0/35), Op3Joystick (0/20)30- Needs hparam work: H1Inplace (4→10), H1Joystick (16→30), SpotFlat (11→30)31- Manipulation: AlohaPeg (188→300), LeapCubeReorient (74→200)32- Infeasible: PandaRobotiqPushCube, AeroCubeRotateZAxis33 34**Currently running:** (to be populated by ops)35 36---37 38## Currently Running (as of 2026-03-14 ~00:00)39 40**Wave V (p5-ppo17) — Constant LR test (4 runs, just launched)**41 42Testing constant LR (Brax default) in isolation — never tested before. Key hypothesis: LR decay hurts late-converging envs.43 44| Run | Env | Spec | Key Change | Old Best | Target |45|---|---|---|---|---|---|46| p5-ppo17-csup | CartpoleSwingup | constlr | constant LR + minibatch=4096 | 576.1 | 800 |47| p5-ppo17-csupsparse | CartpoleSwingupSparse | constlr | constant LR + minibatch=4096 | 296.3 | 425 |48| p5-ppo17-acrobot | AcrobotSwingup | vnorm_constlr | constant LR + vnorm | 173 | 220 |49| p5-ppo17-fteasy | FingerTurnEasy | vnorm_constlr | constant LR + vnorm | 571 | 950 |50 51**Wave IV-H (p5-ppo16h) — Humanoid with wider policy (3 runs, ~2.5h remaining)**52 53New `ppo_playground_humanoid` variant: 2×256 policy (vs 2×64), constant LR, vnorm=true.54Based on Phase 3 Gymnasium Humanoid-v5 success (2661 MA with 2×256 + constant LR).55 56| Run | Env | Old Best | Target |57|---|---|---|---|58| p5-ppo16h-hstand | HumanoidStand | 18.36 | 700 |59| p5-ppo16h-hwalk | HumanoidWalk | 7.68 | 500 |60| p5-ppo16h-hrun | HumanoidRun | 3.19 | 130 |61 62**Wave VI (p5-ppo18) — Brax 4×32 policy + constant LR + vnorm (3 runs, just launched)**63 64Testing Brax default policy architecture (4 layers × 32 units vs our 2 × 64).65Deeper narrower policy may learn better features for precision tasks.66 67| Run | Env | Old Best | Target |68|---|---|---|---|69| p5-ppo18-fteasy | FingerTurnEasy | 571 | 950 |70| p5-ppo18-fthard | FingerTurnHard | 484 | 950 |71| p5-ppo18-fishswim | FishSwim | 463 | 650 |72 73**Wave IV tail (p5-ppo16) — completed**74 75| Run | Env | strength | Target | New best? |76|---|---|---|---|---|77| p5-ppo16-swimmer6 | SwimmerSwimmer6 | 509.3 | 560 | ✅ New best (final_strength=560.6) |78| p5-ppo16-fishswim | FishSwim | 420.6 | 650 | ❌ Worse than 463 |79 80**Wave IV results (p5-ppo16, vnorm=true rerun with reverted spec — completed):**81 82All ran with vnorm=true. CartpoleSwingup/Sparse worse (vnorm=false is better for them — wrong setting).83Precision envs also scored below old bests. Humanoid still failing with standard 2×64 policy.84 85| Env | p16 strength | Old Best | Target | Verdict |86|---|---|---|---|---|87| CartpoleSwingup | 316.2 | 576.1 (false) | 800 | ❌ wrong vnorm |88| CartpoleSwingupSparse | 288.7 | 296.3 (false) | 425 | ❌ wrong vnorm |89| AcrobotSwingup | 145.4 | 173 (true) | 220 | ❌ worse |90| FingerTurnEasy | 511.1 | 571 (true) | 950 | ❌ worse |91| FingerTurnHard | 368.6 | 484 (true) | 950 | ❌ worse |92| HumanoidStand | 12.72 | 18.36 | 700 | ❌ still failing |93| HumanoidWalk | 7.46 | 7.68 | 500 | ❌ still failing |94| HumanoidRun | 3.19 | 3.19 | 130 | ❌ still failing |95 96**CONCLUSION**: Reverted spec didn't help. No new bests. Consistency was negative for CartpoleSwingup/Sparse (high variance).97Need constant LR test (Wave V) and wider policy for Humanoid (Wave IV-H).98 99**Wave III results (p5-ppo13/p5-ppo15, 5-layer value + no grad clip — completed):**100 101Only CartpoleSwingup improved slightly (623.8 vs 576.1). All others regressed.102FishSwim p5-ppo15: strength=411.6 (vs 463 old best). AcrobotSwingup p5-ppo15: strength=95.4 (vs 173).103 104**CONCLUSION**: 5-layer value + no grad clip is NOT a general improvement. Reverted to 3-layer + clip_grad_val=1.0.105 106**Wave H results (p5-ppo12, ALL completed — NONE improved over old bests):**107Re-running same spec (variance reruns + vnorm) didn't help. Run-to-run variance is high but108old bests represent lucky runs. Hyperparameter tuning has hit diminishing returns.109 110**Wave G/G2 results (normalize_v_targets=false ablation, ALL completed):**111 112| Env | p11 strength | Old Best (true) | Target | Change | Verdict |113|---|---|---|---|---|---|114| **PendulumSwingup** | **533.5** | 276 | 395 | +93% | **✅ NEW PASS** |115| **FingerSpin** | **652.4** | 561 | 600 | +16% | **✅ NEW PASS** |116| **CartpoleBalanceSparse** | **690.4** | 545 | 700 | +27% | **⚠️ 99% of target** |117| **CartpoleSwingup** | **576.1** | 443/506 | 800 | +30% | ⚠️ improved |118| **CartpoleSwingupSparse** | **296.3** | 271 | 425 | +9% | ⚠️ improved |119| PointMass | 854.4 | 863 | 900 | -1% | ⚠️ same |120| FishSwim | 293.9 | 463 | 650 | -36% | ❌ regression |121| FingerTurnEasy | 441.1 | 571 | 950 | -23% | ❌ regression |122| SwimmerSwimmer6 | 386.2 | 485 | 560 | -20% | ❌ regression |123| FingerTurnHard | 335.7 | 484 | 950 | -31% | ❌ regression |124| AcrobotSwingup | 105.1 | 173 | 220 | -39% | ❌ regression |125| HumanoidStand | 12.87 | 18.36 | 500 | -30% | ❌ still failing |126 127**CONCLUSION**: `normalize_v_targets: false` helps 5/12, hurts 6/12, neutral 1/12.128- **false wins**: PendulumSwingup, FingerSpin, CartpoleBalanceSparse, CartpoleSwingup, CartpoleSwingupSparse129- **true wins**: FishSwim, FingerTurnEasy/Hard, SwimmerSwimmer6, AcrobotSwingup, PointMass130- **Decision**: Per-env spec selection. New `ppo_playground_vnorm` variant for precision envs.131 132**Wave F results (multi-unroll=16 + proven hyperparameters):**133 134| Env | p10 strength | p10 final_str | Old best str | Target | Verdict |135|---|---|---|---|---|---|136| CartpoleSwingup | 342 | 443 | 443 | 800 | Same |137| FingerTurnEasy | 529 | 685 | 571 | 950 | Better final, worse strength |138| FingerSpin | 402 | 597 | 561 | 600 | Better final (near target!), worse strength |139| FingerTurnHard | 368 | 559 | 484 | 950 | Better final, worse strength |140| SwimmerSwimmer6 | 251 | 384 | 485 | 560 | Worse |141| CartpoleSwingupSparse | 56 | 158 | 271 | 425 | MUCH worse |142| AcrobotSwingup | 31 | 63 | 173 | 220 | MUCH worse |143 144**CONCLUSION**: Multi-unroll adds no benefit over single-unroll for any env by `strength` metric.145The `final_strength` improvements for Finger tasks are offset by `strength` regressions.146Root cause: stale old_net (480 vs 30 steps between copies) makes policy ratio less accurate.147**Spec reverted to single-unroll (num_unrolls=1)**. Multi-unroll code preserved in ppo.py.148 149**Wave E results (multi-unroll + Brax hyperparameters — ALL worse):**150 151Brax-matched spec (clip_eps=0.3, constant LR, 5-layer value, reward_scale=10, minibatch=30720)152hurt every env except HopperStand (which used wrong spec before). Reverted.153 154**Wave C completed results** (all reward_scale=10, divide by 10 for true score):155 156| Run | Env | strength/10 | final_strength/10 | total_reward_ma/10 | Target | vs Old |157|---|---|---|---|---|---|---|158| p5-ppo7-cartpoleswingup | CartpoleSwingup | 556.6 | 670.5 | 705.3 | 800 | 443→557 ✅ improved |159| p5-ppo7-fingerturneasy | FingerTurnEasy | 511.1 | 693.2 | 687.0 | 950 | 571→511 ❌ **WORSE** |160| p5-ppo7-fingerturnhard | FingerTurnHard | 321.9 | 416.8 | 425.2 | 950 | 484→322 ❌ **WORSE** |161| p5-ppo7-cartpoleswingupsparse2 | CartpoleSwingupSparse | 144.0 | 360.6 | 337.7 | 425 | 271→144 ❌ **WORSE** |162 163**KEY FINDING**: time_horizon=480 helps CartpoleSwingup (+25%) but HURTS FingerTurn (-30 to -50%) and CartpoleSwingupSparse (-47%). Long GAE horizons produce noisy advantage estimates for precision/sparse tasks. The official Brax approach is 16×30-step unrolls (short GAE per unroll), NOT 1×480-step unroll.164 165---166 167## Spec Changes Applied (2026-03-13)168 169### Fix 1: reward_scale=10.0 (matches official mujoco_playground)170- `playground.py`: `PlaygroundVecEnv` now multiplies rewards by `self._reward_scale`171- `__init__.py`: threads `reward_scale` from env spec to wrapper172- `ppo_playground.yaml`: `reward_scale: 10.0` in shared `_env` anchor173 174### Fix 2: Revert minibatch_size 2048→4096 (fixes CartpoleSwingup regression)175- `ppo_playground.yaml`: all DM Control specs (ppo_playground, fingerspin, pendulum) now use minibatch_size=4096176- 15 minibatches × 16 epochs = 240 grad steps (was 30×16=480)177- Restores p5-ppo5 performance for CartpoleSwingup (803 vs 443)178 179### Fix 3: Brax-matched spec (commit 6eb08fe9) — time_horizon=480, clip_eps=0.3, constant LR, 5-layer value net180- Increased time_horizon from 30→480 to match total data per update (983K transitions)181- clip_eps 0.2→0.3, constant LR (min_factor=1.0), 5-layer [256×5] value net182- action std upper bound raised (max=2.0 in policy_util.py)183- **Result**: CartpoleSwingup improved (443→557 strength), but FingerTurn and CartpoleSwingupSparse got WORSE184- **Root cause**: 1×480-step unroll computes GAE over 480 steps (noisy), vs official 16×30-step unrolls (short, accurate GAE)185 186### Fix 4: ppo_playground_short variant (time_horizon=30 + Brax improvements)187- Keeps: reward_scale=10, clip_eps=0.3, constant LR, 5-layer value net, no grad clipping188- Reverts: time_horizon=30, minibatch_size=4096 (15 minibatches, 240 grad steps)189- **Hypothesis**: Short GAE + other Brax improvements = best of both worlds for precision tasks190- Testing on FingerTurnEasy/Hard first (Wave D p5-ppo8-*)191 192### Fix 5: Multi-unroll collection (IMPLEMENTED but NOT USED — code stays, spec reverted)193- Added `num_unrolls` parameter to PPO (ppo.py, actor_critic.py). Code works correctly.194- **Brax-matched spec (Wave E, p5-ppo9)**: clip_eps=0.3, constant LR, 5-layer value, reward_scale=10195  - Result: WORSE on 5/7 tested envs. Only CartpoleSwingup improved (443→506).196  - Root cause: minibatch_size=30720 → 7.5x fewer gradient steps per transition → underfitting197- **Reverted spec + multi-unroll (Wave F, p5-ppo10)**: clip_eps=0.2, LR decay, 3-layer value, minibatch=4096198  - Result: Same or WORSE on all envs by `strength` metric. Same fps as single-unroll.199  - Training compute per env step is identical, but old_net staleness (480 vs 30 steps) hurts.200- **Conclusion**: Multi-unroll adds complexity without benefit. Reverted spec to single-unroll (num_unrolls=1).201  Code preserved in ppo.py (defaults to 1). Spec uses original hyperparameters.202 203---204 205## Completed Runs Needing Intake206 207### Humanoid (ppo_playground_loco, post log_std fix) — intake immediately208 209| Run | HF Folder | strength | target | HF status |210|---|---|---|---|---|211| p5-ppo6-humanoidrun | ppo_playground_loco_humanoidrun_2026_03_12_175917 | 2.78 | 130 | ✅ uploaded |212| p5-ppo6-humanoidwalk | ppo_playground_loco_humanoidwalk_2026_03_12_175817 | 6.82 | 500 | ✅ uploaded |213| p5-ppo6-humanoidstand | ppo_playground_loco_humanoidstand_2026_03_12_175810 | 12.45 | 700 | ❌ **UPLOAD FAILED (412)** — re-upload first |214 215Re-upload HumanoidStand:216```bash217source .env && huggingface-cli upload SLM-Lab/benchmark-dev \218  hf_data/data/benchmark-dev/data/ppo_playground_loco_humanoidstand_2026_03_12_175810 \219  data/ppo_playground_loco_humanoidstand_2026_03_12_175810 --repo-type dataset220```221 222**Conclusion**: loco spec still fails completely for Humanoid — log_std fix insufficient. See spec fixes below.223 224### BENCHMARKS.md correction needed (commit b6ef49d9 used wrong metric)225 226intake-a used `total_reward_ma` instead of `strength`. Fix these 4 entries:227 228| Env | Run | strength (correct) | total_reward_ma (wrong) | target |229|---|---|---|---|---|230| AcrobotSwingup | p5-ppo6-acrobotswingup2 | **172.8** | 253.24 | 220 |231| CartpoleBalanceSparse | p5-ppo6-cartpolebalancesparse2 | **545.1** | 991.81 | 700 |232| CartpoleSwingup | p5-ppo6-cartpoleswingup2 | **unknown — extract from logs** | 641.51 | 800 |233| CartpoleSwingupSparse | p5-ppo6-cartpoleswingupsparse | **270.9** | 331.23 | 425 |234 235Extract correct values: `dstack logs p5-ppo6-NAME --since 6h 2>&1 | grep "trial_metrics" | tail -1` → use `strength:` field.236 237Also check FingerSpin: `dstack logs p5-ppo6-fingerspin2 --since 6h | grep trial_metrics | tail -1` — confirm strength value.238 239**Metric decision needed**: strength penalizes slow learners (CartpoleBalanceSparse strength=545 but final MA=992). Consider switching ALL entries to `final_strength`. But this requires auditing every existing entry — do it as a batch before publishing.240 241---242 243## Queue (launch when slots open, all 100M)244 245| Priority | Env | Spec | Run name | Rationale |246|---|---|---|---|---|247| 1 | PendulumSwingup | ppo_playground_pendulum | p5-ppo6-pendulumswingup | action_repeat=4 + training_epoch=4 (code fix applied) |248| 2 | FingerSpin | ppo_playground_fingerspin | p5-ppo6-fingerspin3 | canonical gamma=0.95 run; fingerspin2 used gamma=0.995 (override silently ignored) |249 250---251 252## Full Env Status253 254### ✅ Complete (13/25)255| Env | strength | target | normalize_v_targets |256|---|---|---|---|257| CartpoleBalance | 968.23 | 950 | true |258| AcrobotSwingupSparse | 42.74 | 15 | true |259| BallInCup | 942.44 | 680 | true |260| CheetahRun | 865.83 | 850 | true |261| ReacherEasy | 955.08 | 950 | true |262| ReacherHard | 946.99 | 950 | true |263| WalkerRun | 637.80 | 560 | true |264| WalkerStand | 970.94 | 1000 | true |265| WalkerWalk | 952 | 960 | true |266| HopperHop | 22.00 | ~2 | true |267| HopperStand | 118.2 | ~70 | true |268| PendulumSwingup | 533.5 | 395 | **false** |269| FingerSpin | 652.4 | 600 | **false** |270 271### ⚠️ Below target (9/25)272| Env | best strength | target | best with | status |273|---|---|---|---|---|274| CartpoleSwingup | 576.1 | 800 | false | Improved +30% from 443 (true) |275| CartpoleBalanceSparse | 545 | 700 | true | Testing false (p5-ppo11) |276| CartpoleSwingupSparse | 296.3 | 425 | false | Improved +9% from 271 (true) |277| AcrobotSwingup | 173 | 220 | true | false=105, regressed |278| FingerTurnEasy | 571 | 950 | true | false=441, regressed |279| FingerTurnHard | 484 | 950 | true | false=336, regressed |280| FishSwim | 463 | 650 | true | Testing false (p5-ppo11) |281| SwimmerSwimmer6 | 509.3 | 560 | true | final_strength=560.6 (at target!) |282| PointMass | 863 | 900 | true | false=854, ~same |283 284### ❌ Fundamental failure — Humanoid (3/25)285| Env | best strength | target | diagnosis |286|---|---|---|---|287| HumanoidRun | 3.19 | 130 | <3% target, NormalTanh distribution needed |288| HumanoidWalk | 7.68 | 500 | <2% target, wider policy (2×256) didn't help |289| HumanoidStand | 18.36 | 700 | <3% target, constant LR + wider policy tested, no improvement |290 291**Humanoid tested and failed**: wider 2×256 policy + constant LR + vnorm (Wave IV-H). MA stayed flat at 8-10 for HumanoidStand over entire training. Root cause is likely NormalTanh distribution (state-dependent std + tanh squashing) — a fundamental architectural difference from Brax.292 293---294 295## Spec Fixes Required296 297### Priority 1: Humanoid loco spec (update ppo_playground_loco)298 299Official uses `num_envs=8192, time_horizon=20 (unroll_length)` for loco. We use `num_envs=2048, time_horizon=64`.300 301**Proposed update to ppo_playground_loco**:302```yaml303ppo_playground_loco:304  agent:305    algorithm:306      gamma: 0.97307      time_horizon: 20      # was 64; official unroll_length=20308      training_epoch: 4309  env:310    num_envs: 8192          # was 2048; official loco num_envs=8192311```312 313**Before launching**: verify VRAM by checking if 8192 envs fits A4500 20GB. Run one Humanoid env, check `dstack logs NAME --since 10m | grep -i "memory\|OOM"` after 5 min.314 315**Rerun only**: HumanoidRun, HumanoidWalk, HumanoidStand (3 runs). HopperStand also uses loco spec — add if VRAM confirmed OK.316 317### Priority 2: CartpoleSwingup regression318 319p5-ppo5 scored 803 ✅; p5-ppo6 scored ~641. The p5-ppo6 change was `minibatch_size: 2048` (30 minibatches) vs p5-ppo5's 4096 (15 minibatches). More gradient steps per iter hurt CartpoleSwingup.320 321**Option A**: Revert `ppo_playground` minibatch_size from 2048→4096 (15 minibatches). Rerun only failing DM Control envs (CartpoleSwingup, CartpoleSwingupSparse, + any that need it).322 323**Option B**: Accept 641 and note the trade-off — p5-ppo6 improved other envs (CartpoleBalance 968 was already ✅).324 325### Priority 3: FingerTurnEasy/Hard326 327No official override. At 570/? vs target 950, gap is large. Check:328```bash329grep -A10 "Finger" ~/.cache/uv/archive-v0/ON8dY3irQZTYI3Bok0SlC/mujoco_playground/config/dm_control_suite_params.py330```331 332May need deeper policy network [32,32,32,32] (official arch) vs our [64,64].333 334---335 336## Tuning Principles Learned337 3381. **Check official per-env overrides first**: `dm_control_suite_params.py` has `discounting`, `action_repeat`, `num_updates_per_batch` per env. These are canonical.339 3402. **action_repeat** is env-level, not spec-level. Implemented in `playground.py` via `_ACTION_REPEAT` dict. PendulumSwingup→4. Add others as found.341 3423. **NaN loss**: `log_std` clamp max=0.5 helps but Humanoid (21 DOF) still has many NaN skips. Rate-limited to log every 10K. If NaN dominates → spec is wrong.343 3444. **num_envs scales with task complexity**: Cartpole/Acrobot: 2048 fine. Humanoid locomotion: needs 8192 for rollout diversity.345 3465. **time_horizon (unroll_length)**: DM Control official=30, loco official=20. Longer → more correlated rollouts → less diversity per update. Match official.347 3486. **Minibatch count**: more minibatches = more gradient steps per batch. Can overfit or slow convergence for simpler envs. 15 minibatches (p5-ppo5) vs 30 (p5-ppo6) — the latter hurt CartpoleSwingup.349 3507. **Sparse reward + strength metric**: strength (trajectory mean) severely penalizes sparse/delayed convergence. CartpoleBalanceSparse strength=545 but final MA=992. Resolve metric before publishing.351 3528. **High seed variance** (consistency < 0): some seeds solve, some don't → wrong spec, not bad luck. Fix exploration (entropy_coef) or use different spec.353 3549. **-s overrides are silently ignored** if the YAML key isn't a `${variable}` placeholder. Always verify overrides took effect via logs: `grep "gamma\|lr\|training_epoch" dstack logs`.355 35610. **Loco spec failures**: if loco spec gives <20 on env with target >100, the issue is almost certainly num_envs/time_horizon mismatch vs official, not a fundamental algo failure.357 358---359 360## Code Changes This Session361 362| Commit | Change |363|---|---|364| `8fe7bc76` | `playground.py`: `_ACTION_REPEAT` lookup for per-env action_repeat. `ppo_playground.yaml`: added `ppo_playground_fingerspin` and `ppo_playground_pendulum` specs. |365| `fb55c2f9` | `base.py`: rate-limit NaN loss warning (every 10K skips). `ppo_playground.yaml`: revert log_frequency 1M→100K. |366| `3f4ede3d` | BENCHMARKS.md: mark HopperHop ✅. |367 368---369 370## Resume Commands371 372```bash373# Setup374git pull && uv sync --no-default-groups375 376# Check jobs377dstack ps378 379# Intake a completed run380dstack logs RUN_NAME --since 6h 2>&1 | grep "trial_metrics" | tail -1381dstack logs RUN_NAME --since 6h 2>&1 | grep -iE "Uploading|benchmark-dev"382 383# Pull HF data384source .env && huggingface-cli download SLM-Lab/benchmark-dev \385  --local-dir hf_data/data/benchmark-dev --repo-type dataset \386  --include "data/FOLDER_NAME/*"387 388# Plot389uv run slm-lab plot -t "EnvName" -d hf_data/data/benchmark-dev/data -f FOLDER_NAME390 391# Launch PendulumSwingup (queue priority 1)392source .env && uv run slm-lab run-remote --gpu \393  slm_lab/spec/benchmark_arc/ppo/ppo_playground.yaml ppo_playground_pendulum train \394  -s env=playground/PendulumSwingup -s max_frame=100000000 -n p5-ppo6-pendulumswingup395 396# Launch FingerSpin canonical (queue priority 2)397source .env && uv run slm-lab run-remote --gpu \398  slm_lab/spec/benchmark_arc/ppo/ppo_playground.yaml ppo_playground_fingerspin train \399  -s env=playground/FingerSpin -s max_frame=100000000 -n p5-ppo6-fingerspin3400 401# Launch Humanoid loco (after updating ppo_playground_loco spec to num_envs=8192, time_horizon=20)402source .env && uv run slm-lab run-remote --gpu \403  slm_lab/spec/benchmark_arc/ppo/ppo_playground.yaml ppo_playground_loco train \404  -s env=playground/HumanoidRun -s max_frame=100000000 -n p5-ppo6-humanoidrun2405```406 407---408 409## CRITICAL CORRECTION (2026-03-13) — Humanoid is DM Control, not Loco410 411**Root cause of Humanoid failure**: HumanoidRun/Walk/Stand are registered in `dm_control_suite/__init__.py` — they ARE DM Control envs. We incorrectly ran them with `ppo_playground_loco` (gamma=0.97, 4 epochs, time_horizon=64).412 413Official config uses DEFAULT DM Control params for them: discounting=0.995, 2048 envs, lr=1e-3, unroll_length=30, 16 epochs.414 415**NaN was never the root cause** — intake-b confirmed NaN skips were 0, 0, 2 in the loco runs. The spec was simply wrong.416 417**Fix**: Run all 3 Humanoid envs with `ppo_playground` (DM Control spec). No spec change needed.418 419```bash420# Launch with correct spec421source .env && uv run slm-lab run-remote --gpu \422  slm_lab/spec/benchmark_arc/ppo/ppo_playground.yaml ppo_playground train \423  -s env=playground/HumanoidRun -s max_frame=100000000 -n p5-ppo6-humanoidrun2424 425source .env && uv run slm-lab run-remote --gpu \426  slm_lab/spec/benchmark_arc/ppo/ppo_playground.yaml ppo_playground train \427  -s env=playground/HumanoidWalk -s max_frame=100000000 -n p5-ppo6-humanoidwalk2428 429source .env && uv run slm-lab run-remote --gpu \430  slm_lab/spec/benchmark_arc/ppo/ppo_playground.yaml ppo_playground train \431  -s env=playground/HumanoidStand -s max_frame=100000000 -n p5-ppo6-humanoidstand2432```433 434**HopperStand**: Also a DM Control env. If p5-ppo6-hopperstand (loco spec, 16.38) is below target, rerun with `ppo_playground`.435 436**Do NOT intake** the loco-spec Humanoid runs (2.78/6.82/12.45) — wrong spec, not valid benchmark results. The old ppo_playground runs (2.86/3.73) were also wrong spec but at least the right family.437 438**Updated queue (prepend these as highest priority)**:439 440| Priority | Env | Spec | Run name |441|---|---|---|---|442| 0 | HumanoidRun | ppo_playground | p5-ppo6-humanoidrun2 |443| 0 | HumanoidWalk | ppo_playground | p5-ppo6-humanoidwalk2 |444| 0 | HumanoidStand | ppo_playground | p5-ppo6-humanoidstand2 |445| 0 | HopperStand | ppo_playground | p5-ppo6-hopperstand2 (if loco result ⚠️) |446 447Note on loco spec (`ppo_playground_loco`): only for actual locomotion robot envs (Go1, G1, BerkeleyHumanoid, etc.) — NOT for DM Control Humanoid.448 449---450 451## METRIC CORRECTION (2026-03-13) — strength vs final_strength452 453**Problem**: `strength` = trajectory-averaged mean over entire run. For slow-rising envs this severely underrepresents end-of-training performance. After metric correction to `strength`:454 455| Env | strength | total_reward_ma | target | conclusion |456|---|---|---|---|---|457| CartpoleSwingup | **443.0** | 641.51 | 800 | Massive regression from p5-ppo5 (803). Strength 443 << 665 (65M result) — curve rises but slow start drags average down |458| CartpoleBalanceSparse | **545.1** | 991.81 | 700 | Hits target by end (final MA=992) but sparse reward delays convergence |459| AcrobotSwingup | **172.8** | 253.24 | 220 | Below target by strength, above by final MA |460| CartpoleSwingupSparse | **270.9** | 331.23 | 425 | Below both metrics |461 462**Resolution needed**: Reference scores from mujoco_playground are end-of-training values, not trajectory averages. `final_strength` (= last eval MA) is the correct comparison metric. **Recommend switching BENCHMARKS.md score column to `final_strength`** and audit all existing entries.463 464**CartpoleSwingup regression** is real regardless of metric: p5-ppo5 `final_strength` would be ~800+, p5-ppo6 `total_reward_ma`=641. The p5-ppo6 minibatch change (2048→30 minibatches) hurt CartpoleSwingup convergence speed. Fix: revert `ppo_playground` minibatch_size to 4096 (15 minibatches) — OR accept and investigate if CartpoleSwingup needs its own spec variant.465 466---467 468## Next Architectural Changes469 470Research-based prioritized list of changes NOT yet tested. Ordered by expected impact across the most envs. Wave I (5-layer value + no grad clip) is currently running — results pending.471 472### Priority 1: NormalTanhDistribution (tanh-squashed actions)473 474**Expected impact**: HIGH — affects FingerTurnEasy/Hard, FishSwim, Humanoid, CartpoleSwingup475**Implementation complexity**: MEDIUM (new distribution class + policy_util changes)476**Envs helped**: All continuous-action envs, especially precision/manipulation tasks477 478**What Brax does differently**: Brax uses `NormalTanhDistribution` — samples from `Normal(loc, scale)`, then applies `tanh` to bound actions to [-1, 1]. The log-probability includes a log-det-jacobian correction: `log_prob -= log(1 - tanh(x)^2)`. The scale is parameterized as `softplus(raw_scale) + 0.001` (state-dependent, output by the network).479 480**What SLM-Lab does**: Raw `Normal(loc, scale)` with state-independent `log_std` as an `nn.Parameter`. Actions can exceed [-1, 1] and are silently clipped by the environment. The log-prob does NOT account for this clipping, creating a mismatch between the distribution the agent thinks it's using and the effective action distribution.481 482**Why this matters**:4831. **Gradient quality**: Without jacobian correction, the policy gradient is biased. Actions near the boundary (common in precise manipulation like FingerTurn) have incorrect log-prob gradients. The agent cannot learn fine boundary control.4842. **Exploration**: State-dependent std allows the agent to be precise where it's confident and exploratory where uncertain. State-independent std forces uniform exploration across all states — wasteful for tasks requiring both coarse and fine control.4853. **FingerTurn gap (571/950 = 60%)**: FingerTurn requires precise angular positioning of a fingertip. Without tanh squashing, actions at the boundary are clipped but the log-prob doesn't reflect this — the policy "thinks" it's outputting different actions that are actually identical after clipping. This prevents learning fine-grained control near action limits.4864. **Humanoid gap (<3%)**: 21 DOF with high-dimensional action space. State-independent std means all joints explore equally. Humanoid needs to stabilize torso (low variance) while exploring leg movement (high variance) — impossible with shared std.487 488**Implementation plan**:4891. Add `NormalTanhDistribution` class in `slm_lab/lib/distribution.py`:490   - Forward: `action = tanh(Normal(loc, scale).rsample())`491   - log_prob: `Normal.log_prob(atanh(action)) - log(1 - action^2 + eps)`492   - entropy: approximate (no closed form for tanh-Normal)4932. Modify `policy_util.init_action_pd()` to handle the new distribution4943. Remove `log_std_init` for playground specs — let the network output both mean and std (state-dependent)4954. Network change: policy output dim doubles (mean + raw_scale per action dim)496 497**Risk**: Medium. Tanh squashing changes gradient dynamics significantly. Need to validate on already-solved envs (CartpoleBalance, WalkerRun) to ensure no regression. Can gate behind a spec flag (`action_pdtype: NormalTanh`).498 499---500 501### Fix 6: Constant LR variants + Humanoid variant (commit pending)502 503Added three new spec variants to `ppo_playground.yaml`:504- `ppo_playground_constlr`: DM Control + constant LR + minibatch_size=4096. For envs where vnorm=false works.505- `ppo_playground_vnorm_constlr`: DM Control + vnorm + constant LR + minibatch_size=2048. For precision envs.506- `ppo_playground_humanoid`: 2×256 policy + constant LR + vnorm. For Humanoid DM Control envs.507 508---509 510### Priority 2: Constant LR (remove LinearToMin decay)511 512**Expected impact**: MEDIUM — affects all envs, especially long-training ones513**Implementation complexity**: TRIVIAL (spec-only change)514**Envs helped**: CartpoleSwingup, CartpoleSwingupSparse, FingerTurnEasy/Hard, FishSwim515 516**What Brax does**: Constant LR = 1e-3 for all DM Control envs. No decay.517 518**What SLM-Lab does**: `LinearToMin` decay from 1e-3 to 3.3e-5 (min_factor=0.033) over the full training run.519 520**Why this matters**: By the midpoint of training, SLM-Lab's LR is already at ~5e-4 — half the Brax LR. By 75% of training, it's at ~2.7e-4. For envs that converge late (CartpoleSwingup, FishSwim), the LR is too low during the critical learning phase. Brax maintains full learning capacity throughout.521 522**This was tested as part of the Brax hyperparameter bundle (Wave E) which was ALL worse**, but that test changed 4 things simultaneously (clip_eps=0.3 + constant LR + 5-layer value + reward_scale=10). The constant LR was never tested in isolation.523 524**Implementation**: Set `min_factor: 1.0` in spec (or remove `lr_scheduler_spec` entirely).525 526**Risk**: Low. Constant LR is the Brax default and widely used. If instability occurs late in training, a gentler decay (`min_factor: 0.3`) can be used as fallback.527 528---529 530### Priority 3: Clip epsilon 0.3 (from 0.2)531 532**Expected impact**: MEDIUM — affects all envs533**Implementation complexity**: TRIVIAL (spec-only change)534**Envs helped**: FingerTurnEasy/Hard, FishSwim, CartpoleSwingup (tasks needing faster policy adaptation)535 536**What Brax does**: `clipping_epsilon=0.3` for DM Control.537 538**What SLM-Lab does**: `clip_eps=0.2`.539 540**Why this matters**: Clip epsilon 0.2 constrains the policy ratio to [0.8, 1.2]. At 0.3, it's [0.7, 1.3] — allowing 50% larger policy updates per step. For envs that need to explore widely before converging (FingerTurn, FishSwim), the tighter constraint slows learning.541 542**This was tested in the Brax bundle (Wave E) alongside 3 other changes — all worse together.** Never tested in isolation or with just constant LR.543 544**Implementation**: Change `start_val: 0.2` to `start_val: 0.3` in `clip_eps_spec`.545 546**Risk**: Low-medium. Larger clip_eps can cause training instability with small batches. However, with our 61K batch (2048 envs * 30 steps), it should be safe. If combined with constant LR (#2), the compounding effect should be tested carefully.547 548---549 550### Priority 4: Per-env tuning for FingerTurn (if P1-P3 insufficient)551 552**Expected impact**: HIGH for FingerTurn specifically553**Implementation complexity**: LOW (spec variant)554**Envs helped**: FingerTurnEasy, FingerTurnHard only555 556If NormalTanh + constant LR + clip_eps=0.3 don't close the FingerTurn gap (currently 60% and 51% of target), try:557 5581. **Lower gamma (0.99 → 0.95)**: FingerSpin uses gamma=0.95 officially. FingerTurn may benefit from shorter horizon discounting since reward is instantaneous (current angle vs target). Lower gamma reduces value function complexity.559 5602. **Smaller policy network**: Brax DM Control uses `(32, 32, 32, 32)` — our `(64, 64)` may over-parameterize for manipulation tasks. Try `(32, 32, 32, 32)` to match exactly.561 5623. **Higher entropy coefficient**: FingerTurn has a narrow solution manifold. Increasing entropy from 0.01 to 0.02 would encourage broader exploration of finger positions.563 564---565 566### Priority 5: Humanoid-specific — num_envs=8192567 568**Expected impact**: HIGH for Humanoid specifically569**Implementation complexity**: TRIVIAL (spec-only)570**Envs helped**: HumanoidStand, HumanoidWalk, HumanoidRun571 572**Current situation**: Humanoid was incorrectly run with loco spec (gamma=0.97, 4 epochs). The correction to DM Control spec (gamma=0.995, 16 epochs) is being tested in Wave I (p5-ppo13). However, even with correct spec, the standard 2048 envs may be insufficient.573 574**Why num_envs matters for Humanoid**: 21 DOF, 67-dim observations. With 2048 envs and time_horizon=30, the batch is 61K transitions — each containing a narrow slice of the 21-DOF state space. Humanoid needs more diverse rollouts to learn coordinated multi-joint control. Brax's effective batch of 983K transitions provides 16x more state-space coverage per update.575 576**Since we can't easily get 16x more data per update**, increasing num_envs from 2048 to 4096 or 8192 doubles/quadruples rollout diversity. Combined with NormalTanh (state-dependent std for per-joint exploration), this could be sufficient.577 578**VRAM concern**: 8192 envs may exceed A4500 20GB. Test with a quick 1M frame run first. Fallback: 4096 envs.579 580---581 582### NOT recommended (already tested, no benefit)583 584| Change | Wave | Result | Why it failed |585|---|---|---|---|586| normalize_v_targets: false | G/G2 | Mixed (helps 5, hurts 6) | Already per-env split in spec |587| Multi-unroll (num_unrolls=16) | F | Same or worse by strength | Stale old_net (480 vs 30 steps between copies) |588| Brax hyperparameter bundle (clip_eps=0.3 + constant LR + 5-layer value + reward_scale=10) | E | All worse | Confounded — 4 changes at once. Individual effects unknown except for reward_scale (helps) |589| time_horizon=480 (single long unroll) | C | Helps CartpoleSwingup, hurts FingerTurn | 480-step GAE is noisy for precision tasks |590| 5-layer value + no grad clip | III | Only helped CartpoleSwingup slightly | Hurt AcrobotSwingup, FishSwim; not general |591| NormalTanh distribution | II | Abandoned | Architecturally incompatible — SLM-Lab stores post-tanh actions, atanh inversion unstable |592| vnorm=true rerun (reverted spec) | IV | All worse or same | No new information — variance rerun |593| 4×32 Brax policy + constant LR + vnorm | VI | All worse | FingerTurnEasy 408 (vs 571), FingerTurnHard 244 (vs 484), FishSwim 106 (vs 463) |594| Humanoid wider 2×256 + constant LR + vnorm | IV-H | No improvement | MA flat at 8-10 for all 3 Humanoid envs; NormalTanh is root cause |595 596### Currently testing597 598### Wave V-B completed results (constant LR)599 600| Env | strength | final_strength | Old best | Verdict |601|---|---|---|---|---|602| PointMass | 841.3 | 877.3 | 863.5 | ❌ strength lower |603| **SwimmerSwimmer6** | **517.3** | 585.7 | 509.3 | ✅ NEW BEST (+1.6%) |604| FishSwim | 434.6 | 550.8 | 463.0 | ❌ strength lower (final much better) |605 606### Wave VII completed results (clip_eps=0.3 + constant LR)607 608| Env | strength | final_strength | Old best | Verdict |609|---|---|---|---|---|610| FingerTurnEasy | 518.0 | 608.8 | 570.9 | ❌ strength lower (final much better, but slow start drags average) |611| FingerTurnHard | 401.7 | 489.7 | 484.1 | ❌ strength lower (same pattern) |612| **FishSwim** | **476.9** | 581.4 | 463.0 | ✅ NEW BEST (+3%) |613 614**Key insight**: clip_eps=0.3 produces higher final performance but worse trajectory-averaged strength. The wider clip allows bigger policy updates which increases exploration early (slower convergence) but reaches higher asymptotic performance. The strength metric penalizes late bloomers.615 616### Wave V completed results617 618| Env | strength | final_strength | Old best | Verdict |619|---|---|---|---|---|620| CartpoleSwingup | **606.5** | 702.6 | 576.1 | ✅ NEW BEST (+5%) |621| CartpoleSwingupSparse | **383.7** | 536.2 | 296.3 | ✅ NEW BEST (+29%) |622| CartpoleBalanceSparse | **757.9** | 993.0 | 690.4 | ✅ NEW BEST (+10%) |623| AcrobotSwingup | 161.2 | 246.9 | 172.8 | ❌ strength lower (final_strength much better but trajectory avg worse due to slow start) |624 625**Key insight**: Constant LR is the single most impactful change found. LR decay from 1e-3 to 3.3e-5 was hurting late-converging envs. CartpoleBalanceSparse went from 690→993 (final_strength), effectively solved.626 627### Completed waves628 629**Wave VI** (p5-ppo18): 4×32 Brax policy — **STOPPED, all underperformed**. FingerTurnEasy MA 408, FingerTurnHard MA 244, FishSwim MA 106. All below old bests.630 631**Wave IV-H** (p5-ppo16h): Humanoid wider 2×256 + constant LR + vnorm — all flat at MA 8-10.632 633### Next steps after Wave VII634 6351. **Humanoid num_envs=4096/8192** — only major gap remaining after Wave VII6362. **Consider constant LR + clip_eps=0.3 as new general default** if results hold across envs637 638### Key Brax architecture differences (from source code analysis)639 640| Parameter | Brax Default | SLM-Lab | Impact |641|---|---|---|---|642| Policy | 4×32 (deeper, narrower) | 2×64 | **Testable via spec** |643| Value | 5×256 | 3×256 | Tested Wave III — no help |644| Distribution | tanh_normal | Normal | **Cannot test** (architectural incompatibility) |645| Init | lecun_uniform | orthogonal_ | Would need code change |646| State-dep std | False (scalar) | False (nn.Parameter) | Similar |647| Activation | swish (SiLU) | SiLU | ✅ Match |648| clipping_epsilon | 0.3 | 0.2 | **Testable via spec** |649| num_minibatches | 32 | 15-30 | Close enough |650| num_unrolls | 16 (implicit) | 1 | Tested Wave F — stale old_net hurts |651