huzzle-labs/visual_memory

1# Visual Memory Gym — Model Comparison2 3**Date**: 2026-03-18  4**Gym Version**: `0.1.0`  5**Scenarios**: 10 (across 4 task families)  6**Models**: 5 (3 Anthropic, 2 OpenAI)  7**Reward Modes**: custom (episode-level) and openenv (per-step transforms)8 9---10 11## Overall Results12 13### Custom Rewards (episode-level from `rewards/base.py`)14 15Reward components: Structural (0.25) + Ground Truth (0.60) + Efficiency (0.15) - Hallucination Penalty (up to -1.0)16 17| # | Model | Avg Reward | Best Scenario | Worst Scenario | Total Time |18|---|-------|:---:|---|---|---:|19| 1 | `claude-opus-4-6` | **0.08** | flash_fade_minefield (0.63) | cascading_deduction (-0.83) | 1518.9s |20| 2 | `gpt-5` | **-0.13** | flash_fade_minefield (0.69) | decoy_minefield (-0.78) | 2967.2s |21| 3 | `gpt-5.4` | **-0.16** | partial_intel (0.63) | directional_trap (-0.81) | 225.1s |22| 4 | `claude-opus-4-20250514` | **-0.17** | ambiguous_cluster (0.60) | cascading_deduction (-0.80) | 1197.4s |23| 5 | `claude-sonnet-4-6` | **-0.33** | partial_intel (0.40) | directional_trap (-0.83) | 1105.2s |24 25### OpenEnv Transform Rewards (per-step from `rewards/transforms/`)26 27Reward components: Step Rewards (0.40) + Ground Truth (0.60) - Hallucination Penalty (up to -1.0)28 29| # | Model | Avg Reward | Best Scenario | Worst Scenario | Total Time |30|---|-------|:---:|---|---|---:|31| 1 | `claude-opus-4-6` | **0.31** | directional_trap (0.37) | decoy_minefield (0.14) | 1584.6s |32| 2 | `claude-opus-4-20250514` | **0.31** | ambiguous_cluster (0.55) | delayed_recall_keys (0.13) | 1185.2s |33| 3 | `gpt-5` | **0.28** | flash_fade_minefield (0.53) | ambiguous_cluster (0.12) | 3770.8s |34| 4 | `gpt-5.4` | **0.27** | fog_labyrinth (0.52) | directional_trap (0.08) | 287.4s |35| 5 | `claude-sonnet-4-6` | **0.26** | ambiguous_cluster (0.35) | directional_trap (0.14) | 1048.0s |36 37---38 39## Per-Scenario Breakdown — Custom Rewards40 41| Scenario | gpt-5.4 | gpt-5 | claude-sonnet-4-6 | claude-opus-4-6 | claude-opus-4-20250514 | Avg |42|---|:---:|:---:|:---:|:---:|:---:|:---:|43| ambiguous_cluster_10x10 | 0.40 | -0.75 | 0.38 | 0.36 | 0.60 | **0.20** |44| directional_trap_8x8 | -0.81 | 0.41 | -0.83 | 0.61 | 0.42 | **-0.04** |45| partial_intel_9x9 | 0.63 | 0.42 | 0.40 | 0.41 | 0.44 | **0.46** |46| flash_fade_minefield_7x7 | 0.42 | 0.69 | -0.78 | 0.63 | -0.76 | **0.04** |47| delayed_recall_keys_8x8 | 0.45 | 0.53 | -0.80 | 0.43 | -0.79 | **-0.04** |48| decoy_minefield_8x10 | -0.80 | -0.78 | -0.81 | -0.82 | -0.79 | **-0.80** |49| fog_labyrinth_10x10 | -0.74 | 0.44 | 0.39 | 0.40 | 0.40 | **0.18** |50| fog_key_hunt_8x8 | -0.73 | -0.77 | -0.79 | -0.80 | -0.79 | **-0.78** |51| cascading_deduction_11x11 | -0.78 | -0.76 | -0.79 | -0.83 | -0.80 | **-0.79** |52| safe_zone_identification_9x9 | 0.40 | -0.77 | 0.37 | 0.38 | 0.41 | **0.16** |53 54### Hardest Scenarios (Custom)551. **decoy_minefield_8x10** (avg -0.80): All 5 models fail — hallucination penalty triggered universally562. **cascading_deduction_11x11** (avg -0.79): Large board with partial signals defeats all models573. **fog_key_hunt_8x8** (avg -0.78): Tiny viewport + fatal hazards — no model survives58 59### Easiest Scenario (Custom)601. **partial_intel_9x9** (avg 0.46): Most models achieve positive rewards here61 62---63 64## Per-Scenario Breakdown — OpenEnv Transform Rewards65 66| Scenario | gpt-5.4 | gpt-5 | claude-sonnet-4-6 | claude-opus-4-6 | claude-opus-4-20250514 | Avg |67|---|:---:|:---:|:---:|:---:|:---:|:---:|68| ambiguous_cluster_10x10 | 0.33 | 0.12 | 0.35 | 0.35 | 0.55 | **0.34** |69| directional_trap_8x8 | 0.08 | 0.36 | 0.14 | 0.37 | 0.36 | **0.26** |70| partial_intel_9x9 | 0.32 | 0.34 | 0.34 | 0.36 | 0.36 | **0.34** |71| flash_fade_minefield_7x7 | 0.35 | 0.53 | 0.34 | 0.34 | 0.35 | **0.38** |72| delayed_recall_keys_8x8 | 0.34 | 0.34 | 0.34 | 0.36 | 0.13 | **0.30** |73| decoy_minefield_8x10 | 0.14 | 0.14 | 0.14 | 0.14 | 0.34 | **0.18** |74| fog_labyrinth_10x10 | 0.52 | 0.34 | 0.34 | 0.35 | 0.35 | **0.38** |75| fog_key_hunt_8x8 | 0.13 | 0.13 | 0.15 | 0.15 | 0.15 | **0.14** |76| cascading_deduction_11x11 | 0.13 | 0.14 | 0.15 | 0.35 | 0.14 | **0.18** |77| safe_zone_identification_9x9 | 0.34 | 0.34 | 0.34 | 0.34 | 0.35 | **0.34** |78 79### Hardest Scenarios (OpenEnv)801. **fog_key_hunt_8x8** (avg 0.14): Tiny viewport + fatal hazards — universally low812. **decoy_minefield_8x10** (avg 0.18): Decoy-key confusion trips all models823. **cascading_deduction_11x11** (avg 0.18): Large partial-signal board overwhelms reasoning83 84### Easiest Scenarios (OpenEnv)851. **flash_fade_minefield_7x7** (avg 0.38): Pattern memory — some models excel here862. **fog_labyrinth_10x10** (avg 0.38): Fog navigation with reasonable viewport87 88---89 90## SOTA Average Assessment91 92**SOTA models** (gpt-5.4, claude-sonnet-4-6, claude-opus-4-6):93 94| Reward Mode | SOTA Average | Target Band | Status |95|---|:---:|:---:|---|96| Custom | **-0.14** | 0.60–0.70 | Well below target — no hardening needed |97| OpenEnv | **0.28** | 0.60–0.70 | Below target — no hardening needed |98 99**All 5 models average:**100 101| Reward Mode | Overall Average |102|---|:---:|103| Custom | **-0.14** |104| OpenEnv | **0.28** |105 106The gym is currently **harder than target** across both reward modes. No hardening adjustments are required.107 108---109 110## Reward Mode Comparison111 112| Metric | Custom | OpenEnv |113|---|:---:|:---:|114| Mean across all models | -0.14 | 0.28 |115| Std deviation (models) | 0.15 | 0.02 |116| Min model avg | -0.33 | 0.26 |117| Max model avg | 0.08 | 0.31 |118| Hallucination penalties hit | Frequent (-1.0) | None triggered |119| Reward spread | Very high (variance from penalties) | Compressed (narrow 0.12–0.55 range) |120 121**Key insight**: Custom rewards produce highly volatile scores driven by the -1.0 hallucination penalty. When models make even one incorrect assertion (tools report success but ground truth disagrees), the entire scenario score collapses. OpenEnv transform rewards are more granular and forgiving, rewarding incremental progress per-step.122 123---124 125## Model Speed Rankings126 127| Model | Custom Time | OpenEnv Time | Avg per Scenario |128|---|:---:|:---:|:---:|129| `gpt-5.4` | 225.1s | 287.4s | ~26s |130| `claude-sonnet-4-6` | 1105.2s | 1048.0s | ~108s |131| `claude-opus-4-20250514` | 1197.4s | 1185.2s | ~119s |132| `claude-opus-4-6` | 1518.9s | 1584.6s | ~155s |133| `gpt-5` | 2967.2s | 3770.8s | ~337s |134 135GPT-5.4 is 6x faster than the next model while achieving competitive results.136 137---138 139## Distractor Tool Usage140 141Models occasionally used distractor/trap tools, which indicates susceptibility to misleading tool descriptions:142 143- **`peek_hidden_cell`**: Used by claude-opus-4-6 and claude-sonnet-4-6 (cheating tool — gives hidden info but penalized)144- **`undo_last_action`**: Used by claude-sonnet-4-6 (no-op trap)145- **`reset_scenario`**: Used by multiple models (resets game state — wastes steps)146- **`auto_solve`**: Not used by any model (good — most egregious trap avoided)147 148---149 150## Task Family Analysis151 152| Task Family | Scenarios | Custom Avg | OpenEnv Avg | Difficulty |153|---|---|:---:|:---:|---|154| Hidden Grid (5) | ambiguous, directional, partial, cascading, safe_zone | -0.06 | 0.29 | Medium-Hard |155| Pattern Memory (2) | flash_fade, delayed_recall | 0.00 | 0.34 | Medium |156| Fog of War (2) | fog_labyrinth, fog_key_hunt | -0.30 | 0.26 | Hard |157| Distractor Search (1) | decoy_minefield | -0.80 | 0.18 | Very Hard |158 159---160 161## Files162 163| Type | Path |164|---|---|165| Custom results (all 5 models) | `results/visual_memory/run_visual_memory_custom.md` |166| OpenEnv results (all 5 models) | `results/visual_memory/run_visual_memory_openenv.md` |167| Custom trajectories (all 5 models) | `trajectories/visual_memory/run_visual_memory_custom/` |168| OpenEnv trajectories (all 5 models) | `trajectories/visual_memory/run_visual_memory_openenv/` |169