s1nn3rx69/recall-env

 ┌─────────────────────────────────────────────────────────────────┐
 │                      RECALL Episode                             │
 │                                                                 │
 │  PHASE A: Ingestion (1 turn)                                    │
 │  ┌─────────────────────────────────────────────────────────┐   │
 │  │  Agent sees ALL facts at once                           │   │
 │  │  For each fact: decide STORE or SKIP                    │   │
 │  │  If STORE: write a retrieval ANCHOR (≤64 tokens)        │   │
 │  │  Memory budget is FIXED — you can't store everything    │   │
 │  │  Agent doesn't see the queries yet → uncertainty        │   │
 │  └─────────────────────────────────────────────────────────┘   │
 │                          ▼                                      │
 │  PHASE B: Query Phase (1-2 turns per query)                     │
 │  ┌─────────────────────────────────────────────────────────┐   │
 │  │  Query arrives                                          │   │
 │  │  Agent can RETRIEVE (cosine search over anchors)        │   │
 │  │  Agent must ANSWER                                      │   │
 │  │  Graded against ground truth                            │   │
 │  └─────────────────────────────────────────────────────────┘   │
 └─────────────────────────────────────────────────────────────────┘

Fact stored at time 0:   "Tried 8L-XL with LR=3e-4, got val_acc=0.612."
Query arrives at time 1: "What was the validation accuracy of the 8-layer extra-large run?"
                                        ↑                    ↑
                               "val_acc" → "validation accuracy"
                               "8L-XL"  → "8-layer extra-large"

from envs.recall_env import RecallEnv, RecallAction

async with RecallEnv.from_env("s1nn3rx69/recall-env") as env:
    obs = await env.reset(difficulty=2, seed=0)
    # Ingestion phase: agent sees all facts, writes decisions
    action = RecallAction(
        mode="ingest",
        decisions=[
            {"fact_id": 0, "decision": "store", "anchor": "8L-XL val_acc experiment LR 3e-4"},
            {"fact_id": 1, "decision": "skip"},   # distractor
            ...
        ]
    )
    obs = await env.step(action)
    # Query phase: agent retrieves and answers

def phase1_reward(episode_result, config):
    r = episode_result.correct_answers * 1.0     # primary signal
    r += episode_result.stored_then_retrieved * 0.1  # mild shaping
    r -= episode_result.memory_used * 0.02          # storage cost
    r += episode_result.malformed_count * (-0.5)    # format penalty
    return r

**Why this is hard to hack:** answering UNKNOWN everywhere fails because 70%+ of queries have real answers. Storing nothing fails because you can't beat a baseline that stores *something*. Storing everything hits the budget cap and becomes a poorly-anchored FIFO — still can't beat FIFO itself.

The key diagnostic metric is `train/reward_std_within_group`. If this collapses to zero for 30+ steps, GRPO has no gradient. Stop training. Diagnose.

---


**Store-Everything**: Stores every fact verbatim until budget is exhausted. Simple top-1 retrieval for answers. Hits the cap on every level above L1.

**FIFO** (First-In-First-Out): Stores everything until full, then evicts the oldest fact to make room for each new one. Recency bias hurts when important early facts get evicted. Target accuracy: ~38% at L2/L3.

**LLM-as-Judge**: Uses the *same base model* (Qwen2.5-3B-Instruct) to score each fact's importance via prompting, stores high-scoring facts. Uses verbatim fact text as anchor. This is the direct prior art (Generative Agents 2023). The trained policy must beat this to demonstrate the value of RL over prompting. Target accuracy: ~51% at L2/L3.

**Trained RECALL Policy target**: ≥70% at L2/L3.

---

## Training Setup

- **Base model**: `Qwen/Qwen2.5-3B-Instruct`
- **Adapter**: LoRA via PEFT
- **Trainer**: GRPOTrainer from TRL (group size 8)
- **Hardware**: HuggingFace GPU credits (A100)

**Curriculum schedule**:
| Level | Steps | Bootstrap steps |
|-------|-------|----------------|
| L1 | 200 | 100 |
| L2 | 400 | 200 |
| L3 | 800 | 0 (binary only — rely on L1/L2 transfer) |
| L4 | 600 | 0 |
| L5 | 600 | 0 |

---

## Training Results

### L1 — Action Grammar

The first level was designed to verify the full training pipeline works and the model can learn to emit valid structured decisions.

![L1 Training Metrics](plots/l1_final_training_metrics.png)

**What the plots show:**

- **Loss** stabilises near zero by step 15 and stays there — the policy is converging cleanly without instability
- **Reward** climbs from ~1.5 early in training to a stable plateau of ~2.3–2.5, consistently above the naive baseline across all eval checkpoints (orange diamonds). The model learned to reliably beat random and FIFO at this difficulty level
- **KL Divergence** starts near zero and rises gradually to ~0.001–0.005 by step 50 — the adapter is meaningfully diverging from the base model (learning is happening), but not catastrophically (no instability)
- **Mean Completion Length** stabilises at ~93–94 tokens — the model settled into a consistent response format

**What this means:** The model successfully learned the action grammar (valid JSON, correct field names, store/skip decisions). Eval reward at steps 10, 20, 30, 40, and 50 all landed above the training average, confirming the policy generalises to held-out seeds.

---

### L2 — Recency + Tag Heuristics

L2 is where real learning begins. Budget pressure is introduced, distractors appear, and the model must learn to skip irrelevant facts and prefer tagged/recent ones.

![L2 Training Metrics — First Epoch](plots/first_l2_metrics_plot.png)

**What the plots show:**

- **Reward** starts volatile in early training — reward swings between −1.0 and +1.7 as the model navigates the harder problem space. By epoch 0.6–1.0, reward stabilises consistently above 1.5, with many steps reaching the maximum of ~2.0
- **KL Divergence** now grows substantially — reaching 0.05–0.22 by late training. This is exactly what we want to see: the adapter is meaningfully diverging from the base model because it is learning a genuinely different policy, not just parroting the base model's heuristics
- **Mean Completion Length** drops from ~450 tokens in early training to ~300 tokens by mid-epoch, then stabilises. The model learns to be more concise in its decisions — a sign of policy refinement rather than brute-force enumeration

**The key result**: The trained L2 policy shows substantially higher reward than both the untrained base model and the FIFO baseline on held-out seeds. This validates the core claim: reward signal is teaching the model to prefer important facts and skip distractors, a skill neither the base model nor FIFO possess.We have trained for more than 200 epoch each time so we ran out of time to train our model over the complete curriculumn

The early volatility followed by late stabilisation is the characteristic GRPO learning signature: the model explores diverse strategies in the group of 8 completions, finds the ones that beat the baseline, and exploits them.

---

## Data Generation

The synthetic dataset models a PhD student's transformer experiment journal. All data is generated deterministically from `(difficulty, seed)` — no two training runs see the same episodes unless seeded identically.

**Vocabulary files** (generated once, committed to repo):

- `architectures.json`: 80 descriptions (full name + abbreviation) — "8-layer extra-large" / "8L-XL"
- `hyperparameters.json`: 40 (name, abbrev, value range) triples
- `metrics.json`: 30 (full, abbrev) pairs — "validation accuracy" / "val_acc"
- `papers.json`, `hypotheses.json`, `decisions.json`, `debug_findings.json`, `distractors.json`

**Fact categories**: Experiment results, architectural decisions, paper findings, hypotheses, debugging notes, corrections (L4+), distractors

**Query types**: Specific fact retrieval (30%), aggregation (20%), contradiction resolution (15%), rationale retrieval (15%), negative recall (10%), distractor resistance (10%)

**Ground truth is always mechanical** — assembled from template pairings, never inferred. Same seed always produces the same episode and the same correct answers.

---

## Evaluation Framework

### Metrics Tracked

| Metric                          | Significance                                                           |
| ------------------------------- | ---------------------------------------------------------------------- |
| `train/reward_std_within_group` | If this hits zero, GRPO has no gradient. The most important diagnostic |
| `train/accuracy`                | Fraction of queries answered correctly                                 |
| `train/malformed_action_rate`   | Should drop to <5% in first 50 steps at any level                      |
| `train/memory_utilization`      | Should not sit at 100% (hoarding) or 0% (skipping everything)          |
| `train/skip_rate`               | Should rise as the agent learns selectivity                            |
| `eval/accuracy_L1`, `L2`, ...   | Regression tracking — catastrophic forgetting check                    |
| `eval/failure_mode_breakdown`   | Which step is failing: storage, anchor, retrieval, or reasoning        |

### Eval Protocol

- **Training seeds**: drawn from `range(1000, 100000)` — unbounded fresh seeds per step
- **Held-out eval seeds**: `range(0, 20)` — never seen during training
- **Eval runs**: 20 seeds per condition per level, all four conditions (store-all, FIFO, LLM-judge, trained) on the same seeds

---

## Failure Mode Taxonomy

Understanding _why_ the agent fails is as important as knowing that it fails. The environment records failure attribution per query:

| Failure Mode          | Cause                                                        | Signal                                        |
| --------------------- | ------------------------------------------------------------ | --------------------------------------------- |
| **Storage failure**   | Relevant fact was skipped during ingestion                   | Fact in stream, not in memory                 |
| **Anchor failure**    | Fact was stored but with a poor anchor — retrieval missed it | Fact in memory, not in top-k results          |
| **Retrieval failure** | Anchor was decent but query phrasing didn't match            | Fact retrievable with different query wording |
| **Reasoning failure** | Right content retrieved but wrong answer generated           | Correct fact in context, wrong output         |

The trained policy should show dramatically lower **anchor failure** compared to baselines — that's the specific skill being trained.

---

## Deployment

The environment is deployed on HuggingFace Spaces as an OpenEnv-compliant server:

**🌐 [https://huggingface.co/spaces/s1nn3rx69/recall-env](https://huggingface.co/spaces/s1nn3rx69/recall-env)**

### Local Setup

### Training

---

## Repository Structure

---

## What's Next

**Immediate (v1):**

- Complete L3 training (anchor authoring — the core claim)
- Implement and test F4 (Strengthening Through Repetition) — backend only, zero risk
- Produce final eval plots with all four conditions on the same 20 seeds

**Medium term:**

- F1 (Memory Tagging) + F2 (Memory Permanence) for L4
- F3 (Overwrite Action) for L5 contradiction handling
- Multi-domain vocabulary expansion (from PhD journal to engineering logs, medical records)

**Future work:**

- Multi-tier hot/cold memory (MemGPT direction)
- Graph edges between memory items
- Compression as a learned action
- Multi-agent memory sharing

---

## Open Questions (Active)

---

## Related Work

- **RLM (Recursive Language Models)**: Addresses read-time long-context decomposition. RECALL is the complementary write-time half.
- **MemGPT / Letta**: OS-style virtual memory paging with hardcoded rules. We learn the paging policy.
- **Generative Agents (Park et al. 2023)**: LLM-as-judge importance scoring. Our direct baseline — RECALL learns to beat it.
- **GraphRAG**: Heuristic graph extraction from text. We don't build graphs; we learn task-conditional importance.
- **LongMemEval / LoCoMo**: Evaluate end-to-end task performance. We isolate the memory policy as a separable trainable skill.

---

## References

training link :https://colab.research.google.com/drive/1SnKPeOyWwDYAm8Ke5Hjd5vJ37CbyPpQ5?usp=sharing
blog link : https://www.notion.so/RECALL-We-Trained-an-Agent-to-Remember-The-Hard-Part-Was-Teaching-It-to-Forget-34e222d4ac6080fb97e0d8bd51304187?source=copy_link

---

Every significant design decision in RECALL, explained with the reasoning behind it. Not a spec — the spec files cover what to build. This answers _why_ it was built that way. Use this when something looks arbitrary, when a reviewer asks a hard question, or when a teammate is about to change something without understanding what it protects.

---

## Table of Contents

1. [The Core Problem](#1-the-core-problem)
2. [Why an RL Environment — Not a Benchmark](#2-why-an-rl-environment--not-a-benchmark)
3. [Why the Anchor is the Central Mechanism](#3-why-the-anchor-is-the-central-mechanism)
4. [Why Single-Pass Ingestion](#4-why-single-pass-ingestion)
5. [Why GRPO — and What It Demands From the Environment](#5-why-grpo--and-what-it-demands-from-the-environment)
6. [Why Binary Reward Against FIFO](#6-why-binary-reward-against-fifo)
7. [Why the Two-Phase Reward Structure](#7-why-the-two-phase-reward-structure)
8. [Why the Curriculum Is Layered the Way It Is](#8-why-the-curriculum-is-layered-the-way-it-is)
9. [Why Low-Dimensional Projected Embeddings](#9-why-low-dimensional-projected-embeddings)
10. [Why Retrieval Is Over Anchors Only — Not Content](#10-why-retrieval-is-over-anchors-only--not-content)
11. [Why Lexical Mismatch Is Baked Into the Data](#11-why-lexical-mismatch-is-baked-into-the-data)
12. [Why Templated Data — Not LLM-Generated](#12-why-templated-data--not-llm-generated)
13. [Why the Domain Is a PhD Student's Lab Notebook](#13-why-the-domain-is-a-phd-students-lab-notebook)
14. [Why Memory Tagging Is L3 and Not Earlier](#14-why-memory-tagging-is-l3-and-not-earlier)
15. [Why Permanence Levels Are L4](#15-why-permanence-levels-are-l4)
16. [Why the Overwrite Action Is L5 and Not Before](#16-why-the-overwrite-action-is-l5-and-not-before)
17. [Why Strengthening Is Passive — Not an Action](#17-why-strengthening-is-passive--not-an-action)
18. [Why FIFO Is Precomputed at Reset](#18-why-fifo-is-precomputed-at-reset)
19. [Why Per-Session State Isolation Is Non-Negotiable](#19-why-per-session-state-isolation-is-non-negotiable)
20. [Why the Delete Action Was Removed](#20-why-the-delete-action-was-removed)
21. [Why the Action Parser Has This Fallback Sequence](#21-why-the-action-parser-has-this-fallback-sequence)
22. [Why We Use Qwen2.5-3B and Not a Larger Model](#22-why-we-use-qwen25-3b-and-not-a-larger-model)
23. [How RECALL Differs From Adjacent Work](#23-how-recall-differs-from-adjacent-work)
24. [What We Would Change With More Time](#24-what-we-would-change-with-more-time)

---

## 1. The Core Problem

Current memory systems for agents — vector stores, graph RAG, MemGPT-style paging — all share the same fundamental design: a human or a handcrafted rule decides the storage policy, and the system executes it. The intelligence, if any, is in the retrieval. Storage is treated as a collection problem, not a decision problem.

This breaks down in two ways that matter practically.

**The capacity problem.** Embedding spaces fill up. A 128-dimensional vector store with 10,000 items degrades retrieval quality through crowding. Every new item competes for attention with everything already stored. The conventional response is to compress or paginate. But neither approach asks the prior question: should this have been stored at all?

**The vocabulary mismatch problem.** Facts arrive in one vocabulary; queries arrive in a different one. An experimental note says "LR=3e-4, val_acc=0.612." A query asks "what learning rate produced the best validation accuracy?" Vector similarity over verbatim content does not reliably bridge this. The gap is not a retrieval problem — it is a storage problem. If the storage-time representation had been written to anticipate the query-time vocabulary, retrieval would work.

RECALL trains a policy that addresses both. It learns which facts to keep and how to phrase them so they can be found again. The storage decision and the anchor authoring are both outputs of the trained policy, shaped by the downstream query reward.

---

## 2. Why an RL Environment — Not a Benchmark

The standard approach to this problem would be: collect (fact stream, query, ground truth) triples, train a model to store the right facts and retrieve the right answers, evaluate on a held-out set.

That approach misses what we are actually trying to study. We are not trying to train a model to pass a fixed test. We are trying to train a _policy_ — a decision-making process that generalises to new streams, new query distributions, new importance structures. A policy is not a function from inputs to outputs; it is a function from states to actions, shaped by reward over episodes.

Benchmarks measure performance. RL environments train behaviour. The distinction matters because:

- A benchmark with fixed correct answers leaks information about the query distribution at training time. Our environment generates episodes from seed — the agent never sees the queries during ingestion. Selection under genuine uncertainty is structurally preserved.
- A benchmark trains a function approximator. An RL environment trains a cognitive strategy. We want the latter.
- A benchmark cannot express the temporal structure of importance (facts queried early versus late, facts that get contradicted, facts that are reinforced by repetition). Our curriculum does.

OpenEnv is the right framework because it enforces the episode abstraction — reset, step, reward — that RL training requires, while providing the WebSocket infrastructure GRPO needs for parallel rollouts.

---

## 3. Why the Anchor is the Central Mechanism

The anchor is the most novel element of RECALL. Everything else — budget constraints, curriculum levels, reward design — serves to make anchor authoring the skill being trained.

Here is the problem the anchor solves. When an agent stores a fact, it has two choices:

**Option A**: Store the fact verbatim. Retrieval uses cosine similarity over the verbatim text embedding. This fails the vocabulary mismatch problem — "LR=3e-4" does not match "what learning rate did you use."

**Option B**: Have the agent write a short retrieval key at storage time, indexed by its embedding instead of the fact's. The key can bridge the gap between how the fact is stated and how it will be queried.

Option B is what RECALL implements. The anchor is the agent's prediction of how the fact will be searched for. Writing a good anchor requires:

- Knowing that "LR=3e-4" will be queried as "learning rate" — abbreviation/expansion bridging
- Knowing that "val_acc=0.612" will be queried as "performance" — specific-to-categorical bridging
- Knowing that "grad norm 47" will be queried as "gradient instability" — domain knowledge bridging

None of this can be hardcoded. The mapping between storage vocabulary and query vocabulary is task-dependent and distribution-dependent. It must be learned. That is what the RL training does — it shapes the anchor-authoring policy via reward from downstream query accuracy.

There is published evidence that models trained end-to-end on retrieval tasks develop internal representations that are not semantically interpretable to humans but are optimised for machine retrieval (Sastre & Pascual, arXiv:2506.15001, June 2025). RECALL is deliberately creating the conditions for this to happen at the anchor level.

---

## 4. Why Single-Pass Ingestion

The prior design batched facts in groups of 8, processing each batch in a separate turn. At L3 (50 facts), this produced roughly 43 turns per episode.

GRPO training has a hard constraint on episode length. Based on empirical findings from TRL's own Wordle and Sudoku experiments:

- 1–3 turns: optimal for clean gradient signal
- 3–7 turns: workable with careful reward design
- 7+ turns: credit assignment degrades; training becomes unstable or fails

43 turns is structurally incompatible with GRPO. The gradient cannot reliably attribute reward to a specific storage decision that happened 35 turns earlier.

Single-pass ingestion collapses this to 1 ingestion turn plus a query phase. Episode length at L3 becomes roughly 8 turns. This is workable.

The concern with single-pass was: does it preserve the selection-under-uncertainty property? If the agent sees all 50 facts at once, is this just RAG?

It is not, for one reason: the agent still does not see the queries during ingestion. The uncertainty is about which facts will be queried, not about what the facts say. That uncertainty is entirely preserved in single-pass ingestion. What is lost is the streaming nature — the agent no longer has to predict importance as facts arrive. What is gained is training tractability. This is the correct trade.

---

## 5. Why GRPO — and What It Demands From the Environment

GRPO works by sampling a group of N completions for the same prompt and computing advantage by ranking within the group. No value network is needed — the advantage is estimated from the relative reward of each completion versus the group mean.

This has a critical implication for reward design: **within-group reward variance must be non-zero**. If all eight completions in a group receive the same reward, the advantage is zero everywhere and the gradient vanishes. This is training collapse, and it is silent — the loss may look fine while learning stops entirely.

The primary monitoring metric for RECALL training is therefore `train/reward_std_within_group`, not reward mean. If this drops to near zero for 30 or more consecutive steps, training has collapsed and must be diagnosed before continuing.

What produces within-group variance? Different storage decisions on the same fact stream leading to meaningfully different query accuracy. This requires:

1. The task to be genuinely hard — a random policy and an optimal policy must produce different outcomes
2. The reward to be outcome-sensitive — a policy that stores the right facts and authors good anchors must score better than one that does not
3. The curriculum level to be calibrated — if the level is too easy, every policy scores near 100% and variance collapses; if too hard, every policy scores near 0% and variance collapses again

The curriculum structure, the FIFO baseline comparison, and the 5-point margin in the binary reward are all designed to keep within-group variance in a productive range.

GRPO also opens 8 simultaneous environment connections (`num_generations=8`). Each connection must hit an independent environment instance. This is why `max_concurrent_envs: 8` in `openenv.yaml` is not a performance setting but a correctness requirement. If two connections share state, the training data is corrupted.

---

## 6. Why Binary Reward Against FIFO

The baseline reward design had roughly six dense shaping terms — bonuses for storing facts that later got retrieved, penalties for memory usage, bonuses for correct answers, penalties for malformed actions. This seemed principled. It was counterproductive.

Dense shaping with multiple terms narrows the variance of rewards across a GRPO group. If every completion earns some positive reward from some term, the spread collapses. GRPO has nothing to rank.

Binary reward against FIFO preserves variance. Either the agent beat the baseline on this episode or it did not. The outcomes are distinct. The gradient is sharp.

Why FIFO specifically? FIFO is the dumbest non-trivial baseline — it stores everything in order and evicts the oldest when full, using verbatim fact text as the anchor. It gets some queries right by luck. A trained agent that cannot consistently beat FIFO has not learned anything useful. FIFO is therefore both a meaningful difficulty calibration and a conceptually honest bar: "is your policy better than no policy at all?"

The 5-percentage-point margin in the clean-win condition (`agent_acc > baseline_acc + 0.05`) prevents reward noise from FIFO's seed-dependent variance from contaminating the training signal. A narrow win that could have gone either way is worth 0.3, not 1.0.

---

## 7. Why the Two-Phase Reward Structure

Binary reward against FIFO is the right signal at scale but problematic at the start of training. A completely untrained policy produces mostly malformed actions and random storage decisions. Its accuracy is near zero. FIFO's accuracy is not zero. The binary reward is `0.0` for every episode. Within-group variance is zero. Training does not start.

The bootstrap phase addresses this with a lightweight dense reward: correct answers score 1.0 each, mild bonuses for storage that led to successful retrieval, mild penalties for budget overflow, sharp penalties for malformed actions. This is enough to get the policy off zero — to teach it that answering correctly is better than answering randomly, that malformed output is always penalised.

Once the policy is above zero, the binary phase takes over. The bootstrap shaping is withdrawn entirely, leaving only the clean FIFO comparison signal.

The phase boundary is level-dependent:

- L1: 100 bootstrap steps. Easy level; short bootstrap needed.
- L2: 200 bootstrap steps. Budget pressure requires slightly longer bootstrap.
- L3 and beyond: 0 bootstrap steps. The policy transferred from L2 already has enough structure to produce meaningful variance under binary reward from step one.

The only exceptions in later levels are small L4-specific shaping terms (bonus for core memories that get queried, active only during bootstrap) and an L5 contradiction penalty. Both vanish after bootstrap. They exist purely to give early gradient signal for the new skills those levels introduce.

---

## 8. Why the Curriculum Is Layered the Way It Is

Each curriculum level introduces exactly one new cognitive skill. This is not aesthetic — it is a training requirement.

If two new skills are introduced simultaneously, reward improvement can come from either. The gradient cannot isolate which skill drove the improvement. If one skill is learned and the other is not, the training curve looks healthy while the unlearned skill remains a gap. These gaps surface as catastrophic failures at higher levels when both skills are required together.

One skill per level forces the credit assignment to be clean. The gradient knows what changed.

The skill sequence is ordered by dependency:

**L1 — Action grammar.** The agent learns to output valid structured JSON decisions. Nothing about content quality. Pure format learning. Without this, every subsequent level fails on malformed actions.

**L2 — Importance heuristics.** Budget pressure is introduced with explicit `[IMPORTANT]` tags and distractor facts. The agent learns that not everything should be stored and that some facts signal their importance explicitly. Tags are training wheels — they will be removed.

**L3 — Anchor authoring.** Tags are removed. Lexical mismatch is fully active. The agent must now infer importance from content and write anchors that bridge storage-time and query-time vocabulary. This is the central level. If the trained policy does not beat the LLM-as-judge baseline at L3, the project's claim fails.

**L4 — Permanence calibration.** The memory budget is split into core (permanent) and working (FIFO eviction) pools. The agent must predict which facts will be queried late in the episode and protect them as core. This requires temporal reasoning about importance — a strictly harder skill than L3's static importance estimation.

**L5 — Adversarial robustness.** Contradiction injection (18% of later facts supersede earlier ones), adversarial contradictions (5% of contradictions are false), tighter budget, longer streams. The agent must have learned everything from L1–L4 to navigate L5. L5 is a stress test, not a new skill introduction.

The regression evaluation (running L1–L<n-1> seeds after every L<n> training step) exists because catastrophic forgetting is a real risk. A policy that forgets how to parse valid JSON after L3 training is not useful.

---

## 9. Why Low-Dimensional Projected Embeddings

The embedding model (`sentence-transformers/all-MiniLM-L6-v2`) produces 384-dimensional vectors natively. RECALL projects these down to 128 dimensions using a random Gaussian projection matrix.

Two reasons.

**Training pressure.** At 384 dimensions, cosine similarity is high-resolution enough that verbatim fact text retrieves reasonably well even without careful anchor authoring. The agent can get away with lazy anchors. At 128 dimensions, the space is more compressed — semantically distinct facts cluster more tightly, and naive anchors (verbatim fact text) produce more retrieval collisions. The agent must write _better_ anchors to achieve reliable retrieval. Low dimensionality is a difficulty knob that makes the anchor authoring skill genuinely necessary.

**Deployability story.** The practical argument for RECALL is that trained memory policies can run on memory-constrained hardware — edge devices, mobile, small servers. 128-dimensional anchor embeddings are dramatically cheaper than 768 or 1536-dimensional ones. This is not just a training choice; it is a design choice that makes the trained policy deployable.

Why random Gaussian projection rather than PCA? PCA requires a corpus to fit. RECALL has no fixed corpus — episodes are generated dynamically from seed. Random projection with a seeded matrix is reproducible, requires no fitting, and has Johnson-Lindenstrauss guarantees: pairwise distances are preserved in expectation. Both anchor embeddings and query embeddings use the same projection matrix, so relative similarity is consistent.

---

## 10. Why Retrieval Is Over Anchors Only — Not Content

The retrieval index contains anchor embeddings. When a query arrives, it is embedded and compared against anchor embeddings — not the full fact content.

This is a deliberate constraint, not a simplification. If retrieval were over fact content embeddings, a lazy agent could store facts verbatim, skip anchor authoring entirely, and rely on the content's own embedding to match queries. The anchor would become vestigial.

By indexing anchors only, retrieval quality is a direct function of anchor quality. A bad anchor — verbatim fact text — fails to match query vocabulary. A good anchor — one that bridges abbreviations and specific-to-categorical mappings — succeeds. The gradient flows cleanly: good anchor → successful retrieval → correct answer → reward.

The content is stored separately and returned only after the anchor retrieval step succeeds. The agent never benefits from content-level similarity during the retrieval ranking.

---

## 11. Why Lexical Mismatch Is Baked Into the Data

Two systematic mismatches are built into every L3+ episode by construction:

**Mismatch 1: Abbreviation/expansion.** Facts use abbreviated technical vocabulary (LR, val_acc, MoE, flash attn, ppl). Queries use full forms (learning rate, validation accuracy, mixture of experts, flash attention, perplexity). Every architecture, hyperparameter, and metric in the vocabulary files has both forms.

**Mismatch 2: Specific-to-categorical.** Facts mention specific values ("grad norm 47", "LR=3e-4"). Queries ask categorically ("gradient instability", "what learning rate"). Bridging this requires domain knowledge — the agent must know that norm 47 is high enough to indicate instability.

These mismatches are baked into the vocabulary and template design, not added as post-processing. This is important: the mismatch must be _systematic_, not random. A systematic mismatch can be learned. A random mismatch cannot. The agent learns the abbreviation expansion table and the specific-to-categorical mapping from reward — not from any explicit supervision.

Without these mismatches, the task collapses to a selection problem (which facts to store) with no anchor authoring component. The anchor authoring skill is only necessary when the storage-time and query-time vocabularies diverge.

---

## 12. Why Templated Data — Not LLM-Generated

Three alternatives were considered for data generation: pure templates, LLM-generated content, and a hybrid. We chose templates with Haiku-generated vocabularies (the hybrid).

**Why not pure LLM generation?** LLM-generated fact streams are non-deterministic. The same seed and config would produce different episodes across runs. Ground truth construction becomes probabilistic rather than mechanical. Debugging failures is much harder when the data itself is a black box. Reproducibility is a hard requirement — same `(difficulty, seed)` must always produce the same episode.

**Why not pure hand-written templates?** Hand-writing vocabularies of 80 architectures, 40 hyperparameter types, 60 plausible paper titles, and 50 debug scenarios is weeks of work. Haiku generates the vocabulary content; humans write the templates that impose structure on how vocabulary items combine. This is the right division: templates ensure structural correctness and mismatch preservation; generated vocabularies provide lexical variety.

**Why this matters for training.** Templated answers produce exact-match-after-normalization grading. There is no ambiguity about what the correct answer is — it is extracted mechanically from the same template that generated the fact. This removes a major source of reward noise. If answer grading were fuzzy (LLM judge), training reward would be stochastic in ways unrelated to the agent's memory policy, degrading the gradient signal.

---

## 13. Why the Domain Is a PhD Student's Lab Notebook

The domain was chosen to satisfy four constraints simultaneously:

1. **Technical richness**: facts span experiments, decisions, hypotheses, debugging notes, papers — covering all query types naturally
2. **Plausible distractors**: lab life generates genuine noise (scheduling, coffee machine, advisor meetings) that is semantically unrelated to the technical content but textually similar in surface form
3. **Lexical mismatch density**: experimental vocabulary is full of abbreviations, domain jargon, and specific values that naturally mismatch query language
4. **Coherence over time**: a research project has temporal structure — facts from week 1 can be superseded by facts from week 3, which is necessary for L4/L5 contradiction scenarios

Other domains considered: medical records (too sensitive, legal issues for vocabulary), software engineering (similar to lab notebook but less obvious temporal structure), financial analysis (good but harder to generate plausible corrections). The PhD notebook domain was the cleanest fit.

---

## 14. Why Memory Tagging Is L3 and Not Earlier

Tags (`factual`, `temporal`, `relational`, `identity`, `procedural`) are introduced at L3 for a specific reason: they are only necessary once retrieval precision matters, and retrieval precision only matters once the memory budget is tight and the fact stream is diverse.

At L1, the budget is loose (8 slots, 10 facts) and fact types are homogeneous. A tag filter over 8 items adds no retrieval benefit. At L2, diversity increases but the explicit `[IMPORTANT]` tags already provide an importance signal — the agent does not need category discrimination yet.

At L3, `[IMPORTANT]` tags are removed (by design), the budget is 25 out of 50 facts, and fact types span all five categories. A temporal query for a scheduling fact can fail if the retrieval space is contaminated by identity facts that happened to use similar vocabulary. Tag-filtered retrieval solves this. Without tags, the agent has to find a precision-improving signal through anchor authoring alone, which is harder.

Introducing tagging at L3 rather than L4 or L5 also means the skill is learned before the more complex permanence and overwrite mechanics are introduced. Tags become a stable foundation the agent can rely on when navigating L4's split budget and L5's contradiction scenarios.

Tags are not directly rewarded. The agent learns to tag correctly because correct tagging improves retrieval, which improves answer accuracy, which improves reward. No labelled supervision for tagging is needed or used.

---

## 15. Why Permanence Levels Are L4

Permanence (`core` versus `working`) introduces a temporal dimension to storage decisions that does not exist at L1–L3. It is not enough to know that a fact is important — the agent must predict whether it will still be needed _late in the episode_, after many other facts have been ingested and working memory slots have cycled through.

This is a strictly harder skill than L3's importance estimation. L3 requires: "is this fact likely to be queried?" L4 requires: "is this fact likely to be queried _after_ 50+ more facts have been ingested?" The temporal horizon extends.

The curriculum structure that makes L4 learnable: critical facts appear in the first third of the stream; queries targeting those facts arrive in the final third. The agent experiences the failure mode directly — it stores a critical early fact as `working`, the working pool fills, the fact is evicted, the late query returns no relevant results. The reward drops below the baseline. The gradient shapes the policy toward marking early-arriving high-importance facts as `core`.

The core budget is intentionally limited (10 slots). The agent cannot mark everything as core. It must be selective. This preserves the training pressure that makes the skill necessary.

---

## 16. Why the Overwrite Action Is L5 and Not Before

The `overwrite` action requires the agent to:

1. Recognise that a new fact contradicts an existing memory
2. Identify the correct target slot by matching the new fact against existing anchors
3. Decide to overwrite rather than store a new item or skip
4. Author a new anchor for the updated content
5. Resist adversarial "corrections" that look like genuine supersessions but are not

This is a multi-component skill that depends on all prior skills being stable. The agent must be able to tag facts correctly (L3) to anchor well enough that target identification is possible. It must be able to manage core/working budgets (L4) to understand why budget-neutral overwrite is sometimes preferable to a new store. Only with those skills stable does teaching contradiction handling produce clean gradient signal rather than tangled credit assignment.

There is also an action space expansion risk. Adding a new action type mid-curriculum can destabilise previously learned behaviour if the new action interferes with existing decision patterns. By placing `overwrite` at L5, the final level, we contain the risk. If L5 training is unstable, the L1–L4 checkpoint is unaffected.

L5 also introduces the adversarial contradiction scenario (5% of contradictions are false). The agent must learn to discriminate between genuine supersession and adversarial manipulation. This discrimination depends on syntactic features that distinguish sweeping false corrections ("all previous results were invalid") from specific genuine ones ("run 7 actually achieved val_acc=0.634, not 0.612"). The agent learns this discrimination from reward alone.

---

## 17. Why Strengthening Is Passive — Not an Action

An alternative design would give the agent an explicit `reinforce(slot_id)` action to boost the retrieval priority of an existing memory. We chose instead to make strengthening an automatic backend mechanism triggered by semantic similarity.

Three reasons.

**Turn budget.** An explicit reinforce action costs turns. The GRPO turn budget is tight. An agent that learns to reinforce important items would use those turns on every episode, pushing toward the 7+ turn danger zone.

**Credit assignment.** If the agent explicitly reinforces a memory, the reward gradient must attribute credit for later successful retrieval back to the reinforce action, which may have occurred several turns earlier. This is a long credit assignment chain. Passive strengthening avoids this — the mechanism is always active, requires no action, and the policy does not control it.

**Emergence story.** Passive strengthening means the agent does not know it is being reinforced. It writes an anchor, that anchor gets strengthened when semantically similar facts arrive, and retrieval becomes easier for that item over time. The agent discovers this indirectly — it observes that frequently-reinforced anchors retrieve more reliably, and adjusts its authoring strategy accordingly. The result is an emergent behaviour that mirrors spaced repetition without ever being explicitly programmed. That is a better research story than a designed mechanism.

The strength cap at 3.0 prevents a single frequently-occurring concept from dominating all retrieval regardless of relevance.

---

## 18. Why FIFO Is Precomputed at Reset

The binary reward compares the agent's accuracy on an episode against the FIFO baseline's accuracy on the same episode. The FIFO baseline must be computed on the same seed — not an average over seeds, not a cached historical value. Same seed, same fact stream, same queries.

Computing this during the reward function call would mean running a FIFO simulation on every reward evaluation. This is fast (no LLM involved — FIFO is a deterministic algorithm), but it would add latency inside the reward loop where latency is most costly.

Precomputing at `reset()` time moves the FIFO simulation outside the training hot path. The result is stored in `state.baseline_correct` and available instantly when the reward function needs it. The computational cost is the same; the timing is better.

This also makes the reward function stateless with respect to the environment — it receives everything it needs as arguments and does not need to query the environment during computation. This simplifies testing and makes the reward logic independently verifiable.

---

## 19. Why Per-Session State Isolation Is Non-Negotiable

GRPO with `num_generations=8` opens 8 simultaneous WebSocket connections to the environment server. If any mutable state is shared between connections — at the class level, at the module level, in a shared cache — then completion A in one session will corrupt the episode state seen by completion B in another session. The training data becomes non-IID in a way that is invisible to the trainer. The reward signal is corrupted. The gradient is corrupted. Training will appear to proceed normally while producing a policy that has learned from a broken data distribution.

The requirements:

- No class-level mutable attributes in `RecallEnvironment`
- All state in instance attributes: `self.memory`, `self._state`, `self.rng`, `self.facts`, `self.queries`
- `app.py` uses the factory pattern — `create_app(env_factory, ...)` where `env_factory` creates a fresh instance per session
- `max_concurrent_envs: 8` in `openenv.yaml` — if this is set lower, sessions queue instead of parallelising, breaking the GRPO group sampling

This is the kind of bug that passes every unit test and manifests only as mysteriously slow convergence or unexplained reward variance. The factory pattern prevents it structurally.

---

## 20. Why the Delete Action Was Removed

The prior design included a `delete` action that allowed the agent to remove an existing memory item during ingestion to free budget for a more important incoming fact.

This was removed when ingestion moved to single-pass. The reason: with batched ingestion, the agent ingested 8 facts at a time and could observe consequences of earlier decisions before making later ones. In that context, a delete action allowed recovery from early mistakes.

With single-pass ingestion, the agent sees all 50 facts simultaneously and makes all 50 decisions in a single JSON output. There is no opportunity to observe consequences and recover. A delete action in single-pass ingestion would require the agent to simultaneously decide: which N facts to store, which M existing items to delete, and which replacement items to store. This explodes the action space and makes the decision problem significantly harder without adding any new learnable cognitive skill.

The permanence mechanism at L4 captures what delete was trying to solve — the agent must think about which items are expendable — but in a cleaner way that is compatible with single-pass ingestion.

---

## 21. Why the Action Parser Has This Fallback Sequence

The parser attempts:

1. Strict JSON parse of the raw completion
2. If that fails: extract the first `[...]` block via regex and parse that
3. If that fails: malformed action, apply penalty

**Why not be stricter?** A policy that generates a valid JSON array with extra whitespace or a markdown code fence is not making a wrong decision — it is formatting its output slightly wrong. Penalising formatting errors at the same rate as genuinely wrong decisions confuses the gradient. The agent learns to fix formatting instead of learning to make better storage decisions.

**Why not be more forgiving?** If the parser accepts highly malformed output — truncated JSON, wrong types, missing required fields — the agent never learns that clean output is required. In deployment, a downstream system expecting valid JSON will receive garbage. The parser needs to enforce the contract.

The regex extraction fallback catches the most common failure mode: the model wrapping its JSON in markdown code fences (`json ... `). This is a formatting habit from instruction tuning that does not reflect a decision error. Stripping code fences and retrying is the right response.

The malformed penalty is sharp (-0.5 per malformed action) specifically to make format errors costly relative to correct decisions. Three consecutive malformed actions terminate the episode at -1.0. This prevents a degenerate policy from avoiding difficult decisions by spamming malformed output.

---

## 22. Why We Use Qwen2.5-3B and Not a Larger Model

**Compute constraint.** Training on Colab Pro A100 with GRPO and `num_generations=8` at 3B parameters is feasible within the hackathon time budget. At 7B, training time roughly doubles and memory headroom becomes tight for the long ingest prompts (4096 token max at L3).

**The research question does not require a larger model.** We are not trying to show that a big model can memorise better. We are trying to show that RL training improves memory policy quality over the same model without RL training. The comparison that matters is:

- Qwen2.5-3B + LoRA + GRPO (trained RECALL policy)
- Qwen2.5-3B + no RL (LLM-as-judge baseline)

Both use the same base model. The performance difference, if it exists, is attributable entirely to the RL training. A larger model would make this comparison noisier and the training more expensive.

MemSearcher (Yuan et al., arXiv:2511.02805, Nov 2025) demonstrated that a 3B GRPO-trained model can outperform 7B baselines on memory management tasks when the training objective is well-aligned with the task. This is relevant precedent.

---

## 23. How RECALL Differs From Adjacent Work

**MemGPT / Letta**: Uses hardcoded OS-style paging rules for memory management. The storage policy is designed by humans. RECALL trains the storage policy end-to-end via RL.

**GraphRAG / A-MEM**: Heuristic graph construction from text. No RL training. Storage structure is predetermined. RECALL learns what to store and how to structure retrieval anchors from reward.

**MemSearcher (Yuan et al., 2025)**: Trains LLM agents to manage memory during search using GRPO. Uses the same base model family and training framework. The key difference: MemSearcher trains a memory _update_ policy (what to keep in a running memory buffer during search). RECALL trains a memory _authoring_ policy (how to write anchors that make storage retrievable). MemSearcher's memory is updated turn-by-turn; RECALL's anchor authoring happens in a single pass under query uncertainty. Different skills, adjacent problem.

**Compressive Transformer / AutoCompressor**: Architectural changes to the model's internal attention mechanism. RECALL wraps a frozen base LLM — no architectural modification. The trained policy is a separable artifact deployable with any base model.

**RLM (Recursive Language Models)**: Addresses the read-side problem — how to decompose long inputs for attention. RECALL addresses the write-side and lifetime problem — how to encode information for later retrieval under budget constraints. These are complementary: RLM handles what to attend to now; RECALL handles what to remember across time.

**Mem-α (2025)**: Closest in spirit — trains memory construction via RL with a core/episodic/semantic architecture. Key difference: Mem-α operates over conversational turns; RECALL operates over a fact stream with unknown future queries. Mem-α's retrieval is over full conversation chunks; RECALL's retrieval is over learned anchors specifically designed to bridge vocabulary mismatch.

---

## 24. What We Would Change With More Time

These are not regrets — every decision above was made for a reason under real constraints. But if the project continued past the hackathon, the priority list is:

**1. Compression as a learned action.** Instead of `store(anchor, content)` with full content, a `compress_store(anchor, summary)` action where the agent writes a compressed summary of multiple related facts into one slot. This directly addresses the embedding space crowding problem and makes the most efficient use of the memory budget. It was excluded because it requires the agent to generate both an anchor and a summary simultaneously, doubling the output complexity at ingest time.

**2. Graph edges between memory items.** When two stored facts are semantically related, an edge between them enables multi-hop retrieval — "what learning rate was used with the architecture that achieved the best validation accuracy?" This requires traversal rather than nearest-neighbour search. The data templates at L3+ already generate relational facts that beg for this structure.

**3. Longer context evaluation.** The current max is 120 facts at L5. Real agent deployments accumulate thousands of facts over weeks. The curriculum does not yet test whether the trained policy transfers to longer streams.

**4. Cross-domain transfer evaluation.** The training domain is a PhD lab notebook. Does the trained anchor authoring policy transfer to medical records? Legal documents? Software engineering notes? The hypothesis in section 1 is that the underlying skills are domain-agnostic. This needs empirical validation.

**5. Interpretability analysis of learned anchors.** After training, what do the anchors look like? Do they systematically expand abbreviations? Do they introduce categorical descriptors for specific values? Do any non-human-readable patterns emerge? This analysis would either confirm the anchor authoring hypothesis or reveal that the agent found a shortcut we did not anticipate.

---

_Last updated: 2026-04-26. Maintained alongside the spec docs in `/docs/`. When a design decision changes, update this document with the old rationale and the new one._

## Citation

---

_Built for the OpenEnv Hackathon 2026. If you are training a memory-constrained agent and want to test your policy against this environment, the HF Space accepts standard OpenEnv `reset()` / `step()` calls at the link above._