dhruv-punia-bits/memory-compaction-openenv

Memory Compaction OpenEnv

Memory Compaction OpenEnv is an OpenEnv benchmark for a practical agent problem: how should an assistant compress a long conversation into the smallest trustworthy memory that still supports future work?

This project models a real failure mode in production assistants. Long conversations contain a mix of durable facts, temporary context, corrections, ambiguous statements, and low-value chatter. A useful agent cannot keep everything in raw context forever, but it also cannot safely summarize everything into a vague blob. It must decide:

• —what should become durable memory
• —what should remain only in a rolling summary
• —what should be marked uncertain
• —what should be superseded when the user changes their mind
• —what should be ignored completely

That is the core task simulated here.

The broader product direction behind this benchmark is to use small language models (SLMs) as a dedicated memory-compaction layer for larger LLM systems. Instead of repeatedly sending full raw chat history to a large model, an SLM can learn to compress, normalize, and maintain trustworthy memory over time. In practice, that can reduce context-window cost while also improving accuracy by preserving the right facts, applying corrections cleanly, and filtering out noise before the LLM sees the conversation state.

Why This Matters

Humans do this constantly in real workflows:

• —executive assistants track preferences, deadlines, people, and updates across long conversations
• —support agents keep durable customer context while ignoring incidental small talk
• —project managers retain commitments and corrections while discarding irrelevant details
• —copilots working over long chat histories need to preserve only the information that matters for the next task

This environment turns that real-world memory-management problem into a deterministic RL-ready benchmark.

In that sense, the environment is also a testbed for a practical architecture: SLMs manage memory, LLMs consume compact trustworthy state. The expected outcome is lower inference cost, lower context bloat, and better downstream behavior.

Submission Pitch

The novelty of this project is not “summarization” in the generic sense. The environment is specifically about memory writing under budget.

The agent is not rewarded for producing pretty summaries. It is rewarded for writing small, structured, durable memory that remains:

• —correct
• —update-aware
• —provenance-grounded
• —useful for later tasks

This is what makes the environment more realistic than a toy long-context benchmark. The hard part is not just recalling facts. The hard part is choosing the right memory operations over time:

• —append_memory
• —update_memory
• —delete_memory
• —replace_summary
• —noop

In other words, the environment evaluates whether an agent can behave like a conservative, trustworthy memory manager rather than a generic text compressor.

What Is In The Repo

• —Typed OpenEnv-style HTTP environment with POST /reset, POST /step, and GET /state
• —Additional runtime endpoints for validation and deployment: GET /health, GET /metadata, GET /schema, POST /mcp, GET /tasks
• —Thirty deterministic benchmark tasks: 10 each for easy, medium, and hard
• —Programmatic graders with rewards normalized to [0.0, 1.0]
• —Root-level inference.py using the OpenAI client and the organizer-required stdout format
• —openenv.yaml, Dockerfile, pyproject.toml, and requirements.txt for packaging and validation
• —Offline rollout generation in server/rollouts.py for later RL fine-tuning

Problem Framing

Each episode simulates a long user-assistant conversation. The conversation is revealed one turn at a time. The agent must keep enough memory to answer future hidden queries correctly, but it is penalized for storing too much, keeping stale memories active, or writing unsupported confident facts.

This is intentionally closer to real assistant behavior than a simple recall test.

The benchmark emphasizes five practical memory behaviors:

1. 1.Preserve durable facts that will matter later.
2. 2.Handle corrections safely by superseding stale memory.
3. 3.Keep ambiguous claims uncertain instead of overcommitting.
4. 4.Ignore low-value or explicitly ignorable detail.
5. 5.Stay within a memory budget.

Why The Tasks Are Real-World

The task set is built from realistic assistant-style conversations rather than game mechanics or synthetic string matching. Scenarios include:

• —travel and event planning
• —project coordination
• —deadline updates
• —standing user preferences
• —collaborator, mentor, reviewer, or owner facts
• —tentative future plans
• —distracting but irrelevant conversational details

The conversations are long-form and descriptive on purpose. The benchmark is designed to reduce overfitting to short rigid templates and better reflect the messiness of real chat.

Environment Design

Each episode has two phases.

1. Ingest phase

The environment reveals the next conversation turn. The agent updates its working summary and durable memory.

2. Evaluation phase

The environment asks hidden downstream queries. These represent future tasks the assistant should still be able to perform because it wrote useful memory earlier.

Observation Space

Each observation includes:

• —episode_id
• —difficulty
• —phase
• —current_turn
• —recent_turns
• —working_summary
• —durable_memory
• —token_budget_remaining
• —step_count
• —future_query_queue_size
• —done
• —info

The info field includes task metadata such as current budget, policy hints, ambiguous-turn IDs, and low-value-turn IDs.

Action Space

The agent submits a typed action with:

• —operation
• —memory_items
• —summary_text
• —rationale

Supported operations:

• —append_memory
• —update_memory
• —delete_memory
• —replace_summary
• —noop

Durable memory entries are normalized into the following schema:

• —memory_id
• —type
• —subject
• —predicate
• —object
• —confidence
• —source_turn_ids
• —source_text
• —status
• —updated_from_memory_id
• —expires_at
• —importance
• —task_relevance
• —requires_confirmation

This explicit schema is a core part of the project’s scope. It makes grading deterministic and makes memory quality auditable.

Task Set

The benchmark contains 30 deterministic scenarios.

Easy

10 scenarios with explicit facts and preferences. These test whether the agent can preserve directly stated durable memory without confusing it with summary-only context.

Examples:

• —user identity and travel city
• —project and teammate
• —explicit preference
• —current deadline and preferred tool

Medium

10 scenarios with changes over time. These focus on update semantics rather than simple recall.

Examples:

• —deadline corrections
• —changed user preferences
• —mentor or reviewer facts
• —tentative plans that should stay uncertain

Hard

10 scenarios with denser context and tighter budgets. These test whether the agent can remain selective and trustworthy under pressure.

Examples:

• —noisy detail mixed with durable facts
• —multiple updates across one conversation
• —ambiguous future commitments
• —low-value distractors that should not become durable memory

Canonical Baseline Tasks

For reproducibility and runtime safety, the required root inference.py evaluates a canonical 3-task subset:

• —easy: seed 1101
• —medium: seed 2201
• —hard: seed 3301

The full environment still exposes all 30 tasks through GET /tasks.

Reward Design

The reward function provides signal across the full trajectory, not only at the end.

Per-step reward combines:

• —schema validity
• —coverage of newly introduced durable information
• —precision of active durable memory
• —contradiction and stale-memory penalties
• —trust penalties for unsupported confident memories
• —budget compliance
• —efficiency of the memory store

Terminal reward evaluates:

• —recall of future-query-critical memory
• —consistency under corrections
• —trustworthiness of stored memory
• —compactness under budget

All rewards are clipped to [0.0, 1.0].

This reward design intentionally prioritizes trustworthy memory over naive compression. A slightly larger memory store is preferable to confidently storing the wrong thing.

Grading Logic

The graders are deterministic and programmatic. They do not depend on subjective judgment of summary style.

The main grading principles are:

• —reward the agent for preserving future-task-critical information
• —penalize redundant or bloated memory
• —penalize stale facts left active after corrections
• —penalize unsupported high-confidence writes
• —tolerate uncertainty when the source conversation is uncertain

This makes the environment suitable both for benchmark evaluation and for RL reward shaping.

System Philosophy

This repo intentionally separates:

• —summary writing, which can be flexible
• —durable memory writing, which should be conservative

That is why the practical baseline is hybrid:

• —the model layer handles salience, summarization, ambiguity, and borderline cases
• —deterministic logic handles explicit durable writes, normalization, provenance, and update semantics

This is a stronger real-world design than pure prompt-based memory writing, because durable memory should be harder to write than summary text.

Scope And Novelty

This project is not trying to claim that RL has already solved memory management. The contribution is more useful and more defensible:

• —a realistic environment for long-conversation memory compaction
• —structured, auditable memory objects
• —deterministic graders for future-task utility
• —dense trajectory rewards instead of only terminal pass/fail
• —a benchmark that measures not just recall, but update safety and trust

The environment is RL-ready, but it is also valuable as a benchmark for evaluating memory architectures, retrieval policies, and summarization strategies.

That is the central novelty: learning or benchmarking how to write the smallest trustworthy memory that preserves future task performance.

The intended long-term use case is especially compelling for multi-model systems: train or evaluate an SLM to act as the memory manager for a more capable but more expensive LLM. If the SLM can write compact, accurate memory reliably, the overall system can spend fewer tokens on repeated history while giving the LLM cleaner context to reason over.

API Summary

The service exposes:

• —GET /
• —GET /health
• —GET /metadata
• —GET /schema
• —GET /tasks
• —POST /reset
• —POST /step
• —GET /state
• —POST /mcp

Local Setup

Install dependencies:

pip install -r requirements.txt

Start the environment:

uvicorn server.app:app --host 0.0.0.0 --port 7860

Quick browser/API checks after startup:

curl http://127.0.0.1:7860/
curl http://127.0.0.1:7860/health
curl http://127.0.0.1:7860/tasks
curl -X POST http://127.0.0.1:7860/reset -H 'Content-Type: application/json' -d '{"difficulty":"easy","seed":1101}'

The root route exists so that a deployed Hugging Face Space opens to a clean status payload instead of a 404. For interactive API inspection, use /docs.

Run the baseline:

python inference.py

Baseline Inference Script

The root inference.py:

• —uses the OpenAI client as required by the hackathon
• —reads OPENAI_API_KEY or HF_TOKEN
• —reads API_BASE_URL
• —reads MODEL_NAME
• —emits the required [START], [STEP], and [END] logs
• —runs the canonical three-task baseline

Environment variables:

• —ENV_BASE_URL default: http://127.0.0.1:7860
• —API_BASE_URL default: https://router.huggingface.co/v1
• —MODEL_NAME default: Qwen/Qwen2.5-72B-Instruct
• —OPENAI_API_KEY or HF_TOKEN for the OpenAI client API key

If the environment is not already running, inference.py attempts to start local uvicorn automatically. In restricted sandboxes, start the server yourself and point ENV_BASE_URL at it.

Baseline Scores

Canonical 3-task baseline using the submitted hybrid policy with heuristic fallback:

• —easy seed 1101: 1.000
• —medium seed 2201: 0.859
• —hard seed 3301: 0.867

These scores are intentionally not framed as “maxed out.” They are meant to show a practical, trustworthy baseline on a realistic environment rather than an overfit policy optimized only for benchmark score.

Validation And Tests

Run tests:

pytest

Run the OpenEnv local validator:

openenv validate

Deployment

This repo is prepared for containerized deployment to a Hugging Face Space tagged with openenv.

Key files:

• —Dockerfile
• —openenv.yaml
• —pyproject.toml

After deployment, the default Space URL should return a healthy JSON status document from GET /. The main endpoints to verify on the live Space are:

• —/
• —/health
• —/reset
• —/tasks
• —/docs

Final Note

This project is best understood as an environment for memory governance, not just memory compression.

The strongest assistants will not be the ones that remember everything. They will be the ones that remember the right things, update memory safely, and stay trustworthy as conversations get longer. That is the behavior this benchmark is designed to measure.