Apppublic

megatronwanted/science-env

sourceHugging Facemitupdated 5mo agoView on Hugging Face

0likes

App README

🔭 Science Environment

An OpenEnv environment where an AI agent plays the role of a scientist exploring an unknown universe. The agent can only learn the universe's laws by running experiments — there are no textbooks, no hints, and no variable names. Just data.

What Is This?

Most AI benchmarks test whether a model knows the right answer. This environment tests something different: can an agent figure out the answer from scratch?

The agent is placed in one of three "universes." Each universe has hidden mathematical laws governing how inputs relate to outputs. The agent must:

1.Design experiments (choose input values, observe outputs)
2.Spot patterns in the data
3.Form a mathematical hypothesis
4.Revise it as new evidence arrives
5.Submit the final equation when confident

There is a key twist: all variable names are anonymous (α, β, γ, ω). The agent sees no units, no labels, no domain hints — it must reason purely from numbers, just like a scientist encountering an unknown phenomenon.

The Three Universes

🟢 The Harmonic World — Easy

—Inputs: α, β | Output: ω
—Hidden law: ω = -3.7·α + 0.5·β²
—Challenge: α dominates the output so a linear model seems to work fine. The agent must deliberately probe β to discover the hidden quadratic term.
—Max steps: 25 | Baseline LLM score: ~0.36

🟡 The Drift World — Medium

—Inputs: α, β | Outputs: ωα and ωβ (two outputs simultaneously)
—Hidden laws: ω_α = α·(1.4 − 0.1·β) and ω_β = β·(0.7 + 0.08·α)
—Challenge: Both outputs depend on both inputs at the same time — the dynamics are coupled. The agent must discover this coupling and write two separate equations.
—Max steps: 35 | Baseline LLM score: ~0.00–0.35

🔴 The Broken World — Hard

—Inputs: α, β, γ | Output: ω
—Hidden laws — two different regimes:
—α < 6.5: ω = 2.1·α − 0.4·β + 1.2 (γ does nothing here)
—α ≥ 6.5: ω = 0.3·α² − 1.8·β·α + 4.5·γ (nonlinear, γ now matters)
—Challenge: The laws completely change above a hidden threshold at α=6.5. γ is a deliberate red herring in the low-α regime. The agent must detect the transition, find the threshold, and characterize both regimes separately.
—Max steps: 45 | Baseline LLM score: ~0.25

How an Episode Works

reset(task_id)          → receive initial observation, episode begins
run_experiment(inputs)  → observe ω, earn small reward based on novelty
run_experiment(inputs)  → keep exploring...
flag_observation(text)  → note a qualitative insight (+0.01)
form_hypothesis(eq)     → commit an equation, earn fit reward
revise_hypothesis(eq)   → update the equation based on new data
submit()                → end episode, receive final graded score 0.0–1.0

After 8+ experiments without a hypothesis, the environment fires adaptive hints — qualitative nudges that help without giving away the answer (e.g. "try holding α=0 and varying β to isolate its effect").

Real Example: The Broken World

reset("broken_world")
→ Inputs: α, β, γ  |  Output: ω  |  35 steps remaining

run_experiment({α:2,  β:1, γ:2}) → ω=7.24    reward=+0.03
run_experiment({α:5,  β:1, γ:2}) → ω=11.72   reward=+0.05
run_experiment({α:7,  β:1, γ:2}) → ω=11.02   reward=+0.04  ← dropped! anomaly
run_experiment({α:9,  β:1, γ:2}) → ω=16.72   reward=+0.05
run_experiment({α:3,  β:1, γ:4}) → ω=7.15    reward=+0.02  ← γ irrelevant at low α
run_experiment({α:9,  β:1, γ:4}) → ω=34.88   reward=+0.05  ← γ matters at high α!

HINT: "Behavior at high α looks structurally different. Consider whether the law changes."

flag_observation("phase transition near α=6-7")           reward=+0.01
flag_observation("γ is irrelevant below the threshold")    reward=+0.01

form_hypothesis(id="regime1", equation="ω = 2.1*α - 0.4*β + 1.2")        reward=+0.08
form_hypothesis(id="regime2", equation="ω = 0.3*α**2 - 1.8*β*α + 4.5*γ") reward=+0.07

submit(claimed_threshold=6.5, noted_phase_transition=True)
→ FINAL SCORE: 0.960  ✓

Actions Reference

Send actions as JSON to POST /step:

Action	Purpose	Key fields
`run_experiment`	Probe the universe	`inputs: {α: 1.0, β: 2.0}`
`form_hypothesis`	Record equation guess	`hypothesis_id`, `equation`
`revise_hypothesis`	Update existing guess	`hypothesis_id`, `equation`
`flag_observation`	Note qualitative insight	`observation_text`
`submit`	End episode, get graded	optional: `claimed_threshold`, `noted_phase_transition`, `noted_coupling`

Equation format — Python math syntax with anonymous variable names:

ω = -3.7*α + 0.5*β**2        ✅ correct
ω = α*(1.4 - 0.1*β)          ✅ correct
omega = -3.7*alpha + 0.5*beta**2  ✅ aliases accepted
force = -k*x                  ❌ do not use semantic names

Drift World — submit two hypotheses, one per output:

json

{"action_type":"form_hypothesis","hypothesis_id":"h_alpha","equation":"ω_α = α*(1.4 - 0.1*β)"}
{"action_type":"form_hypothesis","hypothesis_id":"h_beta", "equation":"ω_β = β*(0.7 + 0.08*α)"}
{"action_type":"submit","noted_coupling":true}

Broken World — name hypotheses regime1 and regime2:

json

{"action_type":"form_hypothesis","hypothesis_id":"regime1","equation":"ω = 2.1*α - 0.4*β + 1.2"}
{"action_type":"form_hypothesis","hypothesis_id":"regime2","equation":"ω = 0.3*α**2 - 1.8*β*α + 4.5*γ"}
{"action_type":"submit","claimed_threshold":6.5,"noted_phase_transition":true}

Observation Format

Every step returns:

json

{
  "task_id": "broken_world",
  "universe_name": "The Broken World",
  "difficulty": "hard",
  "available_inputs": ["α", "β", "γ"],
  "available_outputs": ["ω"],
  "experiment_log": [
    {"inputs": {"α": 2.0, "β": 1.0, "γ": 2.0}, "outputs": {"ω": 7.24}, "step": 1}
  ],
  "hypotheses": [
    {"id": "regime1", "equation": "ω = 2.1*α - 0.4*β + 1.2", "step_formed": 9}
  ],
  "flags": ["phase transition near α=6-7"],
  "steps_used": 9,
  "steps_remaining": 36,
  "last_reward": 0.08,
  "cumulative_reward": 0.39,
  "hint": "Reminder: if you detected a regime change, submit TWO hypotheses.",
  "message": "Hypothesis 'regime1' recorded. Quick fit score: 0.082."
}

Scoring

Step rewards (throughout episode):

Event	Reward
Informative new experiment	+0.01 to +0.05
Duplicate/redundant experiment	−0.02
Hypothesis that fits current data	0.00 to +0.10
Qualitative flag	+0.01

Final score on submit (0.0–1.0), evaluated on 400 held-out test points:

The grader uses two-stage R² to correctly separate "found the dominant term" from "found the full structure":

1.Overall fit — does your equation predict outputs at all? → up to 0.40 points
2.Nonlinear capture — how much of what a linear model misses does your equation explain? → up to 0.45 points
3.Structural and efficiency bonuses → up to 0.15 points

For The Broken World, points are split across: detecting the phase transition (0.15), threshold accuracy (0.15), regime 1 fit (0.28), regime 2 fit (0.28), discovering γ irrelevance in regime 1 (0.05), and efficiency (0.05).

All graders are deterministic and reproducible — same inputs always produce the same score.

API Endpoints

Method	Endpoint	Description
`GET`	`/health`	`{"status": "healthy"}`
`GET`	`/metadata`	Environment name and description
`GET`	`/schema`	JSON schemas for action/observation/state
`GET`	`/tasks`	All tasks with difficulty and step limits
`POST`	`/reset`	Start episode: `{"task_id": "harmonic_world"}`
`POST`	`/step`	Execute one action
`GET`	`/state`	Full internal episode state

Setup

Local:

bash

git clone <repo-url>
cd science-env
pip install -r requirements.txt
uvicorn server.app:app --host 0.0.0.0 --port 7860
# API docs at http://localhost:7860/docs

Docker:

bash

docker build -t science-env .
docker run -p 7860:7860 science-env

uv:

bash

uv sync
uv run server

Run inference:

bash

export HF_TOKEN=your_token
export API_BASE_URL=https://router.huggingface.co/v1
export MODEL_NAME=Qwen/Qwen2.5-72B-Instruct
export ENV_BASE_URL=http://localhost:7860
python inference.py

Project Structure

science-env/
├── server/                    ← Main package (all core logic lives here)
│   ├── app.py                 ← FastAPI server + main() entry point
│   ├── environment.py         ← Episode logic: reset / step / state
│   ├── universes/
│   │   ├── harmonic.py        ← The Harmonic World (easy, anonymous vars)
│   │   ├── drift.py           ← The Drift World (medium, coupled dynamics)
│   │   ├── broken.py          ← The Broken World (hard, phase transition)
│   │   └── harmonic_named.py  ← Named variables ablation task
│   └── graders/
│       └── __init__.py        ← Deterministic scoring for all three tasks
├── inference.py               ← Baseline LLM agent (OpenAI client)
├── pyproject.toml             ← [project.scripts] server = "server.app:main"
├── uv.lock                    ← Dependency lockfile
├── openenv.yaml               ← OpenEnv task metadata
├── Dockerfile                 ← Container (CMD: uvicorn server.app:app)
└── requirements.txt

Why Anonymous Variables?

Named variables like mass or velocity let models cheat — they recall equations from training data instead of reasoning from evidence. Anonymous variables (α, β, γ) remove this shortcut entirely.

The harmonic_world_named task uses semantic names (displacement, velocity, net_force) for the same hidden law as harmonic_world. The score gap between the two (~0.35 points) directly measures how much semantic scaffolding helps, and how much of a model's performance is genuine inductive reasoning vs. memorized knowledge.

License

MIT