megatronwanted/metahf

🔭 Science Environment

An OpenEnv environment where AI agents act as scientists — discovering the hidden mathematical laws of unknown universes through experimentation.

Overview

Science Environment places an agent in the role of an empirical scientist. It observes an unknown universe, designs experiments, forms hypotheses, revises them based on new data, and ultimately submits a mathematical law that explains the system's behavior.

All variable names are anonymous (α, β, γ, ω). The agent receives no units, no domain hints, no equation templates. It must discover the governing laws — and in the hardest task, discover that the laws themselves change.

This directly trains and evaluates capabilities that current benchmarks miss:

• —Experimental design — choosing informative inputs, not redundant ones
• —Inductive reasoning — inferring equations from numeric observations
• —Structural discovery — finding interaction terms, nonlinearities, coupled dynamics
• —Anomaly detection — recognizing when a system changes behavior (phase transitions)
• —Hypothesis revision — updating beliefs correctly when predictions fail

Tasks

🟢 The Harmonic World (harmonic_world) — Easy

Two anonymous inputs α, β. One output ω.

The agent must discover a single governing equation with a linear term and a quadratic term. The challenge is distinguishing the contributions of each variable through systematic experimentation.

Max steps: 25 Expected difficulty for frontier LLMs: Solvable in 10–15 experiments with good strategy Baseline score: ~0.880

🟡 The Drift World (drift_world) — Medium

Two anonymous inputs α, β. Two coupled outputs ω_α, ω_β.

The agent must discover that both outputs depend on both inputs simultaneously — a coupled dynamical system. The equations are nonlinear and involve multiplicative interaction terms. The key insight the agent must reach: this is not two independent equations.

Max steps: 35 Expected difficulty: Requires recognizing coupling, then fitting nonlinear terms Baseline score: ~0.885

🔴 The Broken World (broken_world) — Hard

Three anonymous inputs α, β, γ. One output ω. Hidden phase transition.

The universe obeys completely different laws depending on the value of α. Below a hidden threshold, the law is linear and one variable is irrelevant. Above the threshold, the law is nonlinear and all three variables interact.

The agent must:

1. 1.Probe α across its full range and notice the anomaly
2. 2.Identify that a threshold exists and estimate its location
3. 3.Characterize the law in each regime separately
4. 4.Discover which variables matter in which regime

This genuinely challenges frontier models. The phase transition is not labeled. The irrelevant variable in Regime 1 is a deliberate red herring.

Max steps: 45 Expected difficulty: Hard even for GPT-4o / Claude Sonnet Baseline score: ~0.300

Action Space

{
  "action_type": "run_experiment | form_hypothesis | revise_hypothesis | flag_observation | submit",

  // run_experiment — observe how the universe responds
  "inputs": {"α": 2.0, "β": -1.5},

  // form_hypothesis — commit a mathematical hypothesis
  "hypothesis_id": "h1",
  "equation": "ω = -3.7*α + 0.5*β**2",
  "equation_secondary": "ω_β = ...",  // Drift World only

  // flag_observation — record a qualitative insight
  "observation_text": "output seems quadratic in β",

  // submit — end episode with final answer
  "claimed_threshold": 6.5,          // Broken World only
  "noted_phase_transition": true,    // Broken World only
  "noted_coupling": true             // Drift World only
}

Equation Format

Equations must be Python/sympy-compatible expressions using the anonymous variable names:

• —ω = -3.7*α + 0.5*β**2 ✅
• —ω = α*(1.4 - 0.1*β) ✅
• —omega = -3.7*alpha + 0.5*beta**2 ✅ (aliases accepted)
• —force = -k*x ❌ (don't use semantic names)

Observation Space

{
  "task_id": "harmonic_world",
  "universe_name": "The Harmonic World",
  "difficulty": "easy",
  "description": "...",
  "available_inputs": ["α", "β"],
  "available_outputs": ["ω"],
  "experiment_log": [
    {"inputs": {"α": 1.0, "β": 0.0}, "outputs": {"ω": -3.68}, "step": 1},
    ...
  ],
  "hypotheses": [
    {"id": "h1", "equation": "ω = -3.7*α + 0.5*β**2", "step_formed": 5}
  ],
  "flags": ["output is negative when α is positive"],
  "steps_used": 7,
  "steps_remaining": 18,
  "last_reward": 0.05,
  "cumulative_reward": 0.21,
  "message": "Experiment result: inputs={'α': 1.0, 'β': 2.0} → outputs={'ω': -1.72}."
}

Reward Function

Rewards are shaped to provide signal throughout the episode, not just at the end.

Final Grading Breakdown

Harmonic World:

• —Predictive accuracy (R²): up to 0.65
• —Structural bonus (correct form): up to 0.15
• —Efficiency bonus (fewer experiments): up to 0.10
• —Flag/insight bonus: up to 0.10

Drift World:

• —Predictive accuracy (avg R²): up to 0.55
• —Coupling detection bonus: up to 0.15
• —Structural bonus (interaction terms): up to 0.10
• —Efficiency + flag bonuses: up to 0.20

Broken World:

• —Phase transition detection: up to 0.20
• —Threshold accuracy: up to 0.15
• —Regime 1 accuracy (R²): up to 0.25
• —Regime 2 accuracy (R²): up to 0.25
• —γ-irrelevance discovery (bonus): up to 0.05
• —Efficiency bonus: up to 0.10

All scores are deterministic and reproducible — the grader evaluates submitted equations on 200+ held-out points using R² against the true hidden law.

API Endpoints

Setup & Usage

Local (Python)

git clone <repo-url>
cd science-env
pip install -r requirements.txt

# Start server
python app.py
# Server runs at http://localhost:7860

# Run baseline agent (requires HF_TOKEN or OPENAI_API_KEY)
export HF_TOKEN=your_token
export API_BASE_URL=https://router.huggingface.co/v1
export MODEL_NAME=Qwen/Qwen2.5-72B-Instruct
python inference.py

Docker

docker build -t science-env .
docker run -p 7860:7860 \
  -e HF_TOKEN=$HF_TOKEN \
  -e API_BASE_URL=$API_BASE_URL \
  -e MODEL_NAME=$MODEL_NAME \
  science-env

Quick API test

# Reset to easy task
curl -X POST http://localhost:7860/reset \
  -H "Content-Type: application/json" \
  -d '{"task_id": "harmonic_world"}'

# Run an experiment
curl -X POST http://localhost:7860/step \
  -H "Content-Type: application/json" \
  -d '{"action_type": "run_experiment", "inputs": {"α": 2.0, "β": 3.0}}'

# Submit a hypothesis
curl -X POST http://localhost:7860/step \
  -H "Content-Type: application/json" \
  -d '{"action_type": "submit", "equation": "ω = -3.7*α + 0.5*β**2"}'

Baseline Scores

Scores from inference.py running Qwen/Qwen2.5-72B-Instruct via HuggingFace Router:

These are deliberately challenging — the environment is designed to distinguish reasoning capability levels.

Environment Design Notes

Why anonymous variables? Named variables (mass, displacement) would allow models to leverage pre-trained physics knowledge rather than demonstrating actual inductive reasoning. Anonymous variables force the agent to derive structure from data alone — the actual scientific method.

Why partial rewards on experiments? Binary end-of-episode rewards make it impossible to learn how to experiment efficiently. The information-gain signal encourages agents to probe diverse regions of the input space rather than repeating similar experiments.

Why a phase transition in the hard task? Current reasoning models commit to a frame early and optimize within it. A phase transition forces the agent to notice when its model is systematically wrong in a region, abandon the current hypothesis, and search for a structural explanation. This metacognitive skill is genuinely undertrained.

Named variables as extension: A fourth task with semantic variable names (mass, displacement, velocity) is planned as a comparative ablation. Expected result: models score significantly higher, confirming that anonymous mode tests pure inductive reasoning.

Project Structure

science-env/
├── app.py              # FastAPI server
├── environment.py      # Core OpenEnv environment
├── universes/
│   ├── base.py         # Abstract universe base class
│   ├── harmonic.py     # The Harmonic World (easy)
│   ├── drift.py        # The Drift World (medium)
│   └── broken.py       # The Broken World (hard)
├── graders/
│   └── __init__.py     # All graders + equation compiler
├── inference.py        # Baseline agent script
├── openenv.yaml        # OpenEnv metadata
├── Dockerfile
├── requirements.txt
└── README.md

License

MIT