yashu2000/TemporalBenchEnv

1---2title: TemporalBenchEnv MCQ Server3emoji: 🥁4colorFrom: yellow5colorTo: indigo6sdk: docker7pinned: false8app_port: 80009base_path: /web10tags:11  - openenv12---13 14# TemporalBenchEnv15 16OpenEnv environment for **multi-step multiple-choice** time-series reasoning. Each episode samples nine questions from pre-built JSON banks (per-dataset files or merged JSONL in `TSQuestion` shape). Rewards combine per-step correctness and an episode bonus (see `env/reward.py`).17 18## Question bank layout19 20Point the server at a directory containing `PSML_questions.json`, `freshretailnet_questions.json`, `MIMIC_questions.json`, and `causal_chambers_questions.json` (each file is a JSON array of `TSQuestion` records), or set **`TEMPORALBENCH_QUESTION_BANK_DIR`** to that path. If unset, the server uses `tests/fixtures/banks` when present (for local smoke runs).21 22Each record must include at least: `question_id`, `dataset`, `task_type` (`T1U` | `T3` | `T2_MCQ`), `prompt`, `options` (length ≥ 2), `answer`, plus optional `family`, `capability_tags`, `difficulty`, `metadata`.23 24## Quick Start25 26Use the typed client (`TemporalBenchEnvClient`; alias `TemporalbenchenvEnv`):27 28```python29from client import TemporalBenchAction, TemporalBenchEnvClient30 31try:32    env = TemporalBenchEnvClient.from_docker_image("TemporalBenchEnv-env:latest")33    out = env.reset()34    while not out.done:35        q = out.observation36        # Agent picks q.options[i] or equivalent label string37        out = env.step(TemporalBenchAction(answer=q.options[0]))38finally:39    env.close()40```41 42`TemporalBenchEnvClient.from_docker_image()` handles:43- Starting the Docker container44- Waiting for the server to be ready45- Connecting to the environment46- Container cleanup when you call `close()`47 48## Building the Docker Image49 50Before using the environment, you need to build the Docker image:51 52```bash53# From project root54docker build -t TemporalBenchEnv-env:latest -f server/Dockerfile .55```56 57## Deploying to Hugging Face Spaces58 59You can easily deploy your OpenEnv environment to Hugging Face Spaces using the `openenv push` command:60 61```bash62# From the environment directory (where openenv.yaml is located)63openenv push64 65# Or specify options66openenv push --namespace my-org --private67```68 69The `openenv push` command will:701. Validate that the directory is an OpenEnv environment (checks for `openenv.yaml`)712. Prepare a custom build for Hugging Face Docker space (enables web interface)723. Upload to Hugging Face (ensuring you're logged in)73 74### Prerequisites75 76- Authenticate with Hugging Face: The command will prompt for login if not already authenticated77 78### Options79 80- `--directory`, `-d`: Directory containing the OpenEnv environment (defaults to current directory)81- `--repo-id`, `-r`: Repository ID in format 'username/repo-name' (defaults to 'username/env-name' from openenv.yaml)82- `--base-image`, `-b`: Base Docker image to use (overrides Dockerfile FROM)83- `--private`: Deploy the space as private (default: public)84 85### Examples86 87```bash88# Push to your personal namespace (defaults to username/env-name from openenv.yaml)89openenv push90 91# Push to a specific repository92openenv push --repo-id my-org/my-env93 94# Push with a custom base image95openenv push --base-image ghcr.io/meta-pytorch/openenv-base:latest96 97# Push as a private space98openenv push --private99 100# Combine options101openenv push --repo-id my-org/my-env --base-image custom-base:latest --private102```103 104After deployment, your space will be available at:105`https://huggingface.co/spaces/<repo-id>`106 107The deployed space includes:108- **Web Interface** at `/web` - Interactive UI for exploring the environment109- **API Documentation** at `/docs` - Full OpenAPI/Swagger interface110- **Health Check** at `/health` - Container health monitoring111- **WebSocket** at `/ws` - Persistent session endpoint for low-latency interactions112 113## Environment Details114 115### Action (`TemporalBenchAction`)116- `answer` (str) — MCQ label (must match ground truth after optional normalization)117- `confidence`, `reasoning` — optional118 119### Observation (`TemporalBenchObservation`)120- `question`, `options`, `task_type`, `dataset`, `history`, `accuracy_so_far`121- `step_idx`, `steps_remaining`, `max_steps`, `done`, `reward`, `metadata`122 123### Reward124- Per step: `alpha * correctness` (correctness 0 or 1).125- On the final step, adds episode bonus: `lambda_ep * (total_correct / N) * coverage_multiplier` (1.0 if every dataset in the episode has at least one correct answer, else 0.8).126 127## Advanced Usage128 129### Connecting to an Existing Server130 131If you already have a TemporalBenchEnv server running, connect with:132 133```python134from client import TemporalBenchAction, TemporalBenchEnvClient135 136with TemporalBenchEnvClient(base_url="http://localhost:8000") as env:137    r = env.reset()138    r = env.step(TemporalBenchAction(answer=r.observation.options[0]))139```140 141Note: `close()` does not stop a remote server you attached to with `base_url=...`.142 143### Using the Context Manager144 145The client supports context manager usage for automatic connection management:146 147```python148from client import TemporalBenchAction, TemporalBenchEnvClient149 150with TemporalBenchEnvClient(base_url="http://localhost:8000") as env:151    result = env.reset()152    while not result.done:153        ans = result.observation.options[0]154        result = env.step(TemporalBenchAction(answer=ans))155```156 157The client uses WebSocket connections for:158- **Lower latency**: No HTTP connection overhead per request159- **Persistent session**: Server maintains your environment state160- **Efficient for episodes**: Better for many sequential steps161 162### Concurrent WebSocket Sessions163 164The server uses **factory mode** (`create_app(_env_factory, ...)`) so each WebSocket session gets a fresh `TemporalBenchEnvironment`. Tune `max_concurrent_envs` in `server/app.py` as needed.165 166## Development & Testing167 168### Direct environment testing169 170```bash171uv sync --extra dev172uv run pytest tests/173```174 175### Running Locally176 177Run the server locally for development:178 179```bash180uvicorn server.app:app --reload181```182 183## Project Structure184 185```186TemporalBenchEnv/187├── .dockerignore         # Docker build exclusions188├── __init__.py            # Module exports189├── README.md              # This file190├── openenv.yaml           # OpenEnv manifest191├── pyproject.toml         # Project metadata and dependencies192├── uv.lock                # Locked dependencies (generated)193├── client.py              # TemporalBenchEnvClient (alias TemporalbenchenvEnv)194├── models.py              # Action / observation / state re-exports195├── env/                   # Environment, sampler, grading, rewards196├── data/                  # TSQuestion schema + JSON/JSONL loaders197└── server/198    ├── __init__.py        # Server module exports199    ├── app.py             # FastAPI application (HTTP + WebSocket endpoints)200    └── Dockerfile         # Container image definition201```202