c-varun/metaXscalar

OpsFlow Env

OpsFlow Env is a deterministic OpenEnv benchmark focused on realistic operational workflows: support triage, scheduling, and content moderation.

Why this environment

The benchmark targets practical tasks teams perform daily, with deterministic grading and dense trajectory rewards designed for reliable evaluator confidence and reproducible comparisons.

Project structure

• —inference.py - baseline inference runner and required output formatter
• —openenv.yaml - OpenEnv metadata and task definitions
• —Dockerfile - container build for local and HF deployment
• —env/environment.py - OpenEnv-compatible environment implementation
• —env/models.py - typed Pydantic action, observation, reward, and state models
• —env/tasks/ - task logic for easy/medium/hard workflows
• —env/graders.py - deterministic graders and strict open-interval scoring helper
• —env/rewards.py - dense reward shaping
• —env/fixtures/ - deterministic seeded scenarios

Action and observation spaces

Actions (OpsFlowAction)

• —route_ticket(item_id, route)
• —set_priority(item_id, priority)
• —assign_slot(item_id, slot)
• —label_content(item_id, label)
• —set_escalation(item_id, escalate)
• —submit()

Observations (OpsFlowObservation)

Each step includes:

• —task metadata (task_name, instruction, step_count, max_steps)
• —progress tracking (pending_items, completed_items, status)
• —diagnostics (last_action_error)
• —scoring fields (final_score, score_components when done)
• —available action list and item context

Tasks and difficulty mapping

• —triage_easy: classify support tickets by route and priority.
• —schedule_medium: assign meeting slots under constraints and minimize disruption.
• —moderation_hard: assign policy labels and escalation on mixed-content queues.

All scenarios are seeded and deterministic.

Deterministic grading and strict (0,1) policy

Each task has a deterministic rubric and weighted raw score. Raw score is normalized to [0,1] and projected into the strict open interval:

def strict_open_interval_score(normalized_score: float, eps: float = 1e-6) -> float:
    return min(1.0 - eps, max(eps, normalized_score))

The grader asserts 0.0 < score < 1.0 in all scoring paths.

Reward design

Dense step rewards guide trajectories independently from final score:

• —positive reward for valid actions and subgoal completion
• —small penalties for redundant or invalid actions
• —timeout penalty if max steps is reached

Setup

python -m venv .venv
.venv\Scripts\activate
pip install -r requirements.txt

Local run

Set required environment variables:

• —API_BASE_URL (optional, default: https://completions.me/api/v1)
• —MODEL_NAME (optional, default: gpt-5-mini)
• —HF_TOKEN (required, provider API token)

Run baseline inference:

python inference.py

Output format compliance:

• —one [START] line per episode
• —one [STEP] line immediately after each env.step()
• —one [END] line always emitted via finally

Docker run

docker build -t metaXscalar .
docker run --rm -p 7860:7860 metaXscalar

Note: The Dockerfile runs the OpenEnv server (server/app.py) for HuggingFace Space deployment. For local inference testing, use python inference.py directly.

Interacting with deployed Space

When deployed as Docker Space, the app exposes an OpenEnv HTTP/WebSocket server on port 7860.

Base URL format:

• —https://<your-space-subdomain>.hf.space

Useful endpoints:

• —GET /docs - interactive API docs
• —GET /health - health check
• —GET /metadata - environment metadata
• —GET /schema - action/observation/state schemas
• —POST /reset - start a fresh episode (stateless request)
• —POST /step - single step execution (stateless request)
• —GET /state - state snapshot (stateless request)
• —WS /ws - stateful session (recommended for multi-step interaction)

Example quick checks:

curl -s https://<space>.hf.space/health
curl -s https://<space>.hf.space/schema

Stateful interaction over WebSocket (/ws) is the recommended way to run full episodes:

{"type":"reset","data":{"task_name":"triage_easy","seed":11}}
{"type":"step","data":{"action_type":"route_ticket","item_id":"T-101","route":"billing"}}
{"type":"step","data":{"action_type":"set_priority","item_id":"T-101","priority":"high"}}
{"type":"step","data":{"action_type":"submit"}}

Compatibility note:

• —POST /mcp is exposed for validator compatibility and returns JSON-RPC format.

OpenEnv validation

openenv validate

Hugging Face Space deployment notes

• —deploy as a Docker Space
• —keep only primary submission space active during build
• —ensure final status is Running before submission
• —include tag openenv

Baseline reproducibility notes

• —fixed per-task seeds in inference.py
• —deterministic fixtures and scoring
• —same seed re-runs return identical final scores

Baseline performance scores

Deterministic sanity baseline (submit-immediately policy, fixed seeds):

These values satisfy the strict scoring rule 0 < score < 1 and are fully reproducible.

Troubleshooting inference

• —If python inference.py fails with a 401/403/402 API error, your token/provider quota is the issue, not the OpenEnv implementation.
• —Ensure .env (or shell env) contains:
• —HF_TOKEN=<your_provider_api_token>
• —API_BASE_URL=https://completions.me/api/v1 (or another OpenAI-compatible endpoint)
• —MODEL_NAME=gpt-5-mini (or another chat-completions model available to your token)
• —The script intentionally fails fast when LLM calls fail so submissions do not silently use non-LLM behavior.