ashish-doing/supply-chain-env-hf

Supply Chain Disruption Environment

OpenEnv-compatible RL environment — an AI agent manages real-world supply chain crises across 6 task types, an infinite procedural task pool, adversarial market dynamics, quality control gates, and a reward signal engineered for clean RL training.

Live Space: HuggingFace | API Docs: Interactive Swagger UI | Live Demo: Watch agent solve a task

Why this environment

Most RL environments for LLMs are either too simple (toy grids, word games) or too narrow (one task type, fixed scenarios). This environment is designed to train agents that can reason under pressure in logistics — a domain with real economic stakes, multi-step dependencies, and adversarial elements.

Pre-submission Checklist

python validate.py
# Result: 118/118 checks passed — STATUS: READY TO SUBMIT ✓

What the Agent Does

The agent plays a warehouse manager. Each step it calls exactly one tool, receives a text observation and a reward score, and must reach the goal before running out of steps (max 25).

The agent must reason across multiple steps: diagnose the situation with read-only tools, then act decisively. Calling the wrong supplier, ignoring defect rates, or missing the competitor countdown all lead to partial or zero reward.

Episode Lifecycle

+------------------------------------------------------------+
|                     EPISODE LIFECYCLE                      |
+------------------------------------------------------------+
|                                                            |
|  POST /reset?task_id=N                                     |
|        |                                                   |
|        v                                                   |
|  +------------+     +----------------------------------+   |
|  |    Task    |---->| Observation: inventory,          |   |
|  |   Loaded   |     | suppliers, goal, shipments,      |   |
|  +------------+     | budget, countdown                |   |
|                     +----------------------------------+   |
|                                  |                         |
|                     +------------v------------------+      |
|                     |        Agent (LLM)            |      |
|                     | reads observation, picks tool |      |
|                     +------------+------------------+      |
|                                  |                         |
|              POST /step  {"tool":..., "args":...}          |
|                                  |                         |
|                     +------------v------------------+      |
|                     |       Environment             |      |
|                     | executes tool, updates state  |      |
|                     | computes layered reward       |      |
|                     +------------+------------------+      |
|                                  |                         |
|                     +------------v------------------+      |
|                     | Observation + reward + done   |      |
|                     +------------+------------------+      |
|                                  |                         |
|       done=false <---------------+----------> done=true    |
|       (next step)                           (episode ends) |
|                                                            |
|                    [END] log emitted                       |
|                    final score recorded                    |
+------------------------------------------------------------+

Task Types & Rewards

Easy — Tasks 0, 1, 2 · reorder

Scenario: Single product, healthy suppliers, stock running low.

Goal: Check inventory → verify supplier → place reorder.

Medium — Tasks 5, 6 · reroute / demand_spike

Scenario: Primary supplier failed OR demand spiked 3×. Shipment is stranded.

Goal: Identify failure → reroute shipment → place emergency order.

Medium — Task 7 · price_negotiation

Scenario: Critical shortage. 3 suppliers with different prices. Budget cap enforced.

Goal: Use get_market_prices → find cheapest healthy supplier → order within budget.

Hard — Task 10 · multi_product_crisis

Scenario: Three products critically low simultaneously, budget constraint, one supplier failed.

Goal: Assess all suppliers → reroute stranded shipment → order ALL products within budget.

Hard — Task 11 · port_strike

Scenario: All overseas suppliers on strike. Shipment stuck at port. Only one domestic supplier available.

Goal: Assess supplier statuses → cancel the stuck shipment → order all products from the only available supplier.

Hard — Task 12 · quality_control

Scenario: Defective supplier batches detected in a medical supply chain. SupplierA has a 35% defect rate — unacceptable. Only suppliers with defect rate below the threshold may be used.

Goal: Use get_quality_report → cancel defective shipment → order ONLY from compliant suppliers.

Quality gate: Orders placed with suppliers whose defect rate exceeds the threshold are hard-rejected with an Order REJECTED response. The agent must call get_quality_report to identify compliant suppliers first.

Hard — Task 13 · competing_buyer (Adversarial)

Scenario: RivalCorp is bidding for the same limited semiconductor supply. The competitor will lock up capacity in ~6 steps. Time pressure is real — every step counts.

Goal: Use get_competing_bids → secure product before competitor → also order second product.

Countdown mechanic: Every step, competing_bids_countdown in the state dict decrements. When it reaches zero, the competitor places their order and reduces remaining_capacity on the supplier — possibly making the goal unreachable.

Task Summary

Novel Mechanics

Hard quality gate (Task 12)

Agent calls place_order(SupplierA, ...)
         |
         v
  Environment checks defect rate
         |
   defect_rate > threshold?
    YES ──────────────────> "Order REJECTED" returned
    |                       reward stays at 0.50
    |                       agent must re-route
    NO
    |
    v
  Order accepted -> reward 0.75-0.99

Orders from suppliers with defect rate above the threshold are rejected server-side. There is no way to guess around it — the agent must call get_quality_report first. This forces tool sequencing, not random exploration.

Adversarial time pressure (Task 13)

Step 1:  competing_bids_countdown = 6  <- agent sees this
Step 2:  competing_bids_countdown = 5
Step 3:  competing_bids_countdown = 4
  ...
Step 6:  competing_bids_countdown = 0
         --> RivalCorp locks capacity
         --> remaining_capacity reduced
         --> goal may become unreachable

A competitor agent decrements competing_bids_countdown every step. When it hits zero, capacity locks. The agent must act before it has finished reasoning. This tests urgency under uncertainty, not just correctness.

Reward Structure

Why this reward structure works for GRPO

GRPO needs dense, shaped rewards — not sparse 0/1 signals. This environment provides:

• —Strict open interval: all rewards are in (0.01, 0.99) — 0.0 and 1.0 are never returned
• —Non-zero floor: reset returns 0.01; any action scores ≥ 0.05, so no zero-gradient episodes
• —Diagnostic layer: read-only tools earn up to 0.25, rewarding exploration before acting
• —Meaningful gradient: 0.05 → 0.25 → 0.50 → 0.65–0.80 → 0.99 are all reachable and distinguishable
• —Penalty pressure: spam penalty at −0.02/excess call forces the agent to reason between steps

The result: every rollout produces a usable training signal from step 1.

Reward layers

0.01  ---- reset floor (REWARD_MIN — strictly > 0.0)
0.05  ---- participation floor (any tool called)
  |
0.25  ---- diagnostic ceiling (read-only tools: inventory, supplier checks, etc.)
  |
0.50  ---- any order placed / shipment rerouted / cancelled
  |
0.65  ---- medium task: correct order placed (no reroute yet)
0.75  ---- key sub-goal met (correct reroute or cancel)
0.80  ---- quality_control: checked report before ordering
0.80  ---- competing_buyer: ordered before competitor deadline
  |
0.99  ---- ALL task objectives satisfied (REWARD_MAX — strictly < 1.0)
            efficiency bonuses computed internally for done-triggering only,
            clamped to 0.99 before returning to caller

Spam penalty: Calling the same tool more than 2 times total applies a reward penalty of −0.02 per excess call, floor at REWARD_MIN (0.01).

Efficiency bonuses: Fast solves (< 15 steps) and budget-under-spend earn internal bonuses that push the score above 1.0 — this triggers done=True correctly, but the value is clamped to 0.99 before being returned to the caller.

Baseline Agent Results

Scores from Qwen/Qwen2.5-7B-Instruct on all 10 fixed tasks against the live HF Space:

Tasks score between 0.75 and 0.99 — solvable but not trivially solvable. The ideal difficulty range for an RL training environment.

Procedural Task Generation

In addition to the 10 fixed tasks, the environment generates an infinite pool of deterministic tasks from any integer task ID.

Task ID space

0   ------  13   Fixed tasks (backward compatible)
14  ------  49   Easy   | reorder
50  ------  99   Medium | reroute (even IDs) / demand_spike (odd IDs)
100 ------ 149   Medium | price_negotiation
150 ------ 199   Hard   | multi_product_crisis
200 ------ 249   Hard   | quality_control
250 ------ 299   Hard   | competing_buyer
300+       inf   Hard   | cycles all hard types → infinite

Determinism guarantee: generate_task(task_id) always returns the same task for the same ID. The RNG seed is the task_id itself — training runs are fully reproducible. An optional seed parameter overrides randomness for exact reproduction.

from supply_chain_env.generate_tasks import generate_task

task_a = generate_task(175)   # hard multi_product_crisis, always identical
task_b = generate_task(42)    # easy reorder, always identical
task_c = generate_task(220)   # hard quality_control, always identical

# Use in training loop:
for episode in range(1000):
    obs = env.reset(task_id=episode % 300)   # cycles through all types

Available Tools

API Endpoints

Note on `/state`: HTTP is stateless — each call returns a fresh env state dict. For true stateful interaction across multiple steps, use the WebSocket /ws endpoint or the /reset → /step pattern.

WebSocket Usage (/ws)

// Reset
{"action": "reset", "task_id": 0, "seed": 42}

// Step
{"action": "step", "tool": "get_inventory", "args": {}}
{"action": "step", "tool": "place_order",
 "args": {"supplier_name": "SupplierA", "product": "bottled_water", "quantity": 200}}

// Get state
{"action": "state"}

The WebSocket connection persists across many reset/step calls — no reconnect needed.

Action Format

{
  "tool": "place_order",
  "args": {
    "supplier_name": "SupplierA",
    "product": "bottled_water",
    "quantity": 200
  }
}

Observation Format

{
  "text": "SUCCESS: Order placed! Supplier: SupplierA ...",
  "state": {
    "task_id": 0,
    "task_type": "reorder",
    "difficulty": "easy",
    "goal_description": "Warehouse stocks bottled water...",
    "inventory": {"bottled_water": 50},
    "steps": 2,
    "max_steps": 25,
    "orders_placed": [],
    "shipments_rerouted": [],
    "shipments_cancelled": [],
    "pending_shipments": [],
    "spent_budget": 0.0,
    "remaining_budget": 2000.0,
    "competing_bids_countdown": {"semiconductor": 5}
  },
  "reward": 0.5,
  "done": false
}

The state dict always contains all 13 keys. remaining_budget is present on tasks with a budget constraint. competing_bids_countdown is only populated for competing_buyer tasks.

Reward range: All reward values are in the strict open interval (0.01, 0.99). The environment never returns 0.0 or 1.0 to callers.

Quick Start

# Clone the repo
git clone https://github.com/ashish-doing/openenv-supply-chain.git
cd openenv-supply-chain

# Install uv if you don't have it
pip install uv

# Create venv and install dependencies
uv venv
uv pip install -e . --link-mode=copy

# Run the server
uv run server
# Server starts at http://localhost:7860
# Swagger UI at http://localhost:7860/docs

# Run tests
pytest tests/ -v   # runs 36 tests

Running Inference

export HF_TOKEN=your_hf_token
export API_BASE_URL=https://router.huggingface.co/v1
export MODEL_NAME=Qwen/Qwen2.5-7B-Instruct
export ENV_BASE_URL=https://ashish-doing-supply-chain-env-hf.hf.space
python inference.py

Use Qwen/Qwen2.5-7B-Instruct or smaller for inference — the 72B model may exceed the 20-minute runtime limit on constrained hardware. The 7B model completes all 10 tasks in under 6 minutes.

Pre-submission Validation

python validate.py
# Result: 118/118 checks passed — STATUS: READY TO SUBMIT ✓

Repository Structure

openenv-supply-chain/
├── supply_chain_env/                       # Python package (importable)
│   ├── __init__.py
│   ├── models.py                           # Pydantic typed models (Action, Observation, State)
│   ├── generate_tasks.py                   # Procedural task generator (fixed + infinite pool)
│   └── server/
│       ├── __init__.py
│       ├── app.py                          # FastAPI server — OpenEnv endpoints + WebSocket
│       └── supply_chain_env_environment.py # Environment logic (all 6 task types, v4.7)
├── tests/
│   ├── __init__.py
│   └── test_environment.py                 # pytest suite (36 tests)
├── client.py                               # Python client for training code
├── inference.py                            # Baseline LLM agent — emits [START]/[STEP]/[END] logs
├── validate.py                             # Pre-submission validator
├── openenv.yaml                            # OpenEnv spec config
├── Dockerfile                              # Container definition
└── pyproject.toml                          # Package config (entry point: uv run server)

What makes this environment hard for agents

Changelog

v4.7 (current)

• —FIX v4.7: Phase 2 validator requires strict open interval — reward values 0.0 and 1.0 are now never returned.
• —REWARD_MIN = 0.01 (was 0.0) — reset() now returns 0.01 instead of 0.0.
• —REWARD_MAX = 0.99 (was 1.0) — all goals met returns 0.99 instead of 1.0.
• —Spam penalty floor raised to REWARD_MIN (0.01).
• —done still triggers on internal score ≥ 1.0 (pre-clamp) — efficiency bonuses fire correctly.
• —openenv.yaml reward_range updated to [0.01, 0.99].
• —Version bumped to 4.7.0.

v4.6

• —Added diagnostic reward layer (read-only tools earn up to 0.25 before any action).
• —Sub-goal scores revised: quality_control quality-check-before-order → 0.80; competing_buyer order-before-deadline → 0.80; medium order-only → 0.65.
• —openenv.yaml reward shaping updated to reflect new diagnostic and sub-goal layers.

v4.5

• —Added WebSocket /ws endpoint for persistent stateful sessions.
• —Added /quick/reset and /quick/step HTTP convenience endpoints with Swagger examples.
• —inference.py updated to v4.5: /reset now sends task_id + seed as JSON body (not query params); EPISODE_SEED constant defined; 402 credits-depleted detection added; /step body sends action nested and flat for compatibility.

v4.4

• —FIX: Reward hard-capped at 1.0 in step() to comply with hackathon requirement (scores/reward in 0.0–1.0 range). Efficiency bonuses are computed internally and used for done-triggering but do not appear in the returned reward value.
• —openenv.yaml reward_range updated to [0.0, 1.0].
• —openenv.yaml app and entrypoint paths corrected to supply_chain_env.server.app:app / supply_chain_env.server.app:main.
• —validate.py section 12 thresholds updated to match reward cap.
• —Version bumped to 4.4.0.

v4.3

• —Step efficiency bonus formula changed to 0.15*(15-step)/15.
• —min(1.0, ...) cap removed from efficiency/budget bonuses (now handled in v4.4 step()).
• —Fixed absolute package imports in app.py — supply_chain_env.* imports work reliably in all environments (local + HuggingFace). Old relative/fallback chain failed on HF Spaces.
• —Fixed GET /state — no longer returns stale singleton state; returns fresh env state with session guidance.

v4.2

• —Corrected [START]/[STEP]/[END] log format in inference.py.
• —Restructured into proper Python package for clean installs via uv run server.

v4.1

• —Added GET /state endpoint (was in openenv.yaml but missing — returned 404).
• —Fixed _medium_demand_spike() — was incorrectly reusing _medium_reroute() with type renamed, producing tasks with no demand_spike flag and wrong goal structure.
• —Fixed deadlock in /quick/reset — no longer calls localhost:7860; uses direct instantiation.

v4

• —Restructured into proper Python package (supply_chain_env/) for clean installs via uv run server.
• —Procedural task generator: any integer task ID now produces a valid deterministic task.
• —Added _tool_call_log, spam penalty, step efficiency bonus, and budget efficiency bonus.
• —Validator expanded to 118 checks across multiple sections.
• —State dict standardised to 13 fields; goal_description and task_type added.
• —inference.py updated to emit exact [START]/[STEP]/[END] judge log format.
• —Default inference model changed to Qwen/Qwen2.5-7B-Instruct for runtime compliance.
• —Added tests/ folder with 36-test pytest suite.

v3

• —Added quality_control and competing_buyer task types.
• —Added get_quality_report, get_market_prices, get_competing_bids tools.
• —Competing buyer countdown mechanic and quality gate enforcement.

v2

• —Added price_negotiation task type and get_market_prices tool.
• —Max steps raised from 15 to 25.
• —Budget-efficiency bonus for hard tasks.
• —State dict includes remaining_budget and spent_budget.

v1

• —Initial release: reorder, reroute, demand_spike task types.
• —7 tools, 6 fixed tasks, basic layered reward structure.