bhoomiladia/TestTimeComputeOptimizer

Test-Time Compute Optimizer

An OpenEnv-compliant reinforcement learning environment designed to train agents to perfectly allocate reasoning "thinking" budgets using GRPO.

Models like DeepSeek-R1 use reasoning budgets (test-time compute) prior to emitting a final response. This environment incentivizes an agent to dynamically vary the max reasoning tokens based on the complexity of the query itself — allocating huge tokens for logic problems and stripping budgets away to minimal thresholds for trivial facts.

Reward Protocol

Reward is perfectly verifiable and binary, formulated as the accuracy relative to compute used: Reward = accuracy / max(0.01, compute_used / 4096.0)

The environment securely uses the HuggingFace InferenceClient to execute the user agent's ALLOCATE_BUDGET selection on the remote API without deploying expensive GPUs locally.

Agent Setup

1. Configure the .env.local

HF_TOKEN=<YOUR_API_TOKEN_FOR_HUGGING_FACE>

2. Verify OpenEnv Specification

bash validate-submission.sh "http://localhost:7860"

3. Start API Component

server

You can view OpenAPI specs at http://localhost:7860/docs while running.

4. Run Baseline Check

A baseline tester simulates alternating budgets between 50 and 500.

python inference.py