realambuj2001/schemaquake1

SchemaQuake

SchemaQuake is an OpenEnv-compatible reinforcement learning environment for evaluating and training LLM agents under a production failure mode: the world changes silently, and the agent keeps acting confidently.

• —Hackathon theme: #3.1 World Modeling / Professional Tasks
• —Live Space: realambuj2001/schemaquake1
• —Trained model repo: realambuj2001/schemaquake1-lora
• —Blog: Blog.md
• —Training notebook: notebooks/schemaquake_training_colab.ipynb
• —Runnable Colab: SchemaQuake training notebook
• —Full result folder: results
• —Training results summary: training-results.md

Reviewer TL;DR

SchemaQuake is an RL environment first. It tests whether an agent can keep a consistent world model when API schemas, units, refund fields, and policy text silently change mid-task. The agent sees tool observations, chooses actions, and is rewarded for safe task completion while being penalized for silent violations. Training and evaluation artifacts show the environment can produce measurable learning signals.

The Story

Imagine an AI travel assistant at work. You ask it:

Book a refundable flight from BLR to DEL under 8000 rupees.

At the start, everything looks normal. The agent can search flights, read policy, inspect the API schema, book, cancel, ask the user, and submit a final answer.

Then the environment changes without warning.

Maybe price_rupees becomes ticket_price. Maybe rupees become paise. Maybe refundable=true becomes a more subtle refund tier like full, partial, or none. Maybe the written policy changes from a 24-hour cancellation window to a 48-hour window.

A brittle agent does not necessarily crash. It may book the wrong ticket and say the task is complete.

SchemaQuake turns this into an RL environment: agents must notice suspicious changes, re-check schema or policy, avoid silent violations, and only submit when the booking still satisfies the user's real request.

Environment

SchemaQuake exposes reset, step, state, and close through an OpenEnv-compatible wrapper while sharing the same underlying environment logic used by the Gradio demo and training loop.

• —Manifest: openenv.yaml
• —OpenEnv wrapper: envs/schemaquake_env
• —Core environment: schemaquake/env/environment.py

OpenEnv is pinned to the latest PyPI-available release, openenv>=0.1.13, for reproducible Space builds.

The action space:

• —search_flights
• —read_policy
• —inspect_schema
• —book_flight
• —cancel_booking
• —ask_user
• —submit_final

The observation never directly announces drift. The agent must infer it from tool outputs.

Reward Design

SchemaQuake uses composable reward components:

• —task success reward
• —budget bonus
• —partial credit
• —drift detection reward
• —silent violation penalty
• —calibrated uncertainty reward/penalty
• —step penalty
• —probe-spam penalty

The core safety metric is silent violation rate: how often an agent confidently submits a result that violates the user's actual constraints.

Training Pipeline

The core submission is the environment: a partially observable professional task with tool actions, hidden drift, and composable rewards. The training pipeline is included as supporting evidence that the environment produces useful RL signals.

The project includes:

• —heuristic trace generation for SFT-style warm starts
• —TRL GRPO training against real SchemaQuake rollout rewards
• —optional SFT entrypoint
• —hard-mode evaluation
• —Gradio training logs and plots
• —GPU-only trained-model demo in the Space
• —Colab drift demo with an optional GPU trained-model cell

Default model:

Qwen/Qwen2.5-0.5B-Instruct

This smaller model was chosen so judges can rerun the demo on accessible hackathon hardware. The trained model artifacts were pushed to:

https://huggingface.co/realambuj2001/schemaquake1-lora

Training run completed end-to-end: heuristic traces were generated, Qwen was trained with TRL GRPO rollout rewards, and trained model artifacts were pushed to Hugging Face. These results support the environment claim by showing that SchemaQuake can drive measurable before/after behavior. The hard-mode evaluation compares base Qwen, SFT-style behavior, GRPO-style behavior, an imperfect heuristic baseline, and a heuristic upper bound.

Evidence And Results

Submission Checklist

• —Environment: openenv.yaml, envs/schemaquake_env, and schemaquake/env/environment.py
• —Training scripts: schemaquake/training/train_grpo.py, schemaquake/training/train_sft.py, schemaquake/training/traces.py, and notebooks/schemaquake_training_colab.ipynb
• —Runnable Colab: SchemaQuake training notebook
• —Training logs: results/model-training-with-push-on/trainingLogs.txt, results/model-training-with-push-on/trainingOutput.json, results/model-training-with-push-on/trainingCurve.png, and results/model-training-with-push-off
• —Results and plots: results/hard-drift-benchmark/metrics.json, results/hard-drift-benchmark/evalutationChart.png, results/baseline-evaluation-normal-mode/Metrics.json, results/baseline-evaluation-normal-mode/evalutationChart.png, and results/single-episode-trace-reward-breakdown
• —Story/writeup: Blog.md

1. Single Episode Trace

Trace: results/single-episode-trace-reward-breakdown/episodeTrace.txt Reward breakdown: results/single-episode-trace-reward-breakdown/finalReward.json

This shows the environment loop: task, actions, rewards, and final reward metadata.

2. Normal Baseline Evaluation

Metrics: results/baseline-evaluation-normal-mode/Metrics.json

Summary from the saved run:

• —random task success: 0.17
• —random silent violation rate: 0.83
• —imperfect heuristic task success: 0.90
• —imperfect heuristic silent violation rate: 0.10
• —heuristic task success: 1.00
• —heuristic silent violation rate: 0.00

3. Hard Drift Benchmark

Metrics: results/hard-drift-benchmark/metrics.json

Summary from the saved run:

• —base Qwen silent violation rate: 1.00
• —imperfect heuristic silent violation rate: 0.18
• —SFT Qwen silent violation rate: 0.16
• —GRPO-style policy silent violation rate: 0.00
• —heuristic upper bound silent violation rate: 0.00

Hard mode uses earlier drift, stricter budgets, and stronger decoy offers.

4. Training With Hub Push

Training logs: results/model-training-with-push-on/trainingLogs.txt Training output: results/model-training-with-push-on/trainingOutput.json Training curve: results/model-training-with-push-on/trainingCurve.png

The saved run used:

• —model: Qwen/Qwen2.5-0.5B-Instruct
• —curriculum: mixed
• —steps: 50
• —generated traces: 250
• —pushed trained model repo: realambuj2001/schemaquake1-lora

[image]

Judge Quickstart

Open the Space:

https://huggingface.co/spaces/realambuj2001/schemaquake1

Recommended path:

1. 1.Run Episode: choose imperfect_heuristic, seed 42, and inspect how a capable but fallible policy behaves.
2. 2.Evaluate: run normal difficulty and observe random vs imperfect heuristic vs upper-bound heuristic safety.
3. 3.Evaluate Hard: set difficulty to hard, run compare, and inspect the ladder from base Qwen to SFT-style behavior to GRPO-style behavior.
4. 4.Train: run a short mixed GRPO job and inspect live logs + curve.
5. 5.Model Demo: on a GPU Space, preview the flight dataset, inject drift, and run the pushed trained model repo against the same environment.
6. 6.Trained model repo: inspect realambuj2001/schemaquake1-lora.

Space Usage

The Space is the main judge-facing interface.

1. 1.Review the preflight panel to confirm model, CUDA, and token status.
2. 2.Use Run Episode to inspect a single trajectory. random is the weak baseline, imperfect_heuristic is the roughly 80-90% baseline, and heuristic is the upper bound.
3. 3.Use Evaluate to compare normal and hard-mode behavior. The main chart tracks task success and silent violation rate.
4. 4.Use Train on GPU hardware to run short TRL GRPO jobs. The tab streams logs and shows the reward/loss curve when the run finishes.
5. 5.Use Model Demo last when GPU is available. It loads realambuj2001/schemaquake1-lora, shows the small flight dataset, lets you choose a drift, and prints the model action trace.

CPU is enough for demo playback and evaluation. GPU is recommended for training and required for the trained-model Model Demo.

Local Development

python -m venv .venv
source .venv/bin/activate
pip install -e .[dev]
python app.py

Training

pip install -e .[training]
python -m schemaquake.training.traces --episodes 1500 --out artifacts/traces/heuristic_traces.jsonl
python -m schemaquake.training.train_sft --dataset artifacts/traces/heuristic_traces.jsonl --max-steps 200
python -m schemaquake.training.train_grpo --max-steps 200 --curriculum mixed

Space Configuration

Recommended variables:

• —SCHEMAQUAKE_MODEL_ID=Qwen/Qwen2.5-0.5B-Instruct
• —SCHEMAQUAKE_LOAD_IN_4BIT=0
• —SCHEMAQUAKE_BATCH_SIZE=2
• —SCHEMAQUAKE_NUM_GENERATIONS=2
• —SCHEMAQUAKE_GRAD_ACCUM=1
• —SCHEMAQUAKE_ENABLE_SFT=1
• —SCHEMAQUAKE_OUTPUT_REPO=realambuj2001/schemaquake1-lora
• —HF_TOKEN as a secret for uploads

Why This Matters

SchemaQuake is not trying to be just another travel assistant. The travel workflow is a readable surface for a general production-agent skill: maintaining a world model when schemas, policies, and meanings change underneath the agent.

That capability matters for enterprise APIs, billing systems, HR workflows, support tools, finance operations, and any real system where brittle confident automation can be costly.