realambuj2001/schemaquake1

1# SchemaQuake: Teaching Agents To Notice When The World Changes2 3Most AI agent demos quietly make one huge assumption: the world stays still.4 5The API response shape stays the same. The policy document stays the same. A number keeps meaning the same unit. A boolean keeps meaning the same thing. The agent can make a plan at the beginning of the task and then execute that plan as if nothing will change.6 7That is not how production systems behave.8 9In real software, APIs evolve. A field named `price_rupees` may become `ticket_price`. A backend may switch from rupees to paise. A policy document may be updated while a workflow is in progress. A simple `refundable=true` field may become a tiered concept: full refund, partial refund, or no refund.10 11The scary part is that an LLM agent may not crash when this happens. It may continue confidently, book the wrong thing, and report success.12 13SchemaQuake is an RL environment for that failure mode.14 15## The Simple Story16 17SchemaQuake looks like a travel-booking task on the surface.18 19The user asks:20 21> Book a refundable flight from BLR to DEL under 8000 rupees.22 23The agent has tools:24 25- search flights26- read cancellation policy27- inspect API schema28- book a flight29- cancel a booking30- ask the user31- submit the final answer32 33At first, the world looks stable. Then, during the episode, the environment silently changes. The agent is not told that anything changed. It has to infer that from observations.34 35For example, the agent may search flights and see one format. Later, after drift, the same data source may return a renamed price field or a different refundability representation. A careless agent keeps going. A careful agent re-checks schema or policy before committing.36 37That is the core question:38 39> Can an agent notice that its assumptions about the world are no longer reliable?40 41## Why Travel Booking?42 43The point is not to build the world's best travel agent.44 45Travel booking is a clean, understandable professional workflow. Judges can immediately understand the stakes:46 47- the ticket must be under budget48- the ticket must be refundable49- policy matters50- wrong assumptions can lead to a costly booking51 52Underneath that simple surface, SchemaQuake tests a general enterprise-agent skill: updating beliefs in a partially observable, changing system.53 54## What The Agent Sees And Does55 56SchemaQuake is implemented as an OpenEnv-compatible environment with a standard `reset`, `step`, and `state` loop.57 58At reset, the environment creates:59 60- a user request61- flight offers62- a policy document63- a hidden drift schedule64- a hidden drift type65 66At each step, the agent submits an action. The environment returns an observation and reward metadata.67 68The hidden drift may be one of:69 70- field rename71- unit change72- enum mutation73- policy update74 75The observation never says, “drift happened.” That would make the task too easy. The model has to discover that something is inconsistent by using tools.76 77## Reward Design78 79The reward is not just “success” or “failure.”80 81SchemaQuake rewards several behaviors:82 83- completing the user's real task84- staying within budget85- booking a fully refundable option when required86- detecting drift by re-reading schema or policy87- asking the user only when uncertainty is meaningful88- finishing efficiently89 90It penalizes:91 92- silent violations93- malformed or invalid actions94- over-probing the schema95- submitting without a valid booking96- unnecessary high-confidence user interruptions97 98The most important metric is **silent violation rate**.99 100Silent violation rate measures how often the agent confidently submits something wrong. This is the failure mode that matters most for production agents.101 102## Training103 104We use Hugging Face TRL / GRPO with rollout rewards from the SchemaQuake environment.105 106The core submission is the environment. Training is included to prove that the environment exposes useful learning signals and can be used for post-training agents, not because the hackathon is only about producing the strongest final model.107 108The training pipeline does three things:109 1101. Generate heuristic traces that show a drift-aware workflow.1112. Use those traces as a light SFT-style warm start.1123. Run GRPO where generated actions are scored by executing them in SchemaQuake.113 114For the submitted run, we used:115 116- model: `Qwen/Qwen2.5-0.5B-Instruct`117- curriculum: `mixed`118- steps: `50`119- generated heuristic traces: `250`120- trained model repo: [realambuj2001/schemaquake1-lora](https://huggingface.co/realambuj2001/schemaquake1-lora)121 122The live Space is here:123 124[https://huggingface.co/spaces/realambuj2001/schemaquake1](https://huggingface.co/spaces/realambuj2001/schemaquake1)125 126## Results127 128In the normal evaluation run:129 130- random agent task success: `0.17`131- random silent violation rate: `0.83`132- imperfect heuristic task success: `0.90`133- imperfect heuristic silent violation rate: `0.10`134- heuristic task success: `1.00`135- heuristic silent violation rate: `0.00`136 137In hard drift mode:138 139- base Qwen silent violation rate: `1.00`140- imperfect heuristic silent violation rate: `0.18`141- SFT policy silent violation rate: `0.16`142- GRPO-style policy silent violation rate: `0.00`143- heuristic upper bound silent violation rate: `0.00`144 145The imperfect heuristic is intentionally useful but fallible. It gives judges a middle baseline: stronger than random, weaker than the upper-bound policy, and close enough to a real partially trained agent that improvements are visible in the chart.146 147The important takeaway is not that one short run creates a perfect model. The important takeaway is that SchemaQuake creates a measurable RL environment where safer behavior can be trained and compared.148 149## What The Demo Shows150 151The Space has four judge-facing views:152 1531. A single episode trace showing every action and reward.1542. Normal evaluation comparing baseline behavior.1553. Hard drift evaluation where silent violations become obvious.1564. Live GRPO training logs and a training curve, plus a GPU-only Model Demo that shows the small flight dataset, lets the judge inject drift, and runs the pushed trained model repo against the same environment.157 158The Colab notebook also includes a notebook-friendly drift demo and an optional GPU trained-model cell using `realambuj2001/schemaquake1-lora`. The result artifacts are stored under the `results/` folder, including training logs, output JSON, benchmark metrics, and the training curve.159 160## Why This Matters161 162The next generation of LLM agents will not only answer questions. They will operate software.163 164They will touch billing systems, HR tools, support dashboards, finance workflows, APIs, and policy engines. Those systems change. If the agent cannot notice change, it will fail in quiet and expensive ways.165 166SchemaQuake turns that production risk into a training environment.167 168It teaches an agent a simple but powerful instinct:169 170> Before acting with confidence, check whether the world still means what you think it means.171 172That is the behavior we want from real professional agents.173