thekrishdshah/vergil-sota-trainer

VERGIL — Commitment Dependency Graph Engine

OpenEnv Hackathon · India 2026. An OpenEnv-compatible environment that teaches an LLM to manage a graph of interlocking real-world commitments under partial observability, capacity limits and stakeholder trust dynamics. Themes addressed: #2 (Super) Long-Horizon Planning & Instruction Following  ·  #3.2 Personalized Tasks (Executive Assistant)

1. The problem we're targeting

Most RL-for-LLM environments score each task in isolation. Real personal assistants (and most professional schedulers) live in a commitment dependency graph (CDG) where:

• —accepting a task changes the feasibility of every other task,
• —promises break when a prerequisite slips, cascading through downstream edges,
• —counter-proposing a deadline can save trust at the cost of completion-rate,
• —and "do nothing" is sometimes the optimal action — but only when the schedule is genuinely blocked.

VERGIL captures these tensions in an OpenEnv-compatible Gym-style env. The agent's observation is a partially-observable view of the CDG plus a trust score per stakeholder. Its action space is

{accept, decline, counter_propose, do_nothing} × node_id

2. Why this maps to two hackathon themes

Theme #2 — Long-horizon planning. Each commitment has prerequisites, deadlines and durations that ripple downstream. The agent must reason about trajectories tens of steps long, where a single bad accept early on cascades through the graph and tanks fulfillment 15 steps later. The reward is sparse in time (final fulfillment) but rich in shape (trust deltas, feasibility score) — exactly the "deep, multi-step reasoning with sparse/delayed rewards" the theme calls out.

Theme #3.2 — Personalized tasks. The agent is your over-committed self, managing real-world delegations: dinner conflicts, work overlap, vendor deadlines. We embed it as a backend so it could plug into a real EA-style product.

3. Reward design (10% of judging)

The v1 model collapsed to always-accept because the original reward was trivially gameable. The v2 reward in `vergil/agent/rewards.py` adds three correctives:

R(s, a) = R_env(s, a)                                     # honest signal
        + λ_fmt · 1[parseable JSON, valid action label, target ∈ pending]
        + λ_cap · CapacityPressure(s, a)                  # shaping
        + λ_div · GroupDiversity(a; group)                # anti-collapse

• —Format penalty — agent's response must be parseable JSON with a valid action label and a target ∈ the PENDING set.
• —Capacity-pressure shaping — pushes decline / counter_propose when accepting would break the calendar's 85% buffer.
• —Group-diversity bonus — within each GRPO group of N rollouts of the same prompt, under-represented actions get a small bonus so the advantage estimator can't lock in a degenerate policy.

4. Training pipeline (4× L4 on HF Jobs)

The training script is `scripts/train_vergil_sota.py`. The job entrypoint is `scripts/hf_job_train.sh`. You can re-launch it with:

python scripts/hf_jobs_launch.py --skip-eval 0 --grpo-steps 80 --timeout 150m

5. Showing improvement (20% of judging)

All plots and metrics are pushed to the model repo at the end of every training run. They are mirrored on this Space's `/training` dashboard, which auto-refreshes:

• —Combined training curves (plots/training_curves.png) — SFT loss + GRPO reward + reward-component decomposition + action-distribution share over time, all on one image.
• —6-panel GRPO dashboard (plots/grpo_dashboard.png) — mean reward, policy loss, KL, learning rate, components, action share.
• —SFT loss / GRPO reward / GRPO KL as separate close-ups.
• —Eval comparison plots (eval/eval_compare/plots/) — trained vs. heuristic, side-by-side: per-scenario cumulative reward, action distribution, schedule-satisfiability curve on the simultaneous_infeasibility scenario.

6. How the env works (judge-friendly walkthrough)

The interactive demo on this Space lets a judge:

1. 1.Pick one of 12 hand-crafted scenarios (or a fresh curriculum draw).
2. 2.Watch the trained agent decide, with full reasoning visible.
3. 3.Toggle between the trained LoRA agent and the heuristic baseline to confirm the policy actually learned something non-trivial.
4. 4.Inspect trust scores, capacity pressure, and the live CDG.

Tip for re-running locally: ``bash git clone https://huggingface.co/spaces/thekrishdshah/vergil-sota-trainer vergil cd vergil pip install -r requirements-space.txt VERGIL_MODEL_PATH=thekrishdshah/vergil-sota-trainer python app.py ``

7. OpenEnv compliance

• —Env subclasses gymnasium.Env, exposes reset / step / state cleanly.
• —POMDP wrapper in vergil/core/pomdp.py produces partial observations.
• —Reward returned as scalar float per step; rich diagnostics in info.
• —openenv.yaml declares the env; the demo Space is the discoverable URL.
• —No reserved tool names used.

8. Submission materials

License

Apache-2.0.