anupamagarwal001/amc_allocator_env
0
1# AI Investment Committee Environment2 3## A benchmark for conflict-aware LLM agents4 5LLMs can write confident investment opinions. That is not the hard part.6 7The hard part is what happens when professional roles disagree:8 9```text10Research: strong buy signal in high-momentum tech11Risk: volatility and concentration are above mandate pressure12Portfolio Manager: decide what to do now13```14 15This project turns that situation into a trainable OpenEnv environment. The goal is not to build a trading bot. The goal is to test whether an LLM agent can learn a more professional behavior: resolve conflict between opportunity and constraint.16 1718 19## What the environment simulates20 21The environment has three committee roles:22 23- **Research Analyst:** emits useful but noisy market views.24- **Risk Officer:** flags concentration, drawdown, and mandate pressure.25- **Portfolio Manager:** the only trainable agent; chooses queries and portfolio actions.26 27The PM does not get a perfect world model. It sees partial observations, role-specific advice, portfolio state, and changing constraints. It must decide whether to query Research, query Risk, allocate, revise, hold, or move to cash.28 29This makes the environment a fit for three OpenEnv themes:30 31- **Multi-Agent Interactions:** the PM depends on other actors with different incentives.32- **World Modeling:** the PM must infer hidden market regimes from noisy signals.33- **Long-Horizon Planning:** bad decisions affect drawdown, turnover, and compliance later.34 35## The failure mode36 37The benchmark is built around one memorable failure:38 39```text40Bad PM:41 follows Research only42 increases concentration43 ignores Risk warnings44 triggers compliance and drawdown penalties45 46Target PM:47 asks both Research and Risk48 reduces concentration49 keeps some return exposure50 avoids mandate violations51```52 53That is the capability gap: not "can the model pick a stock?", but "can the model make a disciplined decision when two advisors disagree?"54 55The demo trace makes this concrete:56 5758 59## Task ladder60 61The environment has four deterministic tasks:62 63| Task | What it tests |64| --- | --- |65| `guided_allocation` | basic committee workflow |66| `research_risk_conflict` | bullish Research versus tight Risk |67| `regime_shift_recovery` | hidden deterioration and recovery |68| `mandate_drift` | compliance rules tightening mid-episode |69 70Each task is scored from `0.0` to `1.0` using deterministic graders. The tasks are small enough to run quickly, but structured enough to expose whether the PM is using information and respecting constraints.71 72## Reward design73 74The reward is composable:75 76```text77reward78= portfolio_return79+ signal_alignment_bonus80+ information_usage_bonus81+ risk_response_bonus82- transaction_cost83- query_cost84- variance_penalty85- drawdown_penalty86- compliance_penalty87- invalid_action_penalty88```89 90This matters because a single return-only score is easy to game. A PM that goes all-in on Research can look good briefly but fail the committee task. A PM that always goes to cash avoids risk but fails to use opportunity. The reward is designed to favor balanced professional behavior.91 92## Reward Hacking Considerations93 94Mitigation design:95- The reward is multi-objective, so return alone is not enough to score well.96- Drawdown, compliance breaches, transaction costs, and query costs create explicit penalties.97- Partial observability and query budgets force the PM to gather useful committee information.98 99Known failure modes:100- A policy may become over-conservative and move to cash too often.101- A policy may learn minimal-compliance behavior instead of true conflict resolution.102- Short smoke training can still bias toward near-term reward.103 104Reward hacking is a known RL problem, and this environment does not claim to eliminate it. The benchmark is designed to expose these trade-offs through separate return, compliance, information-usage, and risk-response metrics.105 106Anti-hack probe results:107 108| Probe policy | Overall score | What it tries | Why it fails |109| --- | ---: | --- | --- |110| `always_cash` | `0.2000` | Avoid all drawdown and compliance risk | Capped for no investment or information usage |111| `query_spam` | `0.2000` | Spend query budget instead of deciding | Capped for no portfolio action or risk response |112| `concentrated_alpha` | `0.3033` | Chase return with maximum concentration | Loses all compliance credit |113| `heuristic` | `0.4084` | Balance Research, Risk, and allocation | Uses information and accepts measured risk |114 115## Baselines116 117Before training, we compare a random PM against a heuristic PM.118 119| Policy | Overall score | Return | Max drawdown | Information usage |120| --- | ---: | ---: | ---: | ---: |121| Random | `0.2325` | `-0.0061` | `0.0209` | `0.0436` |122| Heuristic | `0.4084` | `0.0217` | `0.0153` | `0.2716` |123 124125 126The heuristic is not perfect. That is intentional. It gives the trainer a meaningful starting point without pretending the task is solved.127 128## Training evidence129 130We trained only the Portfolio Manager. Research and Risk stayed scripted so reward attribution remains clear.131 132Training setup:133 134- model: `Qwen/Qwen3-0.6B`135- trainer: Hugging Face `TRL` `GRPOTrainer`136- adapter: LoRA137- runtime: T4 GPU138- stable smoke config: `max_steps=4`, `repeats_per_task=2`139 140Verified smoke result:141 142| Metric | Value |143| --- | ---: |144| Reward start | `0.0193` |145| Reward end | `0.0275` |146| Reward delta | `+0.0082` |147| Best step | `4` |148 149### What RL is learning (from actual training traces)150 151We also preserved a second HF Jobs run with the LoRA adapter and raw GRPO completion traces. One late training sample shows the verifier preferring a PM candidate that calls both Risk and Research before taking exposure.152 153154 155This is training-rollout evidence, not a deployment claim. The point is narrower: the reward model is reinforcing the intended conflict-resolution behavior.156 157158 159The run also logged the GRPO optimizer loss/objective. GRPO loss can cross below zero because it is not a standard supervised accuracy loss.160 161162 163We also ran an 8-step comparison. It peaked at step `4` and then regressed, which is why the final demo uses the 4-step smoke configuration. That regression is useful evidence rather than a failure: it shows the environment is sensitive enough to expose unstable optimization.164 165## What this proves166 167This does not prove we have a production-quality financial agent. That is not the claim.168 169The claim is narrower and testable:170 171> AI Investment Committee Environment is a working OpenEnv benchmark where verifier-driven RL can produce measurable training signals for conflict-aware decision-making.172 173The project demonstrates:174 175- a non-toy professional environment176- explicit multi-agent roles177- deterministic tasks and graders178- a reward model that captures more than return179- a runnable TRL training surface180- real reward and loss plots from a GPU run181 182## Links183 184- Hugging Face Space: [anupamagarwal001/amc_allocator_env](https://huggingface.co/spaces/anupamagarwal001/amc_allocator_env)185- Live app: [anupamagarwal001-amc-allocator-env.hf.space/web](https://anupamagarwal001-amc-allocator-env.hf.space/web)186- GitHub repository: [anupamagarwal001/hackathon_submission](https://github.com/anupamagarwal001/hackathon_submission)187- Colab notebook: [Google Colab runbook](https://colab.research.google.com/drive/1Rj7rkkYTxhoqCqmpbR5b48dOeNP5Oucw)188- HF Jobs artifacts: [hf-job-20260425-070718](https://huggingface.co/datasets/anupamagarwal001/amc-allocator-job-artifacts/tree/main/hf-job-20260425-070718)189- Trained trace artifacts: [hf-job-20260426-trained-trace](https://huggingface.co/datasets/anupamagarwal001/amc-allocator-job-artifacts/tree/main/hf-job-20260426-trained-trace)190