anupamagarwal001/amc_allocator_env

AI Investment Committee Environment

Can an LLM learn to behave like a real investment committee member when Research and Risk disagree?

This project is a multi-agent OpenEnv benchmark for conflict-aware decision-making. A trainable Portfolio Manager must manage a portfolio while interacting with two scripted environment actors:

• —Research Analyst: finds opportunities and emits noisy buy/sell conviction.
• —Risk Officer: enforces mandate pressure, concentration limits, and drawdown discipline.
• —Portfolio Manager: decides when to query, allocate, revise, hold, or move to cash.

The failure mode is intentionally human: a naive PM chases Research and ignores Risk. The target behavior is not maximum return at all costs; it is professional judgment under conflicting incentives.

Submission Links

Try The Live Environment

The Space exposes both a visual OpenEnv playground and the raw API:

1. 1.Open the live app at `/web`, then click Reset.
2. 2.Try one action in the form, for example action_type=query_research, query_target=SECTOR, then click Step.
3. 3.If the playground is unavailable, use the Swagger fallback at `/docs`: call POST /reset, then POST /step.

Sample POST /step body:

{
  "action": {
    "action_type": "query_research",
    "query_target": "SECTOR",
    "reason": "Demo: get the committee research view before allocating."
  }
}

The One-Screen Story

The benchmark exposes when a Portfolio Manager blindly follows Research while Risk is warning that the mandate is under pressure.

[image]

Falsifiable claim: this environment measures whether verifier-driven RL can improve conflict-aware PM behavior. In a verified HF Jobs/Colab smoke run, reward improved from 0.0193 to 0.0275 over the stable 4-step config.

What The Agent Does In A Run

The clearest visible behavior difference is between a random PM and a committee-aware heuristic PM on the same conflict task.

[image]

This is baseline behavior, not a claim that the trained policy beats the heuristic. The trained evidence is the GRPO reward signal shown below.

What Changed After Training?

This is a small smoke run, not a claim of convergence. The important point is that the full loop exists and produces measurable signals:

• —OpenEnv environment with reset, step, and state
• —verifier-style reward components
• —TRL GRPOTrainer
• —LoRA adapters on Qwen/Qwen3-0.6B
• —exported reward/loss curves and judge-facing reports
• —exported LoRA adapter and GRPO completion traces

[image]

What RL is learning (from actual training traces)

The trained trace run also preserves the actual GRPO completion files and LoRA adapter. A late training sample shows the verifier assigning higher reward to a candidate that gathers both Risk and Research before taking exposure.

[image]

This is intentionally scoped as training-rollout evidence. It shows what the reward model is reinforcing; it is not a claim that the saved adapter already beats the heuristic baseline in deployment.

[image]

[image]

The 8-step comparison is also included because it is useful evidence: the run peaked at step 4 and regressed after that. That is why the final demo uses the 4-step config instead of pretending that longer training was automatically better.

Inference And Evaluation Results

The repository keeps inference separate from training. `inference.py` runs a policy through the OpenEnv tasks and prints structured [START], [STEP], and [END] blocks with per-task scores. That is the judge-facing "agent acts in the environment" path.

Current verified inference/evaluation result:

guided_allocation        heuristic=0.3462
research_risk_conflict   heuristic=0.4071
regime_shift_recovery    heuristic=0.4211
mandate_drift            heuristic=0.4591
overall heuristic score  0.4084
overall random score     0.2325

Why This Is Not A Toy Finance Simulator

The environment is about conflict resolution, not stock picking. The PM sees partial state, noisy role-specific advice, changing constraints, and delayed portfolio consequences. It must decide when to trust Research, when to respect Risk, and when to revise a previous decision.

That maps directly to the Round 2 themes:

• —Multi-Agent Interactions: Research, Risk, and PM have different objectives.
• —World Modeling: the PM never sees hidden market regimes directly.
• —Long-Horizon Planning: drawdown, turnover, and mandate drift compound over time.
• —Self-Improving Systems: the PM is the trainable role; the other actors make the environment stable and measurable.

Environment Design

The task ladder has four deterministic scenarios:

The PM action space is deliberately workflow-shaped:

• —query_research
• —query_risk
• —allocate
• —revise_allocation
• —hold
• —move_to_cash

The observation includes prices, signals, constraints, recent committee notes, portfolio state, reward breakdowns, risk metrics, and remaining query budget.

Reward Model

The reward is composable rather than a single vague score:

reward
= portfolio_return
+ signal_alignment_bonus
+ information_usage_bonus
+ risk_response_bonus
- transaction_cost
- query_cost
- variance_penalty
- drawdown_penalty
- compliance_penalty
- invalid_action_penalty

This makes the verifier harder to game. A PM cannot win by always going to cash, blindly maximizing return, or spamming queries. The score only improves when the agent balances return, information usage, risk response, and mandate discipline.

Reward Hacking Considerations

Mitigation design:

• —The reward is multi-objective, so return alone is not enough to score well.
• —Drawdown, compliance breaches, transaction costs, and query costs create explicit penalties.
• —Partial observability and query budgets force the PM to gather useful committee information.

Known failure modes:

• —A policy may become over-conservative and move to cash too often.
• —A policy may learn minimal-compliance behavior instead of true conflict resolution.
• —Short smoke training can still bias toward near-term reward.

Reward hacking is a known RL problem, and this environment does not claim to eliminate it. The benchmark is designed to expose these trade-offs through separate return, compliance, information-usage, and risk-response metrics.

Anti-hack probe results:

Training Surface

The training setup intentionally trains only one role:

• —trainable: Portfolio Manager
• —scripted environment actors: Research Analyst and Risk Officer

This keeps reward attribution clean. If behavior improves, the change came from the PM policy rather than from moving all agents at once.

Core training files:

• —`training/committee_grpo_colab.ipynb`
• —`training/committee_grpo_train.py`
• —`training/committee_eval.py`
• —`training/committee_artifacts.py`
• —`training/hf_jobs_smoke.py`
• —`training/launch_hf_job.py`

Judge-facing runbooks:

• —`Blog.md`
• —`docs/ROUND2_DEMO_FLOW.md`
• —`docs/ROUND2_TRAINING_RUNBOOK.md`
• —`docs/ROUND2_PITCH_SCRIPT.md`
• —`docs/ROUND2_ONSITE_CHECKLIST.md`
• —`docs/AI_Investment_Committee_Deck.html`

Reproduce The Key Checks

Use Python 3.11.

uv sync --python python3.11 --extra dev

Run the validator-compatible inference entrypoint:

python3.11 inference.py --policy heuristic
python3.11 inference.py --policy random

Run baseline evaluation:

python3.11 training/committee_eval.py --policy heuristic
python3.11 training/committee_eval.py --policy random

Regenerate the committed README plots:

python3 training/generate_readme_assets.py

Launch the HF Jobs smoke run:

python3 training/launch_hf_job.py launch

Validate the OpenEnv server locally:

AMC_TASK_ID=guided_allocation uvicorn server.app:app --host 0.0.0.0 --port 8000
openenv validate --url http://localhost:8000

Project Layout

amc_allocator_env/
├── Blog.md
├── README.md
├── inference.py
├── models.py
├── tasks.py
├── graders.py
├── policies.py
├── data/
├── server/
├── training/
├── docs/
└── tests/

What To Look At First

If you only have three minutes, open these in order:

1. 1.Conflict snapshot
2. 2.Demo trace comparison
3. 3.GRPO behavior sample
4. 4.Reward curve
5. 5.Loss curve
6. 6.HF mini-blog
7. 7.Colab training notebook