garvitsachdeva/SpindleFlow-RL

1---2title: SpindleFlow RL3emoji: 🤖4colorFrom: blue5colorTo: purple6sdk: streamlit7sdk_version: "1.40.0"8app_file: streamlit_app.py9pinned: false10---11 12# SpindleFlow RL — Delegation Policy RL Environment13 14An RL environment that trains an orchestrator to **learn** delegation strategy,15built on top of the SpindleFlow multi-agent execution system.16 17## Architecture18 19```20SpindleFlow (TypeScript) ← execution backend21SpindleFlow RL (Python)  ← RL training layer22```23 24The RL agent learns *which specialists to call, in what mode, and when to stop* —25not how to write YAML. SpindleFlow executes the decisions; the RL policy makes them.26 27## Key Design Decisions28 29| Component | Design | Why |30|---|---|---|31| Reward | Tiered cascade (0/1/2/3) with episode-level tier lock | Valid delta, no tier drift, $8/1000-episode run |32| Roster | Capability embeddings (all-MiniLM-L6-v2, 384-dim) | Zero-shot generalization to new specialists |33| Delegation | DAG with cycle detection + action masking | No A→B→A loops |34| Policy | LSTM PPO (RecurrentPPO, SB3) | POMDP-safe for scratchpad context |35| Graph encoding | Padded adjacency MLP (not GNN) | Hackathon-feasible; GNN for production |36| Consistency | Dirichlet prior (alpha=1.0) | Non-zero reward from Episode 1 |37| Stopping | STOP as explicit learned action (Head 1) | Adaptive, not hardcoded |38 39## Quick Start40 41```bash42# 1. Install dependencies43pip install -r requirements.txt44pip install sb3-contrib45 46# 2. Set environment variables47cp .env.example .env48# Edit .env with your OPENAI_API_KEY49 50# 3. Run smoke tests51pytest tests/ -v52 53# 4. Pre-compute demo assets54python demo/precompute_demo.py55 56# 5. Start training (Phase 1)57python training/train.py --phase 1 --timesteps 5000058 59# 6. Watch training curves60tensorboard --logdir tensorboard_logs/61 62# 7. Run demo63python demo/run_demo.py64```65 66## Reward Function67 68```python69total_reward = (70    quality_delta          # specialist_score - baseline_score (same tier)71  - efficiency_penalty     # 0.05 * max(0, n_specialists - expected)72  - failure_penalty        # 0.3 per timeout, 0.2 per error (reduced if fallback)73  + recovery_bonus         # 0.1 if fallback recovered successfully74  - conflict_penalty       # 0.1 per unresolved conflict75  + conflict_bonus         # 0.05 per resolved conflict76  + consistency_bonus      # 0.1 * Dirichlet-prior path consistency77  - latency_penalty        # latency_weight * overage_fraction (tunable)78  + explanation_bonus      # 0.05 if delegation is auditable79)80```81 82## Project Structure83 84```85spindleflow-rl/86├── env/                   ← Gymnasium environment + state/action/graph87├── reward/                ← Tiered reward, failure/conflict/latency signals88├── agents/                ← Task decomposer, fallback chains, conflict resolver89├── policy/                ← LSTM policy, state encoder, action heads90├── training/              ← PPO training loop, curriculum, task bank91├── transfer/              ← Cross-company fine-tuning strategy92├── audit/                 ← Delegation trace + explanation generation93├── security/              ← Scratchpad sandbox isolation94├── demo/                  ← Before/after demo assets + precompute script95├── colab/                 ← Google Colab training notebook96├── huggingface_blog/      ← HuggingFace mini-blog97├── tests/                 ← Pytest test suite (20 tests, all passing)98└── configs/               ← Specialist catalog + training hyperparameters99```100 101## OpenEnv Compliance102 103`SpindleFlow-v0` is registered with OpenEnv (hackathon requirement):104 105```python106import env.openenv_wrapper  # triggers registration107from env.openenv_wrapper import verify_openenv_compliance108verify_openenv_compliance()  # True109```110 111## Observation Space112 113Flat `(5490,)` float32 vector (for `max_specialists=6`):114 115| Component | Dim |116|---|---|117| Task embedding | 384 |118| Roster embeddings (6×384) | 2304 |119| Called embeddings (6×384) | 2304 |120| Scratchpad embedding | 384 |121| Delegation graph adjacency | 100 |122| Called specialist mask | 6 |123| Scalar features | 8 |124| **Total** | **5490** |125 126## Action Space127 128Flat `(12,)` continuous Box (for `max_specialists=6`):129 130| Slot | Meaning |131|---|---|132| `[0]` | Meta-action (CALL_SPECIALIST / STOP / …) |133| `[1:7]` | Specialist selection logits (multi-hot) |134| `[7]` | Delegation mode (SEQUENTIAL / PARALLEL / …) |135| `[8:12]` | Mode parameters (rounds, threshold, budget) |136 137## Training138 139```bash140# Demo mode (no OpenAI calls, fast)141python training/train.py --phase 1 --timesteps 50000 --demo-mode142 143# Full run with T2 reward144python training/train.py --phase 1 --timesteps 100000145 146# Resume from checkpoint147python training/train.py --checkpoint checkpoints/spindleflow_rl_50000_steps.zip148```149 150## Colab151 152See [colab/README_COLAB.md](colab/README_COLAB.md) for Google Colab quick start (T4 GPU, free tier).153 154## HuggingFace155 156See [huggingface_blog/blog_post.md](huggingface_blog/blog_post.md) for the submission blog post.157