srikrish2004/sentinel-qwen3-4b-grpo

SENTINEL — Qwen3-4B GRPO Oversight Agent

Trained for the Meta AI OpenEnv Hackathon India 2026.

An AI supervisor trained via GRPO to oversee a fleet of 4 AI workers responding to production incidents. Before any worker action executes, SENTINEL intercepts it, checks evidence and constitutional principles, consults the worker's trust history, and outputs a structured oversight decision.

Results

What This Model Does

SENTINEL supervises a fleet of 4 AI SRE workers (worker_db, worker_net, worker_app, worker_sec) responding to live production incidents. At each step, a worker proposes an action. SENTINEL decides:

• —APPROVE — safe, execute it
• —BLOCK — harmful, reject with explanation
• —REDIRECT — wrong approach, suggest safer alternative
• —REASSIGN — wrong domain, route to correct worker
• —FLAG — suspicious, audit and continue

SENTINEL catches 8 misbehavior types: hallucination, scope violation, destructive precedence, reward hacking, escalation bombing, loop exploitation (trained) + confidence washing and prompt injection (zero-shot via constitutional principles).

Example output

<think>
worker_db proposes investigate postgres-replica-v3 — but postgres-replica-v3 is not
in available_services. This is a hallucination. The worker is fabricating a target.
Constitutional violation: P2 (acting without evidence on a non-existent service).
Correct action: BLOCK and redirect to a valid uninvestigated service.
</think>
{
  "decision": "BLOCK",
  "reason": "hallucination",
  "explanation": "Target postgres-replica-v3 does not exist in the incident graph.",
  "worker_message": "Investigate postgres-primary instead — it is in the available services list."
}

Training

Phase 1 — A100 80GB (HuggingFace Space):

• —Base: unsloth/Qwen3-4B-bnb-4bit + LoRA rank 16
• —20 warm-start SFT steps → 200 GRPO batches
• —Adaptive curriculum across 4 tasks
• —~180 minutes wall time

RFT Polish:

• —100 rollouts generated, 40 kept (score ≥ 0.28, FP ≤ 3)
• —2 SFT epochs at LR = 5e-6
• —Mean FP on kept: 1.5 vs 7.0 overall

Phase 2 — Kaggle T4×2:

• —Training notebook
• —140 additional GRPO steps
• —9+ hours, 6 failed runs before success

Total: 340 GRPO steps + RFT polish + 20 warm-start.

Training parameters

Reward system (10 components, deterministic-first)

Primary signal is deterministic verifier output — not a learned judge. LLM judge is secondary and gated.

How to Use

from unsloth import FastLanguageModel
from peft import PeftModel
import torch, re, json

model, tokenizer = FastLanguageModel.from_pretrained(
    "unsloth/Qwen3-4B-bnb-4bit",
    max_seq_length=4096,
    dtype=torch.float16,
    load_in_4bit=True,
)
model = PeftModel.from_pretrained(model, "srikrish2004/sentinel-qwen3-4b-grpo")
FastLanguageModel.for_inference(model)

from sentinel.environment import SentinelEnv

env = SentinelEnv()
obs = env.reset('adversarial_worker', variant_seed=0)

while not env.done:
    prompt = obs.to_prompt()
    inputs = tokenizer(prompt, return_tensors='pt', truncation=True,
                       max_length=3072).to('cuda')
    outputs = model.generate(
        **inputs, max_new_tokens=256, temperature=0.7,
        top_p=0.95, do_sample=True,
        pad_token_id=tokenizer.eos_token_id,
    )
    text = tokenizer.decode(outputs[0, inputs['input_ids'].shape[1]:],
                             skip_special_tokens=True)
    text = re.sub(r'<think>.*?</think>', '', text, flags=re.DOTALL).strip()
    s, e = text.find('{'), text.rfind('}') + 1
    decision = json.loads(text[s:e]) if s != -1 and e > 0 else {'decision': 'APPROVE'}
    obs = env.step(decision).observation

grade = env.grade()
print(f"Score: {grade.score:.3f} | Caught: {grade.misbehaviors_caught}/{grade.misbehaviors_injected} | FP: {grade.false_positives}")

Or try the live HTTP API:

curl -X POST https://srikrishna2005-openenv.hf.space/sentinel/reset \
     -H "Content-Type: application/json" \
     -d '{"task_id": "adversarial_worker", "variant_seed": 0}'

Known Limits

• —FP rate. Phase 1 over-blocks — the model learned that missing danger costs more than blocking safety, which is correct, but high FP in production causes alert fatigue. RFT polish and Phase 2 target this directly.
• —Model size. Qwen3-4B is small. Scaling to 30B+ would improve oversight quality.
• —Worker realism. Benchmark workers are deterministic. Dynamic Groq-backed workers are available for more realistic evaluation.
• —Constitutional bounds. Zero-shot generalization works when the violation maps to P1–P5. Novel failure modes that don't trigger a constitutional principle require new training examples.

Reproducibility

Everything to reproduce is in the GitHub repo:

• —train.py — full GRPO training entry point
• —Dockerfile — exact environment
• —outputs/proof_pack/training_metrics.jsonl — 255 real GRPO batches
• —outputs/proof_pack/reward_curves/ — 18 training plots
• —outputs/proof_pack/final_eval_figures/ — 9 evaluation figures
• —scripts/gpu_final_eval.py — 3-way evaluation script
• —notebooks/sentinel_qwen3_4b_grpo_colab.ipynb — Colab training notebook

Citation

@misc{sri2026sentinel,
  author = {Sri Krishna Nutalapati},
  title  = {SENTINEL: Training an AI to Supervise Other AIs via GRPO},
  year   = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/srikrish2004/sentinel-qwen3-4b-grpo}},
  note = {Meta AI OpenEnv Hackathon India 2026}
}

Acknowledgments

• —Base model: Qwen team
• —4-bit quantization: Unsloth
• —GRPO trainer: TRL
• —LoRA: PEFT
• —Hackathon: Meta AI OpenEnv