aparnasingha400/canary-7b-job-output-v2

Canary Release Manager — Qwen2.5-7B GRPO Adapter

This repository contains a LoRA adapter trained for the Canary Release Manager OpenEnv environment using Qwen2.5-7B-Instruct with SFT warm-start + GRPO.

Model overview

• —Base model: unsloth/Qwen2.5-7B-Instruct
• —Training pipeline: SFT warm-start + GRPO
• —Hardware: NVIDIA A100-SXM4-80GB
• —Environment calls: 2109
• —GRPO runtime: 460.03 s
• —Final GRPO train loss: 0.00210

Task

The model is trained to make rollout decisions in a canary deployment environment from telemetry signals.

It predicts:

• —rollout action
• —structured state assessment
• —breach / warning interpretation
• —short grounded reasoning

Supported rollout actions in the environment include:

• —increase_5
• —increase_10
• —increase_25
• —hold
• —rollback

The environment is hosted separately as an OpenEnv-compatible service.

Training summary

This run completed successfully with:

• —SFT enabled
• —GRPO enabled
• —parse_ok telemetry of 100% during RL
• —rewarderrorrate of 0.0
• —diagnostic verdict: PASS

Evaluation summary

Before training

After SFT

Final model (after GRPO)

Generalization / stress

• —Generalization average: 0.6553
• —Stress average: 0.6544

Heldout / generalization comparison

[image]

Reward curve

[image]

Per-task reward during GRPO

[image]

GRPO loss curve

[image]

Action distribution during RL

[image]

Key observations

• —The final model improved the overall average score from 0.3457 before training to 0.6553 after GRPO.
• —The RL stage improved over the post-SFT checkpoint, increasing the average from 0.6105 to 0.6553.
• —Heldout and generalization averages are identical in this run, suggesting stable behavior across the chosen evaluation seeds.
• —The learned policy uses multiple rollout actions during RL telemetry, including increase_25, increase_10, rollback, and hold.

Reward and telemetry

The final run reported:

• —parse_ok_rate: 1.0
• —reward_error_rate: 0.0
• —diagnostic verdict: PASS

Action counts during RL telemetry:

• —increase_25: 155
• —increase_10: 118
• —rollback: 28
• —hold: 19

Training metrics

• —Global GRPO steps: 80
• —Reward calls: 80
• —Train runtime: 460.03 s
• —Train steps per second: 0.174
• —Training loss: 0.00210

Evidence included in this repository

This repository includes:

• —reward curve
• —per-task reward plot
• —GRPO loss curve
• —before/after/generalization comparison
• —action distribution plot
• —expert trace
• —structured JSON results

Trace note

A saved expert trace (seed=950) achieved an episode score of 0.4975. The trace shows some sensible warning/hold behavior, but it also still includes parse failures in several steps. This means the model is meaningfully improved over the raw baseline, but it is not yet a perfectly reliable controller.

Notes / limitations

• —The final average score (0.6553) is a solid improvement, but it is still below the hand-coded shallow and cautious reference policies on most tasks.
• —The policy still leans toward promotive actions, especially increase_25 and increase_10.
• —Some trace-time generations still fail strict JSON parsing even though RL telemetry reported perfect parser success.
• —Silent-task performance remains weaker than easy, medium, and recovery settings.
• —This is a hackathon-scale training run, not a production deployment or a large-scale research benchmark.

Reproducibility

• —Training notebook: https://colab.research.google.com/drive/1h-SPUGrxL160yXugjWUM0yi7tF0iN5OX?usp=sharing
• —Before/after demo notebook: https://colab.research.google.com/drive/1dZA5Yq1RshhQjMzeGg9ZYm-4y-LTB7m4?usp=sharing
• —Hugging Face Jobs run log: Training job log
• —Environment Space: https://huggingface.co/spaces/aparnasingha400/canary-release-manager
• —Environment repo: https://github.com/Lekhana-Dinesh/Canary-Release-Manager

Blog / write-up

• —Project blog: https://huggingface.co/spaces/aparnasingha400/canary-release-manager/blob/main/Blog.md