swadeshb/noop-summary-grpo-310-conditional

noop-summary-grpo-310-conditional

This is the merged, deployable checkpoint of a conditional Qwen3-4B context manager fine-tuned from `YWZBrandon/summary-sft-qwen3-4b`. It was trained for one epoch with TRL GRPO on 310 preserved software-engineering agent trajectories.

The model supports a hierarchical context-management policy with a strict 8,192-token budget rule:

1. 1.Above the budget, the policy deterministically selects SUM.
2. 2.At or below the budget, a learned gate emits KEEP or SUM.
3. 3.When SUM is selected, a second request generates a plain-text summary of the older interaction history while retaining the system prompt, task, and two most recent interaction messages.

Above budget, each training group contains eight sampled summaries. At or below budget, it contains one virtual KEEP/no-op reference and seven sampled summaries. The summary policy is optimized with GRPO, while the below-budget gate learns the measured counterfactual utility margin from balanced replay.

Loading

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "swadeshb/noop-summary-grpo-310-conditional"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype="auto",
    device_map="auto",
)

The checkpoint contains merged BF16 weights and does not require PEFT at inference time.

Training configuration

• —Base model: YWZBrandon/summary-sft-qwen3-4b
• —Global training steps: 1,443
• —Manager prompt limit: 16,384 tokens
• —Maximum completion length: 2,048 tokens
• —Strict context budget: 8,192 DeepSWE-tokenizer tokens
• —Summary samples per decision: 8 above budget; 7 plus one no-op below budget
• —LoRA rank / alpha: 16 / 32
• —Learning rate: 1e-6
• —TRL KL coefficient: 0.01
• —Seed: 42

Limitations

• —The reward measures preservation of the recorded next DeepSWE action, not long-horizon task success.
• —The learned gate is used only at or below the strict budget boundary.
• —The gate target is a contextual-bandit estimate from seven sampled summaries.
• —The DeepSWE FP8 scoring calibration may be unreliable for very small log-probability differences.
• —The training set is small and specialized; evaluate the model on your target agent and context distribution before deployment.

Framework versions

• —TRL: 1.5.1
• —Transformers: 4.57.6
• —PyTorch: 2.10.0
• —Datasets: 4.8.5
• —Tokenizers: 0.22.2