swadeshb/noop-summary-grpo-310

noop-summary-grpo-310

This is the merged, deployable checkpoint of a Qwen3-4B context manager fine-tuned from `YWZBrandon/summary-sft-qwen3-4b`. It was trained for one epoch with TRL GRPO on 310 preserved software-engineering agent trajectories.

The model supports a hierarchical context-management policy:

1. 1.Given the current history, token length, and budget, the gate emits KEEP or SUM.
2. 2.When the gate emits SUM, a second request generates a plain-text summary of the older interaction history while retaining the system prompt, task, and two most recent interaction messages.

Training compares each retained context against seven sampled summaries using a frozen DeepSWE next-action scorer. The summary policy is optimized with GRPO; the gate learns the measured counterfactual utility margin from balanced replay.

Loading

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "swadeshb/noop-summary-grpo-310"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype="auto",
    device_map="auto",
)

The checkpoint contains merged BF16 weights and does not require PEFT at inference time.

Training configuration

• —Base model: YWZBrandon/summary-sft-qwen3-4b
• —Manager prompt limit: 16,384 tokens
• —Maximum completion length: 2,048 tokens
• —Context budget threshold: 8,192 DeepSWE-tokenizer tokens
• —Summary samples per decision: 7
• —LoRA rank / alpha: 16 / 32
• —Learning rate: 1e-6
• —TRL KL coefficient: 0.01
• —Seed: 42

Limitations

• —The reward measures preservation of the recorded next DeepSWE action, not long-horizon task success.
• —The gate target is a contextual-bandit estimate from seven sampled summaries.
• —The DeepSWE FP8 scoring calibration may be unreliable for very small log-probability differences.
• —The training set is small and specialized; evaluate the model on your target agent and context distribution before deployment.

Framework versions

• —TRL: 1.5.1
• —Transformers: 4.57.6
• —PyTorch: 2.10.0
• —Datasets: 4.8.5
• —Tokenizers: 0.22.2