SpyRL/SpyRL-Qwen3-4B-Summarization

SpyRL-Qwen3-4B-Summarization

Qwen3-4B-Instruct-2507 trained with SpyRL on long-document summarization. In the performing stage each agent summarizes a government report in one paragraph; the spy sees the same report with a contiguous 20% span masked out.

Trained with SpyRL, the reference implementation of RLSVR (Reinforcement Learning with Self-Verifiable Rewards) from the COLM 2026 paper From RLVR to RLSVR: Task Transformation Induces Self-Verifiable Rewards for Open-Ended LLM Self-Improvement.

RLSVR extends RLVR to open-ended tasks the way self-supervised learning extends supervised learning: instead of approximating a missing reward with a judge or reward model, it transforms the task into a proxy environment whose own rules generate the reward. SpyRL instantiates that as a multi-agent self-play game inspired by Who Is the Spy? — civilians receive the full input, one spy receives a masked copy, all agents perform the same target task, and then they vote on who the spy is. Because the environment assigns the spy identity up front, the vote is exactly checkable, and avoiding suspicion requires producing genuinely better output.

No human annotation, no reward model, no LLM judge was used to train this model.

Model details

Exact group size, masking ratio and the rest of the configuration are in the launch script linked below.

Results

ROUGE-L and GPT-4o A/B win rate (%) against the untrained base model.

Trained only on GovReport — the gains on the other four benchmarks are zero-shot transfer.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "SpyRL/SpyRL-Qwen3-4B-Summarization"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto", device_map="auto")

messages = [{"role": "user", "content": "Your prompt here"}]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
outputs = model.generate(inputs, max_new_tokens=1024)
print(tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True))

The chat template and tokenizer are inherited unchanged from the base model.

Training code

Full training code, launch scripts and per-task environments: https://github.com/wangqinsi1/SpyRL-Self-PlaY-Reinforcement-Learning

git clone https://github.com/wangqinsi1/SpyRL-Self-PlaY-Reinforcement-Learning.git && cd SpyRL-Self-PlaY-Reinforcement-Learning
conda create -n spyrl python=3.10 -y && conda activate spyrl
bash setup.sh
bash spyrl/train_summarization.sh

Citation

@inproceedings{wang2026spyrl,
    title     = {From RLVR to RLSVR: Task Transformation Induces Self-Verifiable Rewards for Open-Ended LLM Self-Improvement},
    author    = {Qinsi Wang and Jing Shi and Huazheng Wang and Kun Wan and Yiran Wu and Bo Liu and Qingyun Wu and Hai Helen Li and Yiran Chen and Handong Zhao and Wentian Zhao},
    booktitle = {Conference on Language Modeling (COLM)},
    year      = {2026}
}