DaoyuanLi/mini-verl-qwen3-1.7b-protocol-teacher

miniVERL Qwen3-1.7B protocol teacher

This repository contains a frozen PEFT LoRA adapter for `Qwen/Qwen3-1.7B`. It was trained by miniVERL on deterministic calculator oracle traces expressed in miniVERL's <tool_call>, <tool_result>, and <final> protocol. It is a narrow calculator protocol policy, not a general-purpose improvement to Qwen3.

Only the adapter is distributed here; the pinned base model is downloaded separately. No optimizer state, training checkpoint, user path, or base-model weight is included.

Immutable base and tokenizer identity

• —Base model: Qwen/Qwen3-1.7B
• —Base revision: 70d244cc86ccca08cf5af4e1e306ecf908b1ad5e
• —Tokenizer revision: 70d244cc86ccca08cf5af4e1e306ecf908b1ad5e
• —miniVERL tokenizer fingerprint: f2f5e826dddc3ff1e2481111075f2ed6eced4e553168222d67650931d25be035
• —License: Apache-2.0, matching the pinned base model and miniVERL

Adapter configuration

The adapter is standard PEFT LoRA with rank 16, alpha 32, dropout 0, and no bias. It targets q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, and down_proj. It was trained with QLoRA (NF4 student loading), bfloat16 computation, gradient checkpointing, and a cosine learning-rate schedule on one NVIDIA GeForce RTX 4080 16 GB.

The exact preparation recipe is preserved at the immutable miniVERL source commit: `recipes/qwen3_1.7b_protocol_teacher_sft.yaml`. It used seed 1234, 256 deterministic hard calculator training tasks, 24 optimizer updates, and eight oracle trajectories per update.

Candidate selection and teacher evaluation

Candidate A was the prespecified primary candidate. Candidates B and C were conditional one-variable fallbacks and were not run because A exceeded the prespecified 50% held-out strict-success gate. Candidate selection used the teacher's own held-out tool-policy evaluation and did not inspect downstream OPD outcomes.

On 24 deterministic held-out hard calculator tasks, candidate A measured:

Protocol-token accuracy is not defined for free-running trajectories because they have no aligned reference token sequence. The full competence record and measurement-status fields are in miniverl_adapter_manifest.json.

Downstream miniVERL result

In miniVERL's prespecified two-seed, equal-optimizer-update calculator benchmark, OPD from this protocol teacher reached 100% strict success for both seeds. Continued SFT also reached 100% for both seeds. The protocol teacher prevented the collapse observed with raw and privileged-context teachers, but protocol-teacher OPD tied rather than beat verified SFT and took approximately six times as much training time on this saturated task. No statistical significance or general OPD advantage is claimed.

The byte-preserved benchmark is documented in `docs/rtx4080-baselines.md`.

Load with PEFT

Pin both repositories to immutable revisions:

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_id = "Qwen/Qwen3-1.7B"
base_revision = "70d244cc86ccca08cf5af4e1e306ecf908b1ad5e"
adapter_id = "DaoyuanLi/mini-verl-qwen3-1.7b-protocol-teacher"
adapter_revision = "de529b22edc9a83a617b276761b942e67ef09019"

tokenizer = AutoTokenizer.from_pretrained(base_id, revision=base_revision)
base = AutoModelForCausalLM.from_pretrained(
    base_id,
    revision=base_revision,
    dtype="auto",
)
teacher = PeftModel.from_pretrained(
    base,
    adapter_id,
    revision=adapter_revision,
    is_trainable=False,
)
teacher.eval()

The adapter revision above is the immutable initial artifact commit. miniVERL's shipped benchmark also pins an immutable Hub commit directly.

Load with miniVERL

Use the adapter as a frozen teacher:

models:
  teacher:
    model_id: Qwen/Qwen3-1.7B
    revision: 70d244cc86ccca08cf5af4e1e306ecf908b1ad5e
    tokenizer_revision: 70d244cc86ccca08cf5af4e1e306ecf908b1ad5e
    adapter:
      source: hub
      path: DaoyuanLi/mini-verl-qwen3-1.7b-protocol-teacher
      revision: de529b22edc9a83a617b276761b942e67ef09019
      require_policy_evaluation: true
      minimum_strict_success_rate: 0.5

miniVERL validates the PEFT layout, base revision, tokenizer fingerprint, checksums, and held-out competence record before loading. After attachment it sets every teacher parameter to requires_grad=False and verifies that no trainable teacher parameter remains.

For offline use, download this repository and set source: local plus path to the downloaded directory.

Integrity

The weights use safetensors, not pickle.

Scope and limitations

• —Validated only on miniVERL's deterministic hard calculator environment and exact tool protocol.
• —Held-out evaluation contains 24 tasks; the downstream benchmark has two seeds and a saturated 24-task test split.
• —Not evaluated for open-domain tool use, safety, factuality, coding, multilingual behavior, or other environments.
• —Inherits the limitations and risks of the Qwen3 base model.
• —Intended for research and reproducibility, not deployment as an autonomous agent.