sandeep123/dapo-qwen3-4b-nonthinking-2048-20260919

Qwen3-4B-Instruct-2507: controlled DAPO adaptation

Fresh LoRA rank 16/alpha 32 on the same 2048-question split used by the STRIDE comparisons. Explicit nonthinking (enable_thinking=False), seed42, eight responses per question, context 8192, and four learner plus four sampler GPUs.

This uses DAPO asymmetric clipping, dynamic filtering/refill of mixed-correctness groups, global token-normalized policy loss, and a soft overlength reward. The exact released-code provenance and implemented equations are recorded below. Unlike the original paper's no-KL recipe, this controlled comparison uses the shared KL coefficient 0.01 against the frozen initial base. This is not an exact reproduction of the paper's scale or training recipe and makes no performance claim.

{
  "version": 1,
  "name": "dapo",
  "author_repository": "volcengine/verl",
  "author_commit": "ed498f9fa5c726a6fb46b19bc59c5c33970053c7",
  "clip_low": 0.2,
  "clip_high": 0.28,
  "ratio_bounds": [
    0.8,
    1.28
  ],
  "dual_clip": 10.0,
  "correctness_reward": "2 * raw_answer_correct - 1",
  "correctness_oracle": "shared verify_relaxed_answer integer, finish_reason=stop; native region validity retained",
  "group_filter": "0 < sum(raw_answer_correct) < group_size; independent of shaped reward",
  "soft_cache_fraction": 0.2,
  "soft_cache_tokens": "max(1, actual_response_budget // 5)",
  "length_penalty": "min(0, (actual_response_budget - cache_tokens - response_tokens) / cache_tokens)",
  "reward": "correctness_reward + length_penalty",
  "advantage": "(reward - group_mean) / (sample_std + 1e-6)",
  "truncated_loss": "all real tokens retained; raw answer correctness has no finish-reason veto",
  "loss_reduction": "global real response token mean, DDP compensated by world_size",
  "kl": "separate fixed-base sampled k3, coefficient recorded by run config",
  "optimizer_updates_per_collection": 1,
  "comparison_limit": "single fresh on-policy update limits opportunity for clip-higher to differ",
  "variant": "controlled_DAPO_plus_KL",
  "paper_kl_coefficient": 0.0,
  "applied_kl_coefficient": 0.01,
  "candidate_batch_size": 64,
  "accepted_batch_size": 64,
  "max_generation_batches": 10,
  "generation_batch_multiplier": 1,
  "released_recipe_generation_batch_multiplier": 3,
  "refill_exhaustion": "fail_before_optimizer_step_resume_last_committed_state",
  "candidate_stream": "seed_plus_candidate_epoch_shuffle_keep_partial",
  "selected_order": "candidate_batch_then_rank_then_original_prompt_order",
  "surplus": "record_and_discard_never_carry_across_policy_updates",
  "policy_during_collection": "fixed_committed_adapter",
  "clip_higher_note": "Single on-policy update limits clipping activation compared with the released multi-minibatch recipe",
  "optimizer_budget": "epochs * updates_per_epoch; epochs are accepted-prompt equivalents",
  "fixed_generated_rollout_budget": false,
  "extra_candidates": "record_all_requests_responses_acceptance_tokens_and_raw_dataset_passes",
  "stride_alpha_used": false,
  "format_reward_used": false
}

Each optimizer update retains 64 question groups × eight responses (512). Dynamic sampling generates additional candidates, which are logged and archived; therefore compute and raw-data exposure exceed the fixed-budget baselines. The 128-update budget matches four ×2048/64 accepted-group epoch equivalents, not four literal passes through candidate questions. LR 2e-5, 10-update linear warmup, then constant. LoRA dropout 0; q/k/v/o and gate/up/down targets.

Completion is represented only by checkpoint_index.json; a planned run is not a completed result. Every optimizer-update adapter is public under immutable checkpoint-NNNNNN/ paths. Current selection: checkpoint-000119. Base model revision: cdbee75f17c01a7cc42f958dc650907174af0554. Load that pinned Qwen model with PEFT and the chosen checkpoint, and render its tokenizer with enable_thinking=False at inference.

Adapters contain safetensors, portable base-model configuration, tokenizer, metadata and SHA256 manifests. Per-update adapters are verified at an immutable Hub commit before their local training copies can be removed. Initial state, each completed 32-update epoch equivalent, and final/controlled-stop full state are uploaded under latest-resume/; prior versions remain reachable by pinned commits. Resume includes AdamW, all four learner RNG states, and dynamic candidate stream state. Follow latest-resume/RESUME.md and verify_resume.py for exact continuation. Keep matching scientific source, dataset hash and runtime versions.

A separate verified worker archives all generated candidate/refill request and response envelopes plus selected scored rollouts under training-archives/. No credentials or unrelated files are published. Node-local caches and a shared 4 GiB per-attempt controlled stop bound local disk growth; publication failure stops the run rather than silently discarding unverified data.

Verified server storage policy

Every optimizer-update LoRA adapter remains public. During normal training, full optimizer/RNG resume backups are uploaded at completed epoch boundaries (updates 32, 64, 96, 128). Initial recovery and controlled stop/final flush also back up the exact latest committed update. The newest two local full states are retained. Older optimizer states are removed as obsolete only after verifying the newest local full state, every adapter backup, and the required public full backup; this does not claim every intermediate optimizer state was uploaded. checkpoint_index.json and latest_resume.json deliberately report separate adapter and full-resume watermarks. See retention_policy.json.