continual-internalization/kh-disc-qwen3-30b-a3b-200tok-first-run

Knowledge Horizon DiSC LoRA (Qwen3-30B-A3B-Instruct-2507, 200-token data)

This repository contains a LoRA adapter trained with the DiSC pipeline for continual internalization experiments in Knowledge Horizon.

The adapter was trained on a 200-token-budget version of the easy QA training set and evaluated on easy train/test and hard questions.

Contents

• —adapter_model.safetensors (LoRA weights)
• —adapter_config.json
• —tokenizer files copied from training output (tokenizer.json, tokenizer_config.json, chat_template.jinja)
• —optional intermediate checkpoints:
• —checkpoint-400
• —checkpoint-493 (end of epoch)

Base Model

• —Base model: Qwen/Qwen3-30B-A3B-Instruct-2507
• —This repo is an adapter, not a standalone full model.

Run Snapshot (Published Adapter)

• —Method: DiSC (3-stage pipeline, suffix-only forward KL in stage 3)
• —Slurm job: 6537205 (kh_disc_30b_a2)
• —Date: 2026-04-04 (America/New_York)
• —Cluster/partition: Princeton ailab
• —Hardware: 1 node, 2x NVIDIA H200
• —Wall-clock elapsed: 00:25:52
• —Final output dir: checkpoints/disc_lora__qwen3-30b-a3b__knowledge-horizon__6537205

Training Data

The stage-3 training input came from the following chain:

1. 1.Easy QA train split (200-token budget):
2. 2.data/easy_qa_200tok_train.jsonl (1098 rows)
3. 3.Prepared training parquet:
4. 4.python prepare_training_data.py --input data/easy_qa_200tok_train.jsonl --output data/training_data.parquet
5. 5.output rows: 1098
6. 6.DiSC stage-1 split generation:
7. 7.input rows: 1098
8. 8.unique documents after dedupe: 197
9. 9.output split rows: 985 (stage1_splits.parquet)
10. 10.DiSC stage-2 teacher scoring:
11. 11.scored rows: 985 (stage2_scored.parquet)
12. 12.skipped empty: 0
13. 13.skipped too long: 0

Hard QA files are used for evaluation, not for training.

Training Procedure

Stage 1: split contexts

Executed with:

python disc_stage1_prepare.py \
  --input data/training_data.parquet \
  --output runs/disc_qwen3_30b_ailab2_6537205/stage1_splits.parquet \
  --k_splits 5 \
  --min_sentences 3 \
  --dedupe_by_article_text

Important defaults:

• —stage-1 seed: 42
• —split sampling: k-1 random interior split points + final sentence endpoint

Stage 2: teacher top-k scoring

Executed with:

python disc_stage2_score.py \
  --model models/Qwen3-30B-A3B-Instruct-2507 \
  --input runs/disc_qwen3_30b_ailab2_6537205/stage1_splits.parquet \
  --output runs/disc_qwen3_30b_ailab2_6537205/stage2_scored.parquet \
  --tp 2 \
  --max_model_len 4096 \
  --max_num_batched_tokens 2048 \
  --max_num_seqs 2 \
  --gpu_memory_utilization 0.88 \
  --disable_custom_all_reduce true \
  --enforce_eager true \
  --top_k 128 \
  --max_suffix_tokens 256 \
  --batch_size 1

Stage 3: LoRA training (DiSC objective)

Executed with:

torchrun \
  --nproc-per-node 2 \
  --master_port <job_specific_port> \
  disc_stage3_train.py \
  --model_name models/Qwen3-30B-A3B-Instruct-2507 \
  --train_file runs/disc_qwen3_30b_ailab2_6537205/stage2_scored.parquet \
  --output_dir checkpoints/disc_lora__qwen3-30b-a3b__knowledge-horizon__6537205 \
  --fsdp_config configs/fsdp_config_qwen3_moe.json \
  --lora_r 16 \
  --lora_alpha 32 \
  --lora_dropout 0.1 \
  --lora_target_modules all-linear \
  --learning_rate 1.5e-5 \
  --weight_decay 0.01 \
  --adam_beta1 0.9 \
  --adam_beta2 0.999 \
  --adam_epsilon 1e-8 \
  --num_train_epochs 1 \
  --per_device_train_batch_size 1 \
  --gradient_accumulation_steps 1 \
  --warmup_ratio 0.0 \
  --lr_scheduler_type linear \
  --precision bf16 \
  --temperature 2.0 \
  --save_steps 200 \
  --report_to none \
  --resume_from_checkpoint latest

FSDP config (configs/fsdp_config_qwen3_moe.json):

{
  "transformer_layer_cls_to_wrap": "Qwen3MoeDecoderLayer",
  "use_orig_params": true,
  "sync_module_states": true,
  "activation_checkpointing": false,
  "limit_all_gathers": true
}

Final stage-3 stats

• —Train rows: 985
• —Trainable params: 13,369,344 / 30,545,491,968 (0.0438%)
• —train_runtime: 816.4s
• —train_steps: 493
• —train_steps_per_second: 0.604
• —train_loss: 0.5464
• —epoch: 1.0

Evaluation

Evaluation used:

• —base model: models/Qwen3-30B-A3B-Instruct-2507
• —adapter: this checkpoint
• —eval splits:
• —easy train: 1098
• —easy test: 1098
• —hard v2: 248
• —hard v1 in this run: 0

Main results (heuristic from evaluate.py)

S = strong match, IDK = explicit "I don't know", O = other.

Reproducibility Checklist

• —Dataset preparation command (included above)
• —Exact stage 1/2/3 commands (included above)
• —Hardware and partition (included)
• —Key config files:
• —prepare_training_data.py
• —disc_stage1_prepare.py
• —disc_stage2_score.py
• —disc_stage3_train.py
• —configs/fsdp_config_qwen3_moe.json
• —slurm/train_disc_lora_qwen3_30b_ailab_2gpu.sh
• —Repo snapshot at publication time:
• —git rev-parse HEAD = c19506c82f4aed88daba20fabe21d8f0f75b25d6

Software Environment

Observed environment in this workspace:

• —Python 3.10
• —torch==2.9.0+cu128
• —transformers==5.5.0.dev0
• —peft==0.17.1
• —datasets==4.3.0
• —vllm==0.12.0

Usage

Transformers + PEFT

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_id = "Qwen/Qwen3-30B-A3B-Instruct-2507"
adapter_id = "<this-repo>"

tokenizer = AutoTokenizer.from_pretrained(base_id, trust_remote_code=True)
base = AutoModelForCausalLM.from_pretrained(
    base_id,
    trust_remote_code=True,
    torch_dtype="auto",
    device_map="auto",
)
model = PeftModel.from_pretrained(base, adapter_id)
model.eval()

vLLM LoRA

Use vLLM with LoRA enabled and this adapter as the LoRA path.

Limitations

• —This is a research adapter trained for continual internalization experiments, not a general-purpose instruction-tuning release.
• —Evaluation uses a heuristic string-matching scorer in evaluate.py; treat scores as directional.

Citation

If you use this adapter, please cite:

1. 1.The DiSC paper:

@article{padmanabhan2026updating,
  title={Updating Parametric Knowledge with Context Distillation Retains Post-Training Capabilities},
  year={2026},
  eprint={2602.16093},
  archivePrefix={arXiv},
  primaryClass={cs.CL}
}

1. 1.The OPSD/Knowledge Horizon context paper:

@article{shenfeld2026self,
  title={Self-Distillation Enables Continual Learning},
  year={2026},
  eprint={2601.19897},
  archivePrefix={arXiv},
  primaryClass={cs.LG}
}