jsanzolac/bpe_glove_300_qkv_v_only_hardnegs

bpeglove300qkvvonlyhardnegs

QKV-split LoRA student on top of the 300-d cl100k BPE-GloVe (jsanzolac/drifting-glove-distilled-r300).

• —Q = frozen E.
• —K = E + frozen AK·BK, loaded from jsanzolac/bpe_glove_300_lora_r300_qwen3/rank_300/checkpoint_final.pt.
• —V = E + trainable AV·BV (rank=300, full).

Loss: InfoNCE(v_S vs [v_T ‖ v_hards], τ=0.05) + 0.1·MSE(v_S, v_T) with H = 64 mined hard negatives per anchor at batch=256.

Schedule: lr 0.0005 → 1e-05 cosine over 150,000 steps, warmup 1000. Optimizer: AdamW, weight decay 0.01.

Files under rank_300/:

• —checkpoint_final.pt — AV.weight + BV.weight (frozen E, AK, BK NOT included).
• —config.json
• —vectors_drifted_V.txt — E + B_V(A_V(·)) per vocab row (V-side static drift only).
• —train_log.jsonl

To reconstruct the full model at inference: load E + (AK, BK) from jsanzolac/bpe_glove_300_lora_r300_qwen3, load (AV, BV) from this repo, then run the QKV forward pass.