Jeesup/llama32-1B-qqp-int8-lora-seed43

llama32-1B-qqp-int8-lora-seed43

LoRA adapter trained on GLUE QQP on top of a int8 backbone of meta-llama/Llama-3.2-1B.

Part of a controlled study of whether the backbone bit-width changes what a LoRA adapter learns. For a given (model size, seed) the adapter initialisation is identical across the bf16 / int8 / nf4 arms, and the data order, optimiser, schedule and LoRA hyperparameters are held fixed — so any difference in the learned update is attributable to the backbone.

Result

Test-set majority-class baseline: 0.6385

• —peak GPU memory: 3.94 GiB
• —training time: 48.3 min (936 steps)
• —GPU: NVIDIA GeForce RTX 4090

Setup

• —seed: 43 · adapter init: shared:lora_init_1B_seed43.pt:128tensors
• —LoRA: r=16, alpha=32, dropout=0.0, bias=none, targetmodules=['qproj', 'kproj', 'vproj', 'o_proj']
• —trainable params: 3,407,872
• —epochs 3, lr 0.0002, maxlen 256, batch 4 x gradaccum 16, cosine schedule, warmup 0.03

Prompt format

Trained as causal LM with the loss on the answer letter only (prompt tokens masked to -100):

Question 1: ...
Question 2: ...

Are these two questions semantically equivalent?

A. Not equivalent
B. Equivalent

Answer:

Evaluated by conditional likelihood over the answer letters (Not equivalent, Equivalent).

GLUE test is unlabeled, so the official validation split is used as TEST and the validation set is carved from train (disjoint).