1jamesthompson1/Qwen3.6-27B-nz-wvs-sampled_response-cluster_1
07
Qwen3.6-27B LoRA — Sampled Response, Cluster 1
This model is a LoRA fine-tune of Qwen/Qwen3.6-27B as part of the AIML589 project.
This adapter is licensed under CC BY-SA 4.0.
Dataset
Fine-tuned on the sampled_response config of the wvs-nz-value-alignment dataset, cluster_1 subpopulation.
Part of the wvs-nz-value-alignment collection.
GPU: NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition · Training time: 45m 13s
Training hyperparameters
Training log
{"loss": 4.542441558837891, "grad_norm": 5.839055061340332, "learning_rate": 7.82608695652174e-05, "entropy": 0.6802574105560779, "mean_token_accuracy": 0.5049308283254504, "num_tokens": 24645.0, "epoch": 0.13245033112582782, "step": 10}
{"eval_loss": 0.701973021030426, "eval_runtime": 18.5313, "eval_samples_per_second": 8.094, "eval_steps_per_second": 2.051, "eval_entropy": 0.5663144029676914, "eval_mean_token_accuracy": 0.7451597781557786, "eval_num_tokens": 24645.0, "epoch": 0.13245033112582782, "step": 10}
{"loss": 0.48006811141967776, "grad_norm": 1.5653640031814575, "learning_rate": 0.00016521739130434784, "entropy": 0.4913419775664806, "mean_token_accuracy": 0.8086332559585572, "num_tokens": 48985.0, "epoch": 0.26490066225165565, "step": 20}
{"eval_loss": 0.5742780566215515, "eval_runtime": 18.5413, "eval_samples_per_second": 8.09, "eval_steps_per_second": 2.049, "eval_entropy": 0.4024900254842482, "eval_mean_token_accuracy": 0.7162907311790868, "eval_num_tokens": 48985.0, "epoch": 0.26490066225165565, "step": 20}
{"loss": 0.405932092666626, "grad_norm": 1.8248214721679688, "learning_rate": 0.00019414634146341464, "entropy": 0.3804429121315479, "mean_token_accuracy": 0.8386667922139168, "num_tokens": 73226.0, "epoch": 0.3973509933774834, "step": 30}
{"eval_loss": 0.4871271252632141, "eval_runtime": 18.5655, "eval_samples_per_second": 8.08, "eval_steps_per_second": 2.047, "eval_entropy": 0.401295748490252, "eval_mean_token_accuracy": 0.7628289508192163, "eval_num_tokens": 73226.0, "epoch": 0.3973509933774834, "step": 30}
{"loss": 0.41707577705383303, "grad_norm": 1.1599860191345215, "learning_rate": 0.00018439024390243903, "entropy": 0.4035234354436398, "mean_token_accuracy": 0.8250533357262612, "num_tokens": 97170.0, "epoch": 0.5298013245033113, "step": 40}
{"eval_loss": 0.497173011302948, "eval_runtime": 18.5701, "eval_samples_per_second": 8.078, "eval_steps_per_second": 2.046, "eval_entropy": 0.37418799690510096, "eval_mean_token_accuracy": 0.7913220603215066, "eval_num_tokens": 97170.0, "epoch": 0.5298013245033113, "step": 40}
{"loss": 0.3655890941619873, "grad_norm": 1.1802726984024048, "learning_rate": 0.00017463414634146342, "entropy": 0.32736023105680945, "mean_token_accuracy": 0.8766310185194015, "num_tokens": 121386.0, "epoch": 0.6622516556291391, "step": 50}
{"eval_loss": 0.5745805501937866, "eval_runtime": 18.4802, "eval_samples_per_second": 8.117, "eval_steps_per_second": 2.056, "eval_entropy": 0.359674892829437, "eval_mean_token_accuracy": 0.7820018827915192, "eval_num_tokens": 121386.0, "epoch": 0.6622516556291391, "step": 50}
{"loss": 0.3125191926956177, "grad_norm": 1.571219563484192, "learning_rate": 0.00016487804878048782, "entropy": 0.32015694193542005, "mean_token_accuracy": 0.8709339603781701, "num_tokens": 145580.0, "epoch": 0.7947019867549668, "step": 60}
{"eval_loss": 0.6828431487083435, "eval_runtime": 18.6526, "eval_samples_per_second": 8.042, "eval_steps_per_second": 2.037, "eval_entropy": 0.29719744691331135, "eval_mean_token_accuracy": 0.7787124091073087, "eval_num_tokens": 145580.0, "epoch": 0.7947019867549668, "step": 60}
{"loss": 0.2412199020385742, "grad_norm": 2.2911460399627686, "learning_rate": 0.0001551219512195122, "entropy": 0.24680944327265025, "mean_token_accuracy": 0.8969683617353439, "num_tokens": 170643.0, "epoch": 0.9271523178807947, "step": 70}
{"eval_loss": 0.7515808939933777, "eval_runtime": 18.6193, "eval_samples_per_second": 8.056, "eval_steps_per_second": 2.041, "eval_entropy": 0.25726634425748335, "eval_mean_token_accuracy": 0.8145990010939146, "eval_num_tokens": 170643.0, "epoch": 0.9271523178807947, "step": 70}
{"loss": 0.22225503921508788, "grad_norm": 1.4264882802963257, "learning_rate": 0.0001453658536585366, "entropy": 0.197212149732207, "mean_token_accuracy": 0.91331253239983, "num_tokens": 193746.0, "epoch": 1.0529801324503312, "step": 80}
{"eval_loss": 0.6387484073638916, "eval_runtime": 18.612, "eval_samples_per_second": 8.059, "eval_steps_per_second": 2.042, "eval_entropy": 0.23363343227385103, "eval_mean_token_accuracy": 0.8049185511312986, "eval_num_tokens": 193746.0, "epoch": 1.0529801324503312, "step": 80}
{"loss": 0.11078678369522095, "grad_norm": 0.8206525444984436, "learning_rate": 0.000135609756097561, "entropy": 0.15991744147613646, "mean_token_accuracy": 0.9533217743039131, "num_tokens": 218576.0, "epoch": 1.185430463576159, "step": 90}
{"eval_loss": 0.9373728632926941, "eval_runtime": 18.635, "eval_samples_per_second": 8.049, "eval_steps_per_second": 2.039, "eval_entropy": 0.15989923933949812, "eval_mean_token_accuracy": 0.7970238139754847, "eval_num_tokens": 218576.0, "epoch": 1.185430463576159, "step": 90}
{"loss": 0.11687159538269043, "grad_norm": 1.490201711654663, "learning_rate": 0.00012585365853658536, "entropy": 0.09102535207057372, "mean_token_accuracy": 0.9570705264806747, "num_tokens": 243344.0, "epoch": 1.3178807947019868, "step": 100}
{"eval_loss": 0.8093208074569702, "eval_runtime": 18.5071, "eval_samples_per_second": 8.105, "eval_steps_per_second": 2.053, "eval_entropy": 0.13149637486295482, "eval_mean_token_accuracy": 0.8271773222245669, "eval_num_tokens": 243344.0, "epoch": 1.3178807947019868, "step": 100}
{"loss": 0.11526482105255127, "grad_norm": 0.6740375757217407, "learning_rate": 0.00011609756097560975, "entropy": 0.07816403829492628, "mean_token_accuracy": 0.9651808857917785, "num_tokens": 267786.0, "epoch": 1.4503311258278146, "step": 110}
{"eval_loss": 0.7161492705345154, "eval_runtime": 18.5373, "eval_samples_per_second": 8.092, "eval_steps_per_second": 2.05, "eval_entropy": 0.17198755587206074, "eval_mean_token_accuracy": 0.831203010521437, "eval_num_tokens": 267786.0, "epoch": 1.4503311258278146, "step": 110}
{"loss": 0.0833204448223114, "grad_norm": 0.7950387001037598, "learning_rate": 0.00010634146341463416, "entropy": 0.0967397745582275, "mean_token_accuracy": 0.9726294353604317, "num_tokens": 292317.0, "epoch": 1.5827814569536423, "step": 120}
{"eval_loss": 0.733370840549469, "eval_runtime": 18.511, "eval_samples_per_second": 8.103, "eval_steps_per_second": 2.053, "eval_entropy": 0.13129894957436541, "eval_mean_token_accuracy": 0.8290100285881444, "eval_num_tokens": 292317.0, "epoch": 1.5827814569536423, "step": 120}
{"loss": 0.04042904376983643, "grad_norm": 0.7554147243499756, "learning_rate": 9.658536585365854e-05, "entropy": 0.047646277607418595, "mean_token_accuracy": 0.9857919424772262, "num_tokens": 316521.0, "epoch": 1.7152317880794703, "step": 130}
{"eval_loss": 1.0098978281021118, "eval_runtime": 18.5651, "eval_samples_per_second": 8.08, "eval_steps_per_second": 2.047, "eval_entropy": 0.09740110344596599, "eval_mean_token_accuracy": 0.8333959940232729, "eval_num_tokens": 316521.0, "epoch": 1.7152317880794703, "step": 130}
{"loss": 0.008739496767520904, "grad_norm": 0.23848742246627808, "learning_rate": 8.682926829268293e-05, "entropy": 0.027490738761844113, "mean_token_accuracy": 0.9986842110753059, "num_tokens": 340919.0, "epoch": 1.847682119205298, "step": 140}
{"eval_loss": 1.2972569465637207, "eval_runtime": 18.6353, "eval_samples_per_second": 8.049, "eval_steps_per_second": 2.039, "eval_entropy": 0.07602066293015684, "eval_mean_token_accuracy": 0.806203012403689, "eval_num_tokens": 340919.0, "epoch": 1.847682119205298, "step": 140}
{"loss": 0.01962348073720932, "grad_norm": 0.09159229695796967, "learning_rate": 7.707317073170732e-05, "entropy": 0.0183564211052726, "mean_token_accuracy": 0.9902372866868973, "num_tokens": 364991.0, "epoch": 1.980132450331126, "step": 150}
{"eval_loss": 1.450591802597046, "eval_runtime": 18.591, "eval_samples_per_second": 8.068, "eval_steps_per_second": 2.044, "eval_entropy": 0.061529136898523, "eval_mean_token_accuracy": 0.806203012403689, "eval_num_tokens": 364991.0, "epoch": 1.980132450331126, "step": 150}
{"loss": 0.02778538167476654, "grad_norm": 0.008160561323165894, "learning_rate": 6.731707317073171e-05, "entropy": 0.009240388543970001, "mean_token_accuracy": 0.9909774410097223, "num_tokens": 387642.0, "epoch": 2.1059602649006623, "step": 160}
{"eval_loss": 1.4579640626907349, "eval_runtime": 18.6743, "eval_samples_per_second": 8.032, "eval_steps_per_second": 2.035, "eval_entropy": 0.05447787115668976, "eval_mean_token_accuracy": 0.7963345913510573, "eval_num_tokens": 387642.0, "epoch": 2.1059602649006623, "step": 160}
{"loss": 0.029592815041542053, "grad_norm": 0.14256024360656738, "learning_rate": 5.756097560975609e-05, "entropy": 0.01342463267792482, "mean_token_accuracy": 0.9940184816718102, "num_tokens": 412538.0, "epoch": 2.23841059602649, "step": 170}
{"eval_loss": 1.3962903022766113, "eval_runtime": 18.4901, "eval_samples_per_second": 8.112, "eval_steps_per_second": 2.055, "eval_entropy": 0.060239105289311784, "eval_mean_token_accuracy": 0.803148500229183, "eval_num_tokens": 412538.0, "epoch": 2.23841059602649, "step": 170}
{"loss": 0.004358197003602982, "grad_norm": 0.2387428879737854, "learning_rate": 4.7804878048780485e-05, "entropy": 0.013339337654178962, "mean_token_accuracy": 0.9980769231915474, "num_tokens": 437055.0, "epoch": 2.370860927152318, "step": 180}
{"eval_loss": 1.4089089632034302, "eval_runtime": 18.6222, "eval_samples_per_second": 8.055, "eval_steps_per_second": 2.041, "eval_entropy": 0.059272560578967, "eval_mean_token_accuracy": 0.8072211820828287, "eval_num_tokens": 437055.0, "epoch": 2.370860927152318, "step": 180}
{"loss": 0.005107241496443748, "grad_norm": 0.0845877155661583, "learning_rate": 3.804878048780488e-05, "entropy": 0.00872597399429651, "mean_token_accuracy": 0.997916667163372, "num_tokens": 461493.0, "epoch": 2.5033112582781456, "step": 190}
{"eval_loss": 1.3773205280303955, "eval_runtime": 18.5448, "eval_samples_per_second": 8.089, "eval_steps_per_second": 2.049, "eval_entropy": 0.05987703997801107, "eval_mean_token_accuracy": 0.8110589008582266, "eval_num_tokens": 461493.0, "epoch": 2.5033112582781456, "step": 190}
{"loss": 0.0036820050328969954, "grad_norm": 0.0544927753508091, "learning_rate": 2.8292682926829267e-05, "entropy": 0.008151720702880994, "mean_token_accuracy": 0.997916667163372, "num_tokens": 485528.0, "epoch": 2.6357615894039736, "step": 200}
{"eval_loss": 1.3704338073730469, "eval_runtime": 18.6574, "eval_samples_per_second": 8.04, "eval_steps_per_second": 2.037, "eval_entropy": 0.05895511366390182, "eval_mean_token_accuracy": 0.8116071475179572, "eval_num_tokens": 485528.0, "epoch": 2.6357615894039736, "step": 200}
{"loss": 0.0019064834341406822, "grad_norm": 0.011253349483013153, "learning_rate": 1.853658536585366e-05, "entropy": 0.005897045589517802, "mean_token_accuracy": 0.9982142850756646, "num_tokens": 510081.0, "epoch": 2.7682119205298013, "step": 210}
{"eval_loss": 1.3837242126464844, "eval_runtime": 18.5286, "eval_samples_per_second": 8.096, "eval_steps_per_second": 2.051, "eval_entropy": 0.05662573658628389, "eval_mean_token_accuracy": 0.8094141640161213, "eval_num_tokens": 510081.0, "epoch": 2.7682119205298013, "step": 210}
{"loss": 0.006096714362502098, "grad_norm": 0.04618639871478081, "learning_rate": 8.780487804878048e-06, "entropy": 0.00921966262976639, "mean_token_accuracy": 0.9928314402699471, "num_tokens": 534422.0, "epoch": 2.9006622516556293, "step": 220}
{"eval_loss": 1.3900940418243408, "eval_runtime": 18.7877, "eval_samples_per_second": 7.984, "eval_steps_per_second": 2.023, "eval_entropy": 0.05583108028115116, "eval_mean_token_accuracy": 0.8116071475179572, "eval_num_tokens": 534422.0, "epoch": 2.9006622516556293, "step": 220}
{"eval_loss": 1.396872878074646, "eval_runtime": 19.0916, "eval_samples_per_second": 7.857, "eval_steps_per_second": 1.99, "eval_entropy": 0.055053529700288845, "eval_mean_token_accuracy": 0.8094141640161213, "eval_num_tokens": 552555.0, "epoch": 3.0, "step": 228}
{"train_runtime": 2713.3061, "train_samples_per_second": 1.333, "train_steps_per_second": 0.084, "total_flos": 1.0254994554203136e+17, "train_loss": 0.3316960387716168, "epoch": 3.0, "step": 228}Environment
Intended use
This adapter is intended for research purposes only as part of the AIML589 project, which investigates value alignment of LLMs with New Zealand population distributions from the World Values Survey.
Out-of-scope
This model has not been safety-tuned for general-purpose deployment. It should not be used in production systems, for making decisions about people, or in contexts where reliability and safety are critical.
Limitations and biases
- Fine-tuned on a single WVS wave (Wave 7) for New Zealand only.
- The training data reflects the values of those who responded to the survey and may not represent all New Zealanders.
- LoRA adapters are subject to the limitations and biases of the base model (Qwen/Qwen3.6-27B).