MohammadKhosravi/llama3.1-8b-lora-cefr-steering-52k

Llama-3.1-8B-Instruct LoRA Baseline (52k Steering Dataset)

This repository contains the PEFT LoRA adapter weights trained on the 52,657-sample EFCAMDAT Steering Dataset for explicit CEFR-aligned text generation.

Training Configuration

• —Backbone: meta-llama/Llama-3.1-8B-Instruct
• —LoRA Parameters: $r=16$, $\alpha=32$, dropout $=0.05$
• —Target Modules: All linear attention and MLP projections (q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj)
• —Loss Strategy: Sample-level inverse-frequency weighted causal language modeling loss
• —Dataset Size: 52,657 samples (Stratified 90% Train / 10% Validation)
• —Batch Size: 16 (effective 32 via gradient accumulation)
• —Hardware: 1x NVIDIA A100 40GB

Training Progression Logs

[ { "epoch": 1, "trainloss": 2.0257, "valloss": 1.9005, "valppl": 6.69 }, { "epoch": 2, "trainloss": 1.7041, "valloss": 1.8734, "valppl": 6.51 }, { "epoch": 3, "trainloss": 1.4578, "valloss": 1.9381, "val_ppl": 6.95 } ]

Hardware Profiling

• —Total training time: 15706.42 seconds (4.36 hours)
• —Peak GPU memory: 25.53 GB
• —Average GPU utilization: 97.9%