D1zzYzz/GRIT-BOOLQ-QLORA-llama-3.2-3B-Energy-0.99

meta-llama/Llama-3.2-3B Fine-tuned with GRIT and Lora

This model is a fine-tuned version of meta-llama/Llama-3.2-3B using the GRIT (Geometric Reprojection Instruction Tuning) algorithm and LoRA on the google/boolq dataset.

The base model is quantized to 4-bit (NF4) and optimized with Unsloth to enable efficient fine-tuning.

🚀 Training Details

GRIT Algorithm

• —K-FAC Updates: Every 10 steps (adaptive) for second-order preconditioning.
• —Neural Reprojection: Every 20 steps (adaptive) for rank optimization.
• —Rank Adaptation: Enabled (Threshold: 0.99, Min Rank: 4).
• —Optimized LoRA Modules: ['qproj', 'kproj', 'vproj', 'oproj', 'gateproj', 'upproj', 'down_proj']

Fine-tuning Configuration

• —Base Model: meta-llama/Llama-3.2-3B
• —Quantization: 4-bit (NF4) with bf16 compute.
• —LoRA Rank: 16
• —LoRA Alpha: 32
• —Batch Size: 8 (per device)
• —Gradient Accumulation: 4 (Effective batch = 32)
• —Learning Rate: 2.0e-05
• —Precision: bf16 mixed precision
• —Sequence Length: 1024 tokens
• —Gradient Checkpointing: Enabled

Performance Improvements

• —✅ Faster Convergence: K-FAC preconditioning aligns updates with curvature.
• —✅ Adaptive Rank: Dynamically prunes LoRA rank to improve parameter efficiency.

📊 Training Metrics

• —Total Steps: 295
• —Final Loss: 0.318148963734255
• —Trainable Params: 24,313,856

📝 Algorithm Details

• —K-FAC Preconditioning (Natural Gradient) and Neural Reprojection as per GRIT method.
• —Memory Efficient: Covariance matrices on CPU to reduce GPU load.

🏆 Results

In benchmark comparisons, GRIT has shown faster convergence and better stability than standard LoRA or fine-tuning, making it well-suited for efficient single-epoch training. The use of Unsloth further accelerates this process.

📝 Citation

If you use this model, please cite the original GRIT paper and:

@misc{grit-lora-Llama-3.2-3B-boolq},
  title={ meta-llama/Llama-3.2-3B Fine-tuned with GRIT on google/boolq },
  author={D1zzYzz},
  year={2025},
  publisher={Hugging Face},
  url={https://huggingface.co/D1zzYzz/GRIT-BOOLQ-QLORA-llama-3.2-3B-Energy-0.99}
}

⚖️ License

This model inherits the Apache 2.0 license.