CoolFace
Modelpublic

4x32/Bielik-7B-polish-law

sourceHugging Faceapache-2.0updated 6mo agoView on Hugging Face
0likes
Model Card

Bielik-7B Polish Law Fine-tune (QLoRA)

Fine-tuned version of speakleash/Bielik-7B-Instruct-v0.1 on Polish legal Q&A data using QLoRA (4-bit) with FlashAttention 2.

Model Details

PropertyValue
Base modelspeakleash/Bielik-7B-Instruct-v0.1
ArchitectureMistral-7B
Fine-tuning methodQLoRA (4-bit NF4)
Adapter typeLoRA
LanguagePolish
DomainLegal / Polish law

Training Hardware

ComponentSpec
GPUNVIDIA RTX 3060 12 GB
VRAM usage~10 GB (with FlashAttention 2) / ~10–11 GB (SDPA)
CUDA12.1

Training Parameters

LoRA Configuration

ParameterValue
r (rank)16
lora_alpha32
lora_dropout0.05
biasnone
target_modulesq_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
task_typeCAUSAL_LM

Quantization (BitsAndBytes)

ParameterValue
Quantization4-bit (load_in_4bit=True)
Quant typeNF4
Compute dtypebfloat16
Double quantizationYes (saves ~0.4 GB VRAM)

SFT / Training Config

ParameterValue
Epochs3
Per-device batch size2
Gradient accumulation steps4 (effective batch size: 8)
Learning rate2e-4
LR schedulercosine
Warmup ratio0.05
Max sequence length2048
Precisionbf16
TF32Yes (Ampere GPU benefit)
Optimizerpaged_adamw_8bit
Gradient checkpointingYes (use_reentrant=False)
Group by lengthYes
Dataloader workers4
Seed42
Attention implementationFlashAttention 2 (flash_attention_2)

Software Environment

LibraryVersion
PyTorch2.5.1+cu121
Transformers4.47.0
PEFT0.14.0
TRL0.13.0
BitsAndBytes0.45.0
Datasets3.2.0
Accelerate1.2.1

Dataset

Custom Polish legal Q&A dataset. Each sample is a single text field formatted as an instruction/response pair in Polish.