Modelpublic

psychopenguin/legal_Phi-3-mini-4k-instruct

sourceHugging Facemitupdated 6mo agoView on Hugging Face

0likes15downloads

Model Card

<img src="https://raw.githubusercontent.com/axolotl-ai-cloud/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/> <details><summary>See axolotl config</summary>

axolotl version: 0.16.0.dev0

yaml

seed: 9
auto_resume_from_checkpoints: True
tokenizer_save_jinja_files: True
gradient_checkpointing: True
trust_remote_code: False
tokenizer_use_fast: True
load_best_model_at_end: true

base_model: microsoft/Phi-3-mini-4k-instruct
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer
chat_template: tokenizer_default

datasets: 
  - path: psychopenguin/indian_legal_dataset_qna
    type: alpaca
    split: train

sample_packing: True               
pad_to_sequence_len: True 
sequence_len: 2048
bf16: True
tf32: True
adapter: lora
lora_target_linear: True
lora_r: 16
lora_alpha: 32
lora_dropout: 0.05

flash_attention: True
flash_attn_cross_entropy: True

optimizer: paged_adamw_8bit
learning_rate: 0.0002
lr_scheduler: cosine
gradient_accumulation_steps: 1
micro_batch_size: 9
num_epochs: 5    
neftune_noise_alpha: 5
early_stopping_patience: 5
save_steps: 100
save_total_limit: 3

val_set_size: 0.05
eval_strategy: steps
eval_steps: 100

use_wandb: True
wandb_project: major_project
wandb_name: legal-lm

output_dir: ./final_model
merge_lora: True
hf_use_auth_token: True
hub_model_id: psychopenguin/legal_Phi-3-mini-4k-instruct

</details><br>

legal_Phi-3-mini-4k-instruct

This model is a fine-tuned version of microsoft/Phi-3-mini-4k-instruct on the psychopenguin/indianlegaldataset_qna dataset. It achieves the following results on the evaluation set:

—Loss: 0.9185
—Ppl: 2.5055
—Memory/max Active (gib): 12.9
—Memory/max Allocated (gib): 12.9
—Memory/device Reserved (gib): 22.58

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

—learning_rate: 0.0002
—trainbatchsize: 9
—evalbatchsize: 9
—seed: 9
—optimizer: Use OptimizerNames.PAGEDADAMW8BIT with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
—lrschedulertype: cosine
—lrschedulerwarmup_steps: 35
—training_steps: 1195

Training results

Training Loss	Epoch	Step	Validation Loss	Ppl	Active (gib)	Allocated (gib)	Reserved (gib)
No log	0	0	1.6374	5.1418	12.88	12.88	15.99
1.1167	0.4184	100	1.1282	3.0901	12.9	12.9	22.02
1.0529	0.8368	200	1.0470	2.8491	12.9	12.9	22.58
0.9329	1.2552	300	0.9872	2.6837	12.9	12.9	22.58
0.9142	1.6736	400	0.9427	2.5669	12.9	12.9	22.58
0.7292	2.0921	500	0.9261	2.5246	12.9	12.9	22.58
0.7555	2.5105	600	0.9038	2.4691	12.9	12.9	22.58
0.7313	2.9289	700	0.8775	2.4048	12.9	12.9	22.58
0.6256	3.3473	800	0.8989	2.4569	12.9	12.9	22.58
0.6565	3.7657	900	0.8888	2.4323	12.9	12.9	22.58
0.5440	4.1841	1000	0.9201	2.5095	12.9	12.9	22.58
0.5420	4.6025	1100	0.9185	2.5055	12.9	12.9	22.58

Framework versions

—PEFT 0.18.1
—Transformers 5.3.0
—Pytorch 2.9.1+cu128
—Datasets 4.5.0
—Tokenizers 0.22.2