Modelpublic

sayhan/OpenHermes-2.5-Strix-Philosophy-Mistral-7B-LoRA

sourceHugging Faceapache-2.0updated 3y agoView on Hugging Face

9likes1.9kdownloads

Model Card

image/png

OpenHermes 2.5 Stix Philosophy Mistral 7B

—Finetuned by: sayhan
—License: apache-2.0
—Finetuned from model : teknium/OpenHermes-2.5-Mistral-7B
—Dataset: sayhan/strix-philosophy-qa --- LoRA rank: 8 LoRA alpha: 16 LoRA dropout: 0 Rank-stabilized LoRA: Yes Number of epochs: 3 Learning rate: 1e-5 Batch size: 2 Gradient accumulation steps: 4 Weight decay: 0.01 Target modules:

  - Query projection (`q_proj`)
  - Key projection (`k_proj`)
  - Value projection (`v_proj`)
  - Output projection (`o_proj`)
  - Gate projection (`gate_proj`)
  - Up projection (`up_proj`)
  - Down projection (`down_proj`)