bytebarde/TinyLlama-sft-lora-alpaca

Model Card for Model ID

<!-- Provide a quick summary of what the model is/does. --> TinyLlama/TinyLlama-1.1B-Chat-v1.0 sft on alpaca dataset using LoRA

Model Details

Model Sources [optional]

<!-- Provide the basic links for the model. -->

• —Repository: https://github.com/bytebarde/llm_alpaca

Training Details

Training Procedure

<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->

Training Hyperparameters

• —Training regime: [fp16 mixed precision] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
• —Per device train batch size: 4
• —Epoch: 10
• —Loss: 0.9044

Framework versions

• —PEFT 0.7.1