ruslandev/llama-3-8b-samantha

Uploaded model

• —Developed by: ruslandev
• —License: apache-2.0
• —Finetuned from model : unsloth/llama-3-8b-bnb-4bit

This model is finetuned on the data of Samantha. Prompt format is Alpaca. I used the same system prompt as the original Samantha.

"""Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.

### Instruction:
{SYSTEM_PROMPT}

### Input:
{QUESTION}

### Response:
"""

Training

gptchain framework has been used for training.

Training hyperparameters

• —learning_rate: 2e-4
• —seed: 3407
• —gradientaccumulationsteps: 4
• —perdevicetrainbatchsize: 2
• —optimizer: adamw_8bit
• —lrschedulertype: linear
• —warmup_steps: 5
• —num_epochs: 2
• —weight_decay: 0.01

Training results

2 epoch finetuning from llama-3-8b took 1 hour on a single A100 with Unsloth and Huggingface's TRL library.

<img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" width="200"/>