mrfakename/refusal

I messed up on the previous model. This is a fixed version.

A tiny 1B model that refuses basically anything you ask it! Trained on the refusal dataset. Prompt format is ChatML.

Training results:

Training hyperparemeters:

The following hyperparameters were used during training:

• —learning_rate: 0.0002
• —trainbatchsize: 2
• —evalbatchsize: 2
• —seed: 42
• —gradientaccumulationsteps: 4
• —totaltrainbatch_size: 8
• —optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• —lrschedulertype: cosine
• —lrschedulerwarmup_steps: 10
• —num_epochs: 4

Base model: https://huggingface.co/TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T