Undi95/Mistral-11B-OmniMix-pippa-sharegpt-11b-qlora

Mistral-11B-OmniMix-pippa-sharegpt-11b-qlora

This is a repository of my Mistral-11B-OmniMix Qlora checkpoints of the PIPPA-ShareGPT dataset.

You can read more about the dataset on its relevant page. It's a ShareGPT reformat of the PIPPA dataset by PygmalionAI. The reformat was done to allow for axolotl compatability.

Architecture

• —Model Architecture: Mistral-11B-OmniMix
• —Training Algorithm: QLora
• —Dataset Used: PIPPA-ShareGPT (pippasharegpttrimmed.jsonl)

Training Details

• —Dataset: PIPPA-ShareGPT
• —Datset type: ShareGPT
• —Training Parameters: See Here
• —Training Environment: Axolotl
• —sequence_len: 8196

Instruct Format

ShareGPT gets converted to vicuna format. The dataset uses modified roles of USER and CHARACTER instead of USER and ASSISTANT.

SYSTEM: Enter roleplay mode...
USER: {prompt}
CHARACTER:

Notes

This Qlora was produced as an experiment to see how the public version of PIPPA can affect a model. Also, Mistral is fairly new and training/finetune can be broken. As a result, I have no idea if this lora is of great quality or absolute garbage and was mean to be only used with OmniMix.

Acknowledgments

Thanks to:

• —PygmalionAI: The creators of the PIPPA dataset
• —Axolotl: Finetuning suite
• —Kingbri: The OG author of this LoRA who helped me a lot

Donate?

If you'd like to donate to Kingbri, you can do so here: https://ko-fi.com/kingbri

If you'd like to donate to me, you can also do it here: https://ko-fi.com/undiai

You should not feel obligated to donate, but if you do, we'll appreciate it.

Axolotl stuff

Training procedure

The following bitsandbytes quantization config was used during training:

• —quant_method: bitsandbytes
• —loadin8bit: False
• —loadin4bit: True
• —llmint8threshold: 6.0
• —llmint8skip_modules: None
• —llmint8enablefp32cpu_offload: False
• —llmint8hasfp16weight: False
• —bnb4bitquant_type: nf4
• —bnb4bitusedoublequant: True
• —bnb4bitcompute_dtype: bfloat16

Training hyperparameters

The following hyperparameters were used during training:

• —learning_rate: 0.0002
• —trainbatchsize: 4
• —evalbatchsize: 4
• —seed: 42
• —gradientaccumulationsteps: 4
• —totaltrainbatch_size: 16
• —optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• —lrschedulertype: cosine
• —lrschedulerwarmup_steps: 10
• —num_epochs: 3

Training results

Framework versions

• —Transformers 4.35.0.dev0
• —Pytorch 2.0.1+cu118
• —Datasets 2.14.5
• —Tokenizers 0.14.1