arcee-ai/WitchLM-1.5B

<!-- This model card has been generated automatically according to the information the Trainer had access to. You should probably proofread and complete it, then remove this comment. -->

<img src="https://raw.githubusercontent.com/axolotl-ai-cloud/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>

Model description

WitchLM is cool!

Benchmarks

<details> "leaderboard": { "instlevelstrictacc,none": 0.33573141486810554, "instlevelstrictaccstderr,none": "N/A", "instlevellooseacc,none": 0.39568345323741005, "instlevellooseaccstderr,none": "N/A", "accnorm,none": 0.3493319496692178, "accnormstderr,none": 0.005120138265236575, "acc,none": 0.24418218085106383, "accstderr,none": 0.003916649280281885, "exactmatch,none": 0.04078549848942598, "exactmatchstderr,none": 0.005354025092648956, "promptlevelstrictacc,none": 0.1977818853974122, "promptlevelstrictaccstderr,none": 0.01714125471908492, "promptlevellooseacc,none": 0.25693160813308685, "promptlevellooseaccstderr,none": 0.018802962575636854, "alias": "leaderboard" }, "leaderboardbbh": { "accnorm,none": 0.3591390383613956, "accnormstderr,none": 0.0058684522608536275, "alias": " - leaderboardbbh" }, "leaderboardgpqa": { "accnorm,none": 0.29194630872483224, "accnormstderr,none": 0.013178882651123217, "alias": " - leaderboardgpqa" }, "leaderboardifeval": { "promptlevelstrictacc,none": 0.1977818853974122, "promptlevelstrictaccstderr,none": 0.01714125471908492, "instlevelstrictacc,none": 0.33573141486810554, "instlevelstrictaccstderr,none": "N/A", "promptlevellooseacc,none": 0.25693160813308685, "promptlevellooseaccstderr,none": 0.018802962575636854, "instlevellooseacc,none": 0.39568345323741005, "instlevellooseaccstderr,none": "N/A", "alias": " - leaderboardifeval" }, "leaderboardmathhard": { "exactmatch,none": 0.04078549848942598, "exactmatchstderr,none": 0.005354025092648956, "alias": " - leaderboardmathhard" }, "leaderboardmmlupro": { "acc,none": 0.24418218085106383, "accstderr,none": 0.003916649280281885, "alias": " - leaderboardmmlupro" }, "leaderboardmusr": { "accnorm,none": 0.36507936507936506, "accnormstderr,none": 0.01715613678641816, "alias": " - leaderboardmusr" } </details>

Training hyperparameters

<details>

The following hyperparameters were used during training:

• —learning_rate: 5e-05
• —trainbatchsize: 4
• —evalbatchsize: 4
• —seed: 42
• —distributed_type: multi-GPU
• —num_devices: 4
• —gradientaccumulationsteps: 4
• —totaltrainbatch_size: 64
• —totalevalbatch_size: 16
• —optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• —lrschedulertype: cosine
• —lrschedulerwarmup_steps: 50
• —num_epochs: 5 </details>

Framework versions

• —Transformers 4.44.0
• —Pytorch 2.3.1+cu121
• —Datasets 2.20.0
• —Tokenizers 0.19.1