Modelpublic

RichardErkhov/DevQuasar_-_analytical_reasoning_r16a32_unsloth-Llama-3.2-3B-Instruct-bnb-4bit-4bits

sourceHugging Faceupdated 2y agoView on Hugging Face

0likes11downloads

Model Card

Quantization made by Richard Erkhov.

Github

Discord

Request more models

analyticalreasoningr16a32_unsloth-Llama-3.2-3B-Instruct-bnb-4bit - bnb 4bits

—Model creator: https://huggingface.co/DevQuasar/
—Original model: https://huggingface.co/DevQuasar/analyticalreasoningr16a32_unsloth-Llama-3.2-3B-Instruct-bnb-4bit/

Original model description: --- base_model: unsloth/Llama-3.2-3B-Instruct-bnb-4bit datasets:

—microsoft/orca-agentinstruct-1M-v1 pipelinetag: text-generation libraryname: transformers license: llama3.2 tags:
—unsloth
—transformers model-index:
—name: analyticalreasoningr16a32_unsloth-Llama-3.2-3B-Instruct-bnb-4bit results:
—task: type: text-generation dataset: type: lm-evaluation-harness name: bbh metrics:
—name: accnorm type: accnorm value: 0.4168 verified: false
—task: type: text-generation dataset: type: lm-evaluation-harness name: gpqa metrics:
—name: accnorm type: accnorm value: 0.2691 verified: false
—task: type: text-generation dataset: type: lm-evaluation-harness name: math metrics:
—name: exactmatch type: exactmatch value: 0.0867 verified: false
—task: type: text-generation dataset: type: lm-evaluation-harness name: mmlu metrics:
—name: accnorm type: accnorm value: 0.2822 verified: false
—task: type: text-generation dataset: type: lm-evaluation-harness name: musr metrics:
—name: accnorm type: accnorm value: 0.3648 verified: false
—task: type: text-generation dataset: type: lm-evaluation-harness name: hellaswag metrics:
—name: acc type: acc value: 0.5141 verified: false
—name: accnorm type: accnorm value: 0.6793 verified: false

image/png

Eval

The fine tuned model (DevQuasar/analyticalreasoningr16a32_unsloth-Llama-3.2-3B-Instruct-bnb-4bit) has gained performace over the base model (unsloth/Llama-3.2-3B-Instruct-bnb-4bit) in the following tasks.

Test	Base Model	Fine-Tuned Model	Performance Gain
leaderboardbbhlogicaldeductionseven_objects	0.2520	0.4360	0.1840
leaderboardbbhlogicaldeductionfive_objects	0.3560	0.4560	0.1000
leaderboardmusrteam_allocation	0.2200	0.3200	0.1000
leaderboardbbhdisambiguation_qa	0.3040	0.3760	0.0720
leaderboardgpqadiamond	0.2222	0.2727	0.0505
leaderboardbbhmovie_recommendation	0.5960	0.6360	0.0400
leaderboardbbhformal_fallacies	0.5080	0.5400	0.0320
leaderboardbbhtrackingshuffledobjectsthreeobjects	0.3160	0.3440	0.0280
leaderboardbbhcausal_judgement	0.5455	0.5668	0.0214
leaderboardbbhweboflies	0.4960	0.5160	0.0200
leaderboardmathgeometry_hard	0.0455	0.0606	0.0152
leaderboardmathnumtheoryhard	0.0519	0.0649	0.0130
leaderboardmusrmurder_mysteries	0.5280	0.5400	0.0120
leaderboardgpqaextended	0.2711	0.2802	0.0092
leaderboardbbhsports_understanding	0.5960	0.6040	0.0080
leaderboardmathintermediatealgebrahard	0.0107	0.0143	0.0036

Framework versions

—unsloth 2024.11.5
—trl 0.12.0

Training HW

—V100

I'm doing this to 'Make knowledge free for everyone', using my personal time and resources.

If you want to support my efforts please visit my ko-fi page: https://ko-fi.com/devquasar

Also feel free to visit my website https://devquasar.com/