latimar/Phind-Codellama-34B-v2-exl2

Phind-CodeLlama-34B-v2 EXL2

Weights of Phind-CodeLlama-34B-v2 converted to EXL2 format.

Each separate quant is in a different branch, like in The Bloke's GPTQ repos.

export BRANCH=5_0-bpw-h8
git clone --single-branch --branch ${BRANCH} https://huggingface.co/latimar/Phind-Codellama-34B-v2-exl2

There are the following branches:

5_0-bpw-h8
5_0-bpw-h8-evol-ins
4_625-bpw-h6
4_4-bpw-h8
4_125-bpw-h6
3_8-bpw-h6
2_75-bpw-h6
2_55-bpw-h6

• —Calibration dataset used for conversion: wikitext-v2
• —Evaluation dataset used to calculate perplexity: wikitext-v2
• —Calibration dataset used for conversion of 5_0-bpw-h8-evol-ins: wizardLM-evol-instruct_70k
• —Evaluation dataset used to calculate ppl for Evol-Ins: : nikrosh-evol-instruct
• —When converting 4_4-bpw-h8 quant, additional -mr 32 arg was used.

PPL was measured with the test_inference.py exllamav2 script:

python test_inference.py -m /storage/models/LLaMA/EXL2/Phind-Codellama-34B-v2 -ed /storage/datasets/text/evol-instruct/nickrosh-evol-instruct-code-80k.parquet