ikawrakow/llama-v1-2bit-gguf

This repository contains 2-bit quantized LLaMA-v1 models in GGUF format for use with llama.cpp. All tensors are quantized with Q2_K, except for output.weight, which is Q6_K, and, in the case of LLaMA-v2-70B, attn_v, which is Q4_K. The quantized models differ from the standard llama.cpp 2-bit quantization in two ways:

• —These are actual 2-bit quantized models instead of the mostly 3-bit quantization provided by the standard llama.cpp Q2_K quantization method
• —The models were prepared with a refined (but not yet published) k-quants quantization approach

The table shows Wikitext perplexities for a context length of 2048 tokens computed with these models using llama.cpp | Model | Perplexity | |---|---| | 7B | 6.4023 | | 13B | 5.3967 | | 30B | 4.5065 | | 65B | 3.9136 |