CoolFace
Modelpublic

tiiuae/Falcon3-10B-Instruct-1.58bit-GGUF

sourceHugging Faceotherupdated 2y agoView on Hugging Face
7likes242downloads
Model Card

image/png

Table of Contents

  1. 1.TL;DR
  2. 2.Model Details
  3. 3.Training Details
  4. 4.Usage
  5. 5.Evaluation
  6. 6.Citation

TL;DR

Model Details

Model Description

  • —Developed by: https://www.tii.ae
  • —Model type: Causal decoder-only - instruct / chat version
  • —Architecture: Pure-transformer - 1.58bit version
  • —Language(s) (NLP): Mainly English
  • —License: TII Falcon License 2.0

Training details

The model has been trained following the training strategies from the recent 1-bit LLM HF blogpost and 1-bit LLM paper. For more details about the training protocol of this model, please refer to the Falcon-3 technical report, section Compression.

Usage

Currently to use this model you can rely on BitNet library. You can also play with the model using the falcon-1.58bit playground (only for the 7B instruct version).

BitNet

git clone https://github.com/microsoft/BitNet && cd BitNet
pip install -r requirements.txt
huggingface-cli download tiiuae/Falcon3-10B-Instruct-1.58bit-GGUF ggml-model-i2_s.gguf --local-dir models/Falcon3-10B-1.58bit/
python run_inference.py -m models/Falcon3-10B-1.58bit/ggml-model-i2_s.gguf -p "You are a helpful assistant" -cnv

Evaluation

We report in the following table our internal pipeline benchmarks:

Note evaluation results are normalized score from v2 leaderboard tasks - reported results of original models in the blogpost are raw scores

<table border="1" style="width: 100%; text-align: center; border-collapse: collapse;"> <colgroup> <col style="width: 10%;"> <col style="width: 10%;"> <col style="background-color: rgba(80, 15, 213, 0.5); width: 7%;"> </colgroup> <thead> <tr> <th>Benchmark</th> <th>Llama3-8B-1.58-100B-tokens</th> <th>Falcon3-10B-Instruct-1.58bit</th> </tr> </thead> <tbody> <tr> <td>IFEval</td> <td>17.91</td> <td><b>54.37</b></td> </tr> <tr> <td>MUSR</td> <td><b>4.87</b></td> <td>2.57</td> </tr> <tr> <td>GPQA</td> <td>1.83</td> <td><b>4.27</b></td> </tr> <tr> <td>BBH</td> <td>5.36</td> <td><b>6.59</b></td> </tr> <tr> <td>MMLU-PRO</td> <td>2.78</td> <td><b>6.62</b></td> </tr> <tr> <td>MATH</td> <td>0.26</td> <td><b>2.44</b></td> </tr> <tr> <td>Average</td> <td>5.5</td> <td><b>12.81</b></td> </tr> </tbody> </table>

Useful links

Citation

If the Falcon3 family of models were helpful to your work, feel free to give us a cite.

@misc{Falcon3,
    title = {The Falcon 3 Family of Open Models},
    author = {Falcon-LLM Team},
    month = {December},
    year = {2024}
}