mozilla-ai/TriLM-llamafile

TriLM - llamafile

This is a 1.58 bit ternary LLM whose weights consist of {-1, 0, +1}. It's highly optimized for CPU performance, thanks to the `Q2_K_S` quantization format.

• —Model creator: SpectraSuite
• —Original model: TriLMs-Unpacked

This repository packages and distributes TriLM as executable weights, which we call llamafiles. The files you download here will run on Linux, MacOS, Windows, FreeBSD, OpenBSD, and NetBSD for AMD64 and ARM64.

Quickstart

Running the following on a desktop OS will launch a tab in your web browser with a completions interface.

wget https://huggingface.co/Mozilla/TriLM-llamafile/resolve/main/TriLM_3.9B.llamafile
chmod +x TriLM_3.9B.llamafile
./TriLM_3.9B.llamafile

You can also use the command line interface:

./TriLM_3.9B.llamafile -p "this is my prompt"

For further information, please see the llamafile README.

Having trouble? See the "Gotchas" section of the README.

Prompting

This is a base model. It hasn't been fine-tuned for chat. It's recommended that the completions interface be used.

It's recommended with the smaller TriLM models (e.g. 99M) that a high repeat penalty be set, e.g. --repeat-penalty 10. When using the CLI mode, this flag is specified by default in the .args file embedded within the llamafiles from this repo.

Benchmarks

About llamafile

llamafile is a new format introduced by Mozilla Ocho on Nov 20th 2023. It uses Cosmopolitan Libc to turn LLM weights into runnable llama.cpp binaries that run on the stock installs of six OSes for both ARM64 and AMD64.

TriLM 3.9B Unpacked

TriLM (ternary model), unpacked to FP16 format - compatible with FP16 GEMMs. After unpacking, TriLM has the same architecture as LLaMa.

import transformers as tf, torch
model_name = "SpectraSuite/TriLM_3.9B_Unpacked"

# Please adjust the temperature, repetition penalty, top_k, top_p and other sampling parameters according to your needs.
pipeline = tf.pipeline("text-generation", model=model_id, model_kwargs={"torch_dtype": torch.float16}, device_map="auto")

# These are base (pretrained) LLMs that are not instruction and chat tuned. You may need to adjust your prompt accordingly.
pipeline("Once upon a time")

• —License: Apache 2.0
• —We will use our GitHub repo for communication (including HF repo related queries). Feel free to open an issue here https://github.com/NolanoOrg/SpectraSuite