unsloth/grok-2-GGUF

<div> <p style="margin-bottom: 0; margin-top: 0;"> <strong>Learn how to run Grok 2 correctly - <a href="https://docs.unsloth.ai/basics/grok-2">Read our Guide</a>.</strong> </p> <p style="margin-top: 0;margin-bottom: 0;"> <em><a href="https://docs.unsloth.ai/basics/unsloth-dynamic-v2.0-gguf">Unsloth Dynamic 2.0</a> achieves superior accuracy & outperforms other leading quants.</em> </p> <div style="display: flex; gap: 5px; align-items: center; "> <a href="https://github.com/unslothai/unsloth/"> <img src="https://github.com/unslothai/unsloth/raw/main/images/unsloth%20new%20logo.png" width="133"> </a> <a href="https://discord.gg/unsloth"> <img src="https://github.com/unslothai/unsloth/raw/main/images/Discord%20button.png" width="173"> </a> <a href="https://docs.unsloth.ai/basics/grok-2"> <img src="https://raw.githubusercontent.com/unslothai/unsloth/refs/heads/main/images/documentation%20green%20button.png" width="143"> </a> </div> <h1 style="margin-top: 0rem;">Grok 2 Usage Guidelines</h1> </div>

• —Use --jinja for llama.cpp. You must use PR 15539. For example use the code below: <br>
• —git clone https://github.com/ggml-org/llama.cpp
• —cd llama.cpp && git fetch origin pull/15539/head:MASTER && git checkout MASTER && cd ..

Utilizes Alvaro's Grok-2 HF compatible tokenizer as provided here

Grok 2

This repository contains the weights of Grok 2, a model trained and used at xAI in 2024.

Usage: Serving with SGLang

• —Download the weights. You can replace /local/grok-2 with any other folder name you prefer.

  hf download xai-org/grok-2 --local-dir /local/grok-2

You might encounter some errors during the download. Please retry until the download is successful. If the download succeeds, the folder should contain 42 files and be approximately 500 GB.

• —Launch a server.

Install the latest SGLang inference engine (>= v0.5.1) from https://github.com/sgl-project/sglang/

Use the command below to launch an inference server. This checkpoint is TP=8, so you will need 8 GPUs (each with > 40GB of memory).

  python3 -m sglang.launch_server --model /local/grok-2 --tokenizer-path /local/grok-2/tokenizer.tok.json --tp 8 --quantization fp8 --attention-backend triton

• —Send a request.

This is a post-trained model, so please use the correct chat template.

  python3 -m sglang.test.send_one --prompt "Human: What is your name?<|separator|>\n\nAssistant:"

You should be able to see the model output its name, Grok.

Learn more about other ways to send requests here.

License

The weights are licensed under the Grok 2 Community License Agreement.