eric8810/dream-quant

Dream-Coder GGUF Q8_0 Quantization Guide

This guide is specifically designed for GGUF Q8_0 quantization of the Dream-Coder v0-Instruct-7B model.

Quick Start

1. Environment Setup

# 1. Clone and compile llama.cpp
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make -j$(nproc)

# 2. Install Python dependencies
pip install transformers>=4.46.2 torch safetensors numpy

2. Execute Quantization

Method 1: Use the provided script

# Set llama.cpp path
export LLAMA_CPP_PATH=/path/to/llama.cpp

# Run quantization script
./quantize_example.sh

Method 2: Manual execution

python quantize_dream_q8_0.py \
    --model_path /path/to/Dream-Coder-v0-Instruct-7B \
    --llama_cpp_path /path/to/llama.cpp \
    --output_dir ./gguf_output \
    --keep_f16

3. Parameter Description

• —--model_path: Dream-Coder model path (default: current directory)
• —--llama_cpp_path: llama.cpp project path (required)
• —--output_dir: Output directory (default: ./gguf_output)
• —--keep_f16: Keep F16 intermediate files

Architecture Adaptation

Dream-Coder Special Configuration Handling

This quantization script specifically handles the following special configurations of Dream-Coder:

1. 1.Architecture Mapping: DreamModel → LlamaForCausalLM (compatibility)
2. 2.Special Token IDs:
3. 3.mask_token_id: 151666 (critical diffusion token)
4. 4.bos_token_id: 151665
5. 5.eos_token_id: 151643
6. 6.pad_token_id: 151643

1. 1.Model Parameters:
2. 2.Vocabulary size: 152,064
3. 3.Hidden dimension: 3,584
4. 4.Attention heads: 28 (4 key-value heads)
5. 5.Layers: 28
6. 6.Context length: 32,768

1. 1.Diffusion Features:
2. 2.Preserve mask_token_id metadata
3. 3.RoPE theta: 1,000,000.0
4. 4.Activation function: SiLU

Output Description

File Structure

gguf_output/
├── dream-coder-7b-f16.gguf      # F16 intermediate file (optionally kept)
└── dream-coder-7b-q8_0.gguf     # Final Q8_0 quantized file

Performance Expectations

Usage

llama.cpp Command Line

Since Dream-Coder is a diffusion-based model, you need to use the dedicated llama-diffusion-cli tool:

# Basic usage
./llama.cpp/build/bin/llama-diffusion-cli \
    -m gguf_output/dream-coder-7b-q8_0.gguf \
    -p "def quicksort(arr):" \
    -n 512 \
    -c 2048 \
    --diffusion-steps 128

# Advanced parameters
./llama.cpp/build/bin/llama-diffusion-cli \
    -m gguf_output/dream-coder-7b-q8_0.gguf \
    -p "Write a binary search function" \
    -n 256 \
    -c 2048 \
    --temp 0.1 \
    --top-p 0.95 \
    --repeat-penalty 1.1 \
    --diffusion-steps 128 \
    --diffusion-algorithm 4 \
    --diffusion-alg-temp 0.0 \
    -t 8

# Visualize generation process
./llama.cpp/build/bin/llama-diffusion-cli \
    -m gguf_output/dream-coder-7b-q8_0.gguf \
    -p "def fibonacci(n):" \
    -n 256 \
    --diffusion-steps 64 \
    --diffusion-visual

Diffusion Parameter Description

• —--diffusion-steps N: Diffusion denoising steps (default: 128)
• —--diffusion-algorithm N: Algorithm selection:
• —0 = ORIGIN (original algorithm)
• —1 = ENTROPY_BASED (entropy-based)
• —2 = MARGIN_BASED (margin-based)
• —3 = RANDOM (random)
• —4 = LOW_CONFIDENCE (low confidence, default)
• —--diffusion-alg-temp F: Algorithm temperature (default: 0.0)
• —--diffusion-visual: Enable visualization mode, show generation progress
• —--diffusion-eps F: Time step epsilon value

Python (llama-cpp-python)

pip install llama-cpp-python

from llama_cpp import Llama

# Load model
llm = Llama(
    model_path="gguf_output/dream-coder-7b-q8_0.gguf",
    n_ctx=2048,
    n_threads=8,
    n_gpu_layers=0  # CPU inference, set >0 to enable GPU acceleration
)

# Generate code
output = llm(
    "def fibonacci(n):",
    max_tokens=512,
    temperature=0.1,
    top_p=0.95,
    repeat_penalty=1.1
)

print(output['choices'][0]['text'])

With GPU Acceleration

If compiled with CUDA support:

# Compile CUDA version
cd llama.cpp
make clean
make LLAMA_CUBLAS=1 -j$(nproc)

# Use GPU acceleration (partial layers)
./build/bin/llama-diffusion-cli \
    -m gguf_output/dream-coder-7b-q8_0.gguf \
    -p "def quicksort(arr):" \
    -n 512 \
    --diffusion-steps 128 \
    -ngl 20  # Number of GPU layers

Troubleshooting

Common Issues

1. 1.Conversion Failure:
2. 2.Ensure llama.cpp is compiled correctly
3. 3.Check Python dependency versions
4. 4.Verify model file integrity

1. 1.Quantization Failure:
2. 2.Check disk space (~20GB temporary space needed)
3. 3.Ensure sufficient memory (32GB+ recommended)

1. 1.Inference Errors:
2. 2.Verify GGUF file integrity
3. 3.Check context length settings
4. 4.Try reducing n_gpu_layers

Model Validation

# File integrity check
ls -lh gguf_output/dream-coder-7b-q8_0.gguf

# Simple inference test  
echo "def hello():" | ./llama.cpp/build/bin/llama-diffusion-cli -m gguf_output/dream-coder-7b-q8_0.gguf -n 20 --diffusion-steps 64

Performance Optimization

CPU Optimization

• —Use -t parameter to set thread count
• —Enable AVX2/AVX512 compilation options
• —Adjust batch size (-b parameter)

GPU Optimization

• —Use CUDA/OpenCL compilation
• —Adjust GPU layer count (-ngl)
• —Monitor GPU memory usage

Memory Optimization

• —Use --mmap to enable memory mapping
• —Adjust --mlock parameter
• —Set appropriate context length

Important Notes

1. 1.Diffusion Features: Dream-Coder uses diffusion generation, different from traditional autoregressive models
2. 2.Dedicated Tool: Must use llama-diffusion-cli instead of the regular main tool
3. 3.Special Tokens: Maintain correct handling of mask_token_id (151666)
4. 4.Context Length: Supports maximum 32K tokens, but 2K-4K recommended for optimal performance
5. 5.Generation Parameters: Recommend using lower temperature (0.1-0.3) and appropriate top_p (0.9-0.95)
6. 6.Diffusion Steps: Recommend 64-128 steps, more steps may improve quality but increase inference time

Technical Support

If you encounter issues, please check:

1. 1.llama.cpp version and compilation status
2. 2.Python dependency version compatibility
3. 3.Model file integrity
4. 4.System resources (memory/disk)

For more information, refer to:

• —llama.cpp GitHub
• —GGUF Format Documentation