kurai021/Llama-3.2-3B-Instruct-4bit-fable5

Llama-3.2-3B-Instruct-4bit-fable5 (Upcasted to FP16)

This repository contains a customized version based on the mlx-community/Llama-3.2-3B-Instruct-4bit checkpoint, which has been upcasted back to 16-bit precision (safetensors).

The model incorporates distillation traces from Fable 5, inducing an instruction-guided Chain of Thought (CoT) behavior via <think>...</think> tags.

🚀 Model Features

• —Base Model: mlx-community/Llama-3.2-3B-Instruct-4bit
• —Current Format: safetensors (FP16 container)
• —Behavior: Induced Chain of Thought (CoT). The model is optimized to break down its analytical logic within thinking blocks before delivering the final response.
• —Primary Language: English / Spanish

📊 Benchmark Results (GSM8K)

The model was evaluated on the full GSM8K dataset (1,319 mathematical reasoning problems) using a 5-shot configuration. It demonstrates a remarkable retention of analytical capabilities despite its scale and lossy origin:

Note: Since this model inherits its core structure from a previously 4-bit quantized base, the performance remains impressively close to Meta's native BF16 model (~77.7%). The underlying quantization loss is significantly compensated for by the execution of the Chain of Thought.

🛠️ Output Structure & UI Rendering

The model is aligned to structure its outputs simulating an internal monologue.

• —In UI environments like Open WebUI, the interface will automatically parse and collapse the analytical block into a clean, interactive container.
• —In raw streaming environments (such as Ollama CLI), the tokens inside the tags will be fully visible in real-time.

Output Example:

<think>
[Logical breakdown, problem analysis, variable assignment]
</think>
[Clean, final response or refactored code block]