Modelpublic

sizzlebop/Llama-3.2-3B-TechWriter-Instruct-GGUF

sourceHugging Facellama3.2updated 14d agoView on Hugging Face

0likes316downloads

Model Card

Llama-3.2-3B TechWriter Instruct GGUF

This repository provides GGUF quantizations for Shankarblr/Llama-3.2-3B-TechWriter-Instruct.

Llama-3.2-3B TechWriter is a fine-tune of meta-llama/Llama-3.2-3B-Instruct specialized for semiconductor and data-center interconnect technical documentation, product briefs, datasheets, application notes, and CLI user guides.

All GGUF files were converted from the original safetensors weights using llama.cpp at native F16 precision, followed by quantization into standard k-quant variants.

Available Files and Quantizations

File	Quant Type	Size	Description / Recommendation
`Llama-3.2-3B-TechWriter-Instruct-F16.gguf`	F16	5.99 GB	Full precision base conversion. Highest fidelity reference weights.
`Llama-3.2-3B-TechWriter-Instruct-Q8_0.gguf`	Q8_0	3.19 GB	Near-lossless 8-bit quantization. Recommended for best quality.
`Llama-3.2-3B-TechWriter-Instruct-Q6_K.gguf`	Q6_K	2.46 GB	High quality retention with minimal degradation. Excellent balance.
`Llama-3.2-3B-TechWriter-Instruct-Q5_K_M.gguf`	Q5KM	2.16 GB	Solid balance between memory usage and generation accuracy.
`Llama-3.2-3B-TechWriter-Instruct-Q4_K_M.gguf`	Q4KM	1.88 GB	Fast, lightweight 4-bit quant. Recommended default for mobile and edge setups.
`Llama-3.2-3B-TechWriter-Instruct-Q3_K_M.gguf`	Q3KM	1.57 GB	Compact footprint when memory headroom is strictly limited.
`Llama-3.2-3B-TechWriter-Instruct-Q2_K.gguf`	Q2_K	1.27 GB	Maximum compression for ultra-constrained environments.

Domain Capabilities

This model is fine-tuned to produce structured, consistent technical documentation:

—Product Briefs & Datasheets: Feature summaries, pinout descriptions, electrical specs, and register descriptions.
—Hardware & CLI Documentation: Host-adapter (HBA/NIC), switch, and DPU command-line user manuals.
—Architecture Notes: Process node, throughput, bus width, and protocol stack consistency.
—Spec Extraction: Grounded extraction and QA from raw hardware documentation excerpts.

Prompt Format

This model uses the standard Llama 3.2 Instruct template format:

text

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

You are a technical marketing and documentation writer for semiconductor and data-center interconnect products. Write clear, structured content. Match the requested document type. Keep specifications internally consistent: one process node, one primary throughput, and one form factor unless the source explicitly lists options.<|eot_id|><|start_header_id|>user<|end_header_id|>

Write a short feature overview for a 400G PCIe Gen5 NIC.<|eot_id|><|start_header_id|>assistant<|end_header_id|>

How to Use

1. With llama.cpp

Run interactive or single-turn generation with llama-cli:

bash

llama-cli -m ./Llama-3.2-3B-TechWriter-Instruct-Q4_K_M.gguf \
    -p "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nYou are a helpful technical writer.<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nWrite a short paragraph explaining what an API endpoint is.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n" \
    -n 256 --temp 0.2

Launch a local OpenAI-compatible HTTP server:

bash

llama-server -m ./Llama-3.2-3B-TechWriter-Instruct-Q4_K_M.gguf \
    --host 127.0.0.1 --port 8080 -c 4096

2. With Ollama

Create a Modelfile:

dockerfile

FROM ./Llama-3.2-3B-TechWriter-Instruct-Q4_K_M.gguf

TEMPLATE """{{ if .System }}<|start_header_id|>system<|end_header_id|>

{{ .System }}<|eot_id|>{{ end }}{{ if .Prompt }}<|start_header_id|>user<|end_header_id|>

{{ .Prompt }}<|eot_id|>{{ end }}<|start_header_id|>assistant<|end_header_id|>

{{ .Response }}<|eot_id|>"""

PARAMETER stop "<|start_header_id|>"
PARAMETER stop "<|end_header_id|>"
PARAMETER stop "<|eot_id|>"
PARAMETER temperature 0.3

Then create and run the model:

bash

ollama create techwriter-3b -f Modelfile
ollama run techwriter-3b

3. With LM Studio

1.Copy the desired .gguf file (e.g., Llama-3.2-3B-TechWriter-Instruct-Q4_K_M.gguf) into your LM Studio models directory.
2.Select Llama 3 as the chat preset.
3.Load the model and begin prompting.