pbhappliedsystems/qwen-2.5-3B-instruct-gguf-F16

Qwen2.5-3B-Instruct · GGUF F16

Converted and evaluated by [PBH Applied Systems, LLC](https://pbhappliedsystems.com) — Applied AI/ML Consulting · LLM Optimization & Deployment · Quantized AI Infrastructure

🔬 This repository is part of a production-oriented evaluation series. Every model published under `pbhappliedsystems` has been independently evaluated using quant_eval v7.21 — a proprietary behavioral evaluation harness developed by PBH Applied Systems.

📌 This is the full-precision F16 baseline repository. The evaluated Q4\K\M deployment variant is published at `pbhappliedsystems/qwen-2.5-3B-instruct-gguf-Q4-K-M`. That card documents the full F16 vs. Q4\K\M comparison, including the ms_easy_01 schema recovery, mixed_brief_json recovery (0.000 → 1.000), and the runner-specific output format differences documented below.

⚖️ License Notice: This model is governed by the Qwen Research License, which permits non-commercial use only. Commercial use requires a separate license from Alibaba Cloud. See the License section for full details.

Try the Live AI Agent Demo

**Launch the PBH Applied Systems AI Agent Demo →**

This model is part of the PBH Applied Systems evaluated model series that supports the live AI Agent Demo. The demo lets visitors interact with production-style agent workflows powered by open-weight language models evaluated through PBH Applied Systems' quant_eval framework.

The F16 model serves a different role than the Q4KM deployment variant. F16 is the full-precision baseline used to measure what the model can do before quantization. quant_eval then compares the quantized model against this baseline to identify which capabilities are preserved, which degrade, and which tasks require guardrails or a higher-precision deployment.

This comparison is central to the demo. It helps determine which model belongs in which agent role:

• —Reasoning models are selected for planning, analysis, and auditable decision workflows.
• —Document models are selected for long-context extraction, summarization, and structured Q&A.
• —Code models are selected for task completion, structured output, API scaffolding, and automation workflows.
• —Quantized variants are selected when they preserve enough behavior to reduce cost, latency, and GPU requirements.
• —F16 variants remain important when maximum fidelity, cleaner tool execution, or reduced quantization risk matters more than speed or cost.

The live demo shows the deployment side of that process. The F16 card documents the reference behavior. The Q4KM card shows what changes after compression. Together, they explain how PBH Applied Systems uses quant_eval to choose the correct LLM for the correct agent type instead of guessing from model size or leaderboard reputation.

Model Description

This repository contains the full-precision F16 GGUF of `Qwen/Qwen2.5-3B-Instruct`, a 3-billion parameter instruction-tuned model from Alibaba Cloud.

In the PBH Applied Systems evaluation pipeline, this F16 run (20260221_034857) operated in cache-generation mode (skip_quant=true), producing the full_weight_cache.json used as the reference baseline for the subsequent Q4\K\M comparison run (20260221_041137). The F16 evaluation results documented here are identical to the F16 baseline data shown in the Q4\K\M card — confirmed by matching timing profiles, identical family results, and the same 5 failing fuzz cases on both runs.

Key Characteristics

• —Parameters: 3B
• —Format: GGUF F16 (full precision)
• —File size: 6.18 GB
• —SHA256: 65a0239fc9f9a40e2d4f79ae5e158cad423c2476fe089c744d5e6a6ff6fc9330
• —Minimum VRAM (GPU inference): ~8 GB
• —Recommended GPU tier: RTX 3060 · T4 · Any 8 GB+ GPU
• —Context window: 32,768 tokens
• —Inference speed (eval hardware): avg 1.438 sec/case on RTX 4090
• —License: Qwen Research License (non-commercial)

On F16 vs. Q4\_K\_M inference speed: The F16 averages 1.438 sec/case vs 0.390 sec/case for Q4\K\M — a 3.7× speedup at quantization. Both are fast by series standards. The Q4\K\M card notes MCQ at 10ms and stateful followup at 125ms — performance that remains broadly available at F16 too given the 3B parameter count.

PBH Applied Systems Evaluation — quant\_eval v7.21

Evaluation conducted by PBH Applied Systems, LLC using quant_eval v7.21 Run ID: 20260221_034857 · Fixtures: golden_oracle_fixtures_v7_21 (SHA256: 6d71a0b9147c...) · Seed: 42 Hardware: NVIDIA RTX 4090 · Runner: full_weight_transformers (F16 only) · Total rows: 42

Per-Family Pass Rates — F16 (full_weight_transformers)

json\_multistep — Case-Level Breakdown

No case passes at F16. The Q4\K\M variant recovers ms\easy\01 (1.000 on all gating signals). The checks_consistent_ok signal is the persistent blocker — 0.200 at both precision levels, confirming this is a 3B parameter count limitation.

F16-Specific Findings

Finding 1: Role-Token Contamination in Outputs

The HuggingFace Transformers runner includes truncated chat role tokens as literal text in several response families. Raw output inspection reveals:

The stateful_followup family passes despite the contamination because the JSON state objects are extracted correctly regardless of the "ician" prefix. The json family passes for the same reason. mixed_brief_json fails because answer_line_ok specifically requires the ANSWER: prefix to appear at the start of a line — the "user" token breaks this pattern.

Finding 2: toolcall — Valid Dispatch, No Final Answer

At F16, toolcall achieves stage1_tool_parse_ok=1 and stage1_tool_schema_ok=1 on both cases — the tool call JSON is correctly formed and schema-valid. However, the final answer number is never emitted:

The model produces the tool call and then terminates with a "user" role token, never computing or reporting the result. Contrast with Q4\_K\_M where the same cases produce {tool call}<|im_end|> 5<|im_end|> — the arithmetic result is present but EOS-contaminated. At F16, the result is absent entirely. Both are final_mismatch, but for different reasons: F16 = no answer produced; Q4\K\M = correct answer with stop-token contamination.

Finding 3: ms\easy\01 — Wrong Output Type at F16

ms_easy_01 at F16 produces an array instead of the required schema object:

[{"shelf": "A", "item": "P", "can_place": 1}]

Expected format: {"plan": [...], "checks": [...], "final": {...}}. This schema_ok=0 failure is unique in the 3B evaluation — no other multistep case produces an entirely wrong output type. At Q4\K\M, this same case passes cleanly at 0.785 seconds with all signals correct.

Finding 4: toolcall\_only — Args Format Difference Between Runners

Both runners fail toolcall_only on args_ok, but with slightly different incorrect schemas:

Both use "tool" instead of "tool_name" as the outer key (non-gating schema issue). Both provide numeric values as an array rather than named keys. The underlying 3B model behavior is the same at both precision levels — it cannot resolve the exact required schema for bare tool dispatch without explicit key-name enforcement in the prompt.

Finding 5: MCQ — Empty Output at Both Precision Levels

All 5 MCQ cases produce invalid_choice raw='' at F16 in under 30 milliseconds. No visible output is emitted. The Q4\K\M runner produces single letter outputs (B, A, C) on the same cases, with 3/5 correct. The F16 Transformers runner does not surface the choice letter in a form the extractor can process.

Signal-Level Diagnostics (F16)

json\_multistep

The checks_consistent_ok signal is invariant across precision levels — it reflects the 3B model's reasoning consistency ceiling, not quantization effects. Every other signal improves at Q4\K\M.

stateful\_followup

toolcall\_only

mixed\brief\json

When to Deploy F16 vs. Q4\K\M

For this model, Q4\_K\_M is the stronger deployment choice across every measurable family. The F16 runner exhibits role-token contamination that degrades families the Q4\K\M runner handles cleanly. The only advantage of F16 is slightly higher VRAM availability allowing larger context windows.

Hardware Requirements

Usage

Installation

pip install llama-cpp-python huggingface_hub

For GPU acceleration (CUDA):

CMAKE_ARGS="-DGGML_CUDA=on" pip install llama-cpp-python --force-reinstall --no-cache-dir

Python — llama-cpp-python

from huggingface_hub import hf_hub_download
from llama_cpp import Llama

model_path = hf_hub_download(
    repo_id="pbhappliedsystems/qwen-2.5-3B-instruct-gguf-F16",
    filename="qwen-2.5-3B-instruct-gguf-F16.gguf"
)

llm = Llama(
    model_path=model_path,
    n_ctx=4096,
    n_gpu_layers=-1,
    verbose=False,
)

response = llm.create_chat_completion(
    messages=[
        {
            "role": "system",
            "content": "You are a precise assistant. Return structured JSON when asked. Use exactly the key names specified."
        },
        {
            "role": "user",
            "content": "Return a JSON object with keys: summary, sentiment, action_items."
        }
    ],
    temperature=0.15,
    max_tokens=512,
)

print(response["choices"][0]["message"]["content"])

For stateful multi-turn use (reliable at F16 — role-token prefix does not block JSON extraction):

import json, re

conversation = [
    {"role": "system", "content": "You are a stateful assistant tracking structured data."},
    {"role": "user", "content": "Initialize a counter at 1. Return JSON only: {\"counter\": N}"},
]

r1 = llm.create_chat_completion(messages=conversation, temperature=0.15, max_tokens=64)
t1 = r1["choices"][0]["message"]["content"]
# Strip role-token prefix if present
t1_clean = re.sub(r'^[a-z]+\s*', '', t1.strip())

conversation.append({"role": "assistant", "content": t1_clean})
conversation.append({"role": "user", "content": "Increment the counter by 1."})

r2 = llm.create_chat_completion(messages=conversation, temperature=0.15, max_tokens=64)
t2_clean = re.sub(r'^[a-z]+\s*', '', r2["choices"][0]["message"]["content"].strip())
print(json.loads(t2_clean))

CLI — llama-cli

llama-cli \
  --model qwen-2.5-3B-instruct-gguf-F16.gguf \
  --chat-template qwen2 \
  --system-prompt "You are a precise assistant. Use exactly the key names specified." \
  --prompt "Return a JSON object with keys: summary, risk_level, action_items." \
  --n-predict 512 \
  --ctx-size 4096 \
  --n-gpu-layers -1 \
  --temp 0.15

For server deployment:

llama-server \
  --model qwen-2.5-3B-instruct-gguf-F16.gguf \
  --chat-template qwen2 \
  --ctx-size 4096 \
  --n-gpu-layers -1 \
  --port 8080 \
  --host 0.0.0.0

Query via the OpenAI-compatible API:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="not-required")

response = client.chat.completions.create(
    model="qwen-2.5-3B-instruct-gguf-F16",
    messages=[{"role": "user", "content": "Your prompt here"}],
    temperature=0.15,
)
print(response.choices[0].message.content)

Artifact Provenance

The F16 GGUF was converted from Qwen/Qwen2.5-3B-Instruct using a custom-built llama.cpp conversion pipeline developed by PBH Applied Systems.

Two-pass architecture: This F16 run (20260221_034857) operated in cache-generation mode (skip_quant=true), producing the full_weight_cache.json used as the reference baseline for the Q4\K\M comparison run (20260221_041137). Timing profile identity between this run and the F16 baseline in the comparison run confirms clean cache reuse and run integrity.

Evaluation Methodology

quant_eval v7.21 — proprietary behavioral evaluation harness, PBH Applied Systems.

Fixture set: golden_oracle_fixtures_v7_21 (SHA256: 6d71a0b9147c079371b02a94f3c149eb78a6adc03dc16ff6833b964fbf4174f0)

Evaluation hardware: NVIDIA RTX 4090 · F16 evaluation date: February 21, 2026 · Seed: 42

🔬 About quant_eval & This Evaluation Series

**quant_eval** is a proprietary behavioral evaluation harness developed by PBH Applied Systems, LLC. It measures real agent-adjacent task performance across structured output, tool dispatch, multi-turn state retention, and multi-step planning — not perplexity or leaderboard proxies. Every model published under `pbhappliedsystems` has been independently evaluated using quant_eval before being recommended for any production role.

See it in action: **Live AI Agent Demo →** The demo runs production-style agent workflows powered by open-weight models selected through the quant_eval evaluation pipeline.

Need a deployment recommendation? Not sure which quantization level is right for your hardware, latency target, or agent type? **→ pbhappliedsystems.com**

Evaluated and published by [PBH Applied Systems, LLC](https://pbhappliedsystems.com) · [patrick@pbhappliedsystems.com](mailto:patrick@pbhappliedsystems.com)

About PBH Applied Systems

**PBH Applied Systems, LLC** is an Oklahoma City–based applied machine learning and AI systems company specializing in production-grade model evaluation, quantization pipelines, agentic AI infrastructure, and scalable AI-driven application development.

Patrick Hill, M.S. — Founder · Data Scientist · AI/ML Engineer · Author of [Applied Machine Learning: Concepts, Tools, and Case Studies](https://a.co/d/05qat7Xz) (required reading, UAT CSC 373)

📞 Work With PBH Applied Systems

The F16 vs. Q4\K\M comparison for this model documents that the Q4\K\M runner handles role tokens and stop tokens more cleanly than the Transformers F16 runner — resulting in better practical outcomes at lower VRAM requirements. That counter-intuitive finding is only visible when both variants are evaluated against the same fixture set.

👉 [Book a Scoping Call](https://pbhappliedsystems.com) · 👉 [Request an Evaluation Report](https://pbhappliedsystems.com) — from $2,500

Connect

License

Qwen Research License Agreement — non-commercial use only. Inherited from `Qwen/Qwen2.5-3B-Instruct`.

Commercial use requires a separate license from Alibaba Cloud. Full terms: Qwen Research License

GGUF conversion and behavioral evaluation performed by [PBH Applied Systems, LLC](https://pbhappliedsystems.com) · quant_eval v7.21 · F16 Run ID: `20260221_034857`