matt-hans93/ups-tools-qwen2.5-7b

UPS Tool-Use Qwen2.5 7B (v5a)

UPS Tool-Use Qwen2.5 7B is a QLoRA fine-tune of Qwen2.5-7B-Instruct for generating structured Hermes function-calling payloads for a UPS MCP server. It maps natural-language shipping and logistics requests to 18 UPS MCP tools covering tracking, rating, shipment creation, pickup workflows, address validation, paperless documents, landed cost, and location lookup.

This is the v5a release, trained on top of the v3 champion with replay + targeted augmentation data. It achieves 100% held-out accuracy and improves all stress-test axes over the prior release.

This repository publishes the PEFT LoRA adapter at the repo root and a Q4KM GGUF export for local inference with llama.cpp-compatible runtimes such as Ollama.

This model is not affiliated with or endorsed by UPS.

Model Details

• —Developed by: Matthew Hans
• —Model type: Qwen2.5-7B-Instruct causal language model with LoRA adapter fine-tuning
• —Base model: Qwen/Qwen2.5-7B-Instruct
• —Training base: unsloth/Qwen2.5-7B-Instruct-bnb-4bit
• —Fine-tuning method: QLoRA with rank-stabilized LoRA (rsLoRA)
• —Language: English
• —Output format: Hermes FC v1 tool call blocks
• —Release: v5a (May 2026)

Intended Use

The model is intended to sit behind a UPS MCP server and produce one structured tool call for each user request. A downstream orchestrator should parse the <tool_call_> block, validate arguments against the MCP tool schema, and then decide whether to call the UPS API.

<tool_call_>
{"name": "track_package", "arguments": {"inquiryNumber": "1Z999AA10123456784"}}
</tool_call_>

The system prompt should include the complete set of UPS MCP tool definitions in Hermes-compatible JSON:

<tools>
[{"type": "function", "function": {"name": "track_package", "parameters": {...}}}]
</tools>

Supported UPS MCP Tools

Evaluation

Held-out accuracy (122 test cases, disjoint from training)

Stress-test suite (~2,000+ inference calls across 9 axes)

Four graders run in parallel on every case:

• —Shape grader — required keys present in the tool call
• —Value-grounded grader — argument values match what the prompt said
• —Nested-structural grader — deep paths populated correctly
• —Schema-typed grader — types and enums match the MCP schema

Confidence intervals are Wilson 95%.

Changes from previous release (v3 → v5a)

Training Data

The dataset is synthetic, generated from UPS MCP server tool schemas.

The v5a training data includes:

• —Full replay of the original training set
• —20 targeted augmentation examples focused on multi-turn workflow steps and negative/refusal boundaries
• —Oversampling of initially weak tools (3x): rate_shipment, get_time_in_transit, upload_paperless_document, get_service_center_facilities, get_landed_cost_quote

No UPS API credentials or private customer shipment data are included.

Training Procedure

Hardware and Software

Training was run locally on an NVIDIA GeForce RTX 3090 with 24 GB VRAM.

Artifacts

Usage

Transformers and PEFT

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_id = "Qwen/Qwen2.5-7B-Instruct"
adapter_id = "matt-hans93/ups-tools-qwen2.5-7b"

base = AutoModelForCausalLM.from_pretrained(base_id, device_map="auto")
model = PeftModel.from_pretrained(base, adapter_id)
tokenizer = AutoTokenizer.from_pretrained(adapter_id)

Ollama

Download ups-tools-v5a-Q4_K_M.gguf and Modelfile, then create a local Ollama model:

ollama create ups-tools-qwen25 -f Modelfile

The Modelfile uses a low temperature and enough output budget for nested JSON payloads:

FROM ./ups-tools-v5a-Q4_K_M.gguf

PARAMETER temperature 0.1
PARAMETER top_p 0.9
PARAMETER num_predict 2048
PARAMETER stop "<|im_end|>"

Limitations and Risks

• —Paraphrase fragility. Value-grounded accuracy drops to 68.6% on paraphrased prompts (terse SMS-style is worst at ~57%). Free-form text interfaces will see higher error rates.
• —Sibling-tool confusion. On adversarial prompts with similar tools, shape accuracy is 88.3%. The most confused pairs are get_political_divisions vs find_locations and rate_shipment vs create_shipment.
• —Incomplete prompts. The model fabricates missing fields instead of asking for clarification. Production orchestrators must validate arguments against the MCP schema before execution.
• —`rate_shipment` cross-field combinations. The model can pair UPS service codes with incompatible packaging types. Validate service↔packaging compatibility before submitting.
• —Multi-step workflow cascading. Step accuracy is 96.3% but full-workflow accuracy is 80% — early errors cascade. Validate each turn in multi-turn flows.
• —Specialization to 18 UPS MCP tools. Do not expect generalization to unrelated APIs without additional evaluation.

Out-of-Scope Use

Do not use this model as an autonomous authority for shipping purchases, billing decisions, customs declarations, legal compliance, or cancellation actions. It should generate candidate MCP tool calls only; production systems should validate, log, and gate execution.

License

Apache 2.0. The base Qwen2.5 model is also released under Apache 2.0.