laskar-ks/qwen-1.5b-product-category-classifier

Product Category Classifier (Qwen2.5-1.5B + LoRA)

Selects the correct product category from a shortlist of 8–9 candidates produced by a retrieval step. This is the rerank stage — it does not replace retrieval.

Trained on Shopify's official product-catalogue dataset (38,511 rows after cleaning), evaluated on the official test split (9,652 rows).

Results

95% CI on n=9,652. Differences below ~1.9 points are not distinguishable from noise.

The zero-shot baseline had a 0.0% parse failure rate, so the entire 31.7-point gain is accuracy — none of it comes from the model learning to follow the output format.

Both the fine-tuned model and the zero-shot baseline are evaluated with constrained scoring: the logits for digit tokens 1–n are compared directly, so an out-of-range answer is structurally impossible for either.

Accuracy by difficulty

difficulty = how many of the candidates share the same top-level category as the correct answer. 1 = the answer stands alone; 9 = all candidates are siblings.

The largest gain is at difficulty 1, which is counterintuitive — those should be the easiest cases. The likely explanation is that these rows encode Shopify's internal taxonomy conventions rather than semantic similarity. For example, a Dutch scrub brush (Alpina schrobborstel) is labelled Shaving Brushes even though Scrub Brushes is among the candidates. That is not noise: it is a consistent, non-obvious convention, and the model learns it. Embedding similarity cannot.

Effect of product descriptions

Titles carry most of the signal. Catalogues with missing or poor descriptions lose about 5 points, not the task.

Generalization to unseen categories

Three ablation models, all with hyperparameters identical to the main run. Only the training data differs. Runs B2 and C are evaluated on the same 1,494-row slice.

C vs B2 — the main finding. These two differ by 0.9 points, well inside the margin of error. Having seen a category during training gives essentially no advantage. The entire drop from 66.0% to 59.1% is explained by training on 27% of the data, not by facing categories the model had never encountered.

The model is not memorising a category list. It learns how to read a product and match it against taxonomy structure, and that ability transfers intact to leaves absent from training. Practically: Shopify's taxonomy grows, and this model handles new leaves without retraining.

B vs B2 — the interesting failure. Removing a category from the answers while leaving it among the candidates is not an ablation. In Run B, 15% of training rows still contained the held-out categories as distractors, so the model saw them thousands of times and always as the wrong choice. It learned an active negative bias: those categories scored 17 points below the untuned base model.

Takeaway for anyone training a reranker: candidate composition in the training data matters as much as label distribution. Ignorance is far better than a learned negative bias.

Caveat: Run B used a different held-out set (701 leaves, n=1,008) than B2 and C (1,566 leaves, n=1,494), so its magnitude is not directly comparable. The 44-point gap far exceeds any plausible set-to-set variation, but the sets are not identical.

Inference cost

Benchmarked on A100 40GB, bfloat16, median prompt 368 tokens, constrained scoring (one forward pass per product, no generation loop).

At $1.43/hr for an A100 40GB (Spheron, 2026-08-23).

Comparison to a hosted API, with an important limitation. At $0.15 per 1M input tokens, the same 1,000 products would cost about $0.0736 in input tokens alone — roughly 8.8× more.

This compares cost, not accuracy. No API model was evaluated on this test set. A larger hosted model would very likely score above 66.0%. The case for a local 1.5B model rests on cost, data residency, and throughput predictability — not on matching frontier models on accuracy. Anyone choosing between the two should measure both.

Two things that do not appear in the cost table but often decide the question:

• —Catalogue data never leaves your infrastructure. Product naming and category structure are competitive assets.
• —No rate limits. Batching 100,000 products through an API means retries and queueing; on your own GPU, throughput is bounded only by hardware.

Usage

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
from huggingface_hub import snapshot_download

BASE = "Qwen/Qwen2.5-1.5B-Instruct"
tok = AutoTokenizer.from_pretrained(BASE)
tok.padding_side = "left"
if tok.pad_token is None:
    tok.pad_token = tok.eos_token

base = AutoModelForCausalLM.from_pretrained(
    BASE, torch_dtype=torch.bfloat16, device_map="cuda"
)
path = snapshot_download("laskar-ks/product-category-classifier-qwen-1.5b")
model = PeftModel.from_pretrained(base, path).eval()

SYSTEM = (
    "You are a product taxonomy classifier. "
    "Given a product and a numbered list of candidate categories, "
    "reply with ONLY the number of the correct category. No explanation."
)

def classify(title, description, candidates):
    listing = "\n".join(f"{i+1}. {c}" for i, c in enumerate(candidates))
    user = (
        f"Product title: {title}\n"
        f"Description: {description or '(no description)'}\n\n"
        f"Candidate categories:\n{listing}\n\n"
        f"Answer with the number only (1-{len(candidates)}):"
    )
    prompt = tok.apply_chat_template(
        [{"role": "system", "content": SYSTEM},
         {"role": "user", "content": user}],
        tokenize=False, add_generation_prompt=True,
    )
    enc = tok(prompt, return_tensors="pt").to("cuda")

    with torch.no_grad():
        logits = model(**enc).logits[0, -1, :]

    # Constrained scoring: only digits 1..n are considered
    digit_ids = [tok(str(i + 1), add_special_tokens=False)["input_ids"][0]
                 for i in range(len(candidates))]
    return candidates[int(torch.argmax(logits[digit_ids]).item())]


print(classify(
    "Alpina schrobborstel | 2 stuks",
    "Ruige schrobborstel voor hardnekkige schoonmaak. Materiaal: kunststof.",
    ["Home & Garden > Household Supplies > Household Cleaning Supplies > Scrub Brushes",
     "Health & Beauty > Personal Care > Shaving & Grooming > Shaving Brushes"],
))

The model expects candidates to come from a retrieval step. Feeding it arbitrary or unrelated candidate lists is outside what it was trained for.

Training

Candidate order is randomly permuted during both training and evaluation, so the model cannot learn a positional shortcut.

Loss is computed only on the answer token (prompt tokens masked with -100). Without this, LoRA capacity is spent memorising prompt text that is always supplied anyway.

Data preparation

Derived from Shopify/product-catalogue (Apache 2.0). Official train/test splits preserved.

• —Dropped product_image (text-only task)
• —Removed rows with an empty title, or where ground_truth_category was absent from potential_product_categories
• —Deduplicated on product_title within each split (81 removed from train, 6 from test)
• —39 titles appeared in both the official train and test splits. Removed from train only; the test split was left untouched so results stay comparable to other work on this dataset
• —Truncated titles to 200 chars, descriptions to 1,000 chars (one description ran to 46,773 chars)
• —Added difficulty and has_desc columns for the analyses above

Final: 38,511 train / 9,652 test.

Reproducibility

Hardware: A100 40GB (Colab Pro). Seed 42 throughout.

Package versions matter here — the HF ecosystem moved quickly around this release and several combinations fail:

torch 2.11.0+cu128
transformers 5.15.0
peft 0.13.2

Known issue. peft==0.13.2 calls hf_hub_download(use_auth_token=...), which newer huggingface_hub releases removed. Loading the adapter directly by repo ID raises a misleading Can't find 'adapter_config.json' error. Download first, then load from the local path (as in the usage example above), or pin huggingface_hub<0.26.

transformers 5.x also removed warmup_ratio from TrainingArguments; warmup_steps works on both 4.x and 5.x.

Limitations

• —Rerank only. Requires a retrieval step to produce 8–9 candidates. It does not classify into the full Shopify taxonomy from scratch.
• —Cost comparison is not an accuracy comparison. No API model was benchmarked on this test set.
• —Ceiling unknown. The dataset's own labels were not audited. Some appear to encode non-obvious conventions rather than errors, but no manual audit was run, so the maximum achievable accuracy on this test set is not established.
• —English-dominant. The training data includes Spanish, Dutch, Russian, and German products, but the distribution was not measured and per-language accuracy was not broken out.
• —Single epoch, no validation set. Training loss was still declining at the end of epoch 1. Whether a second epoch helps was not tested, because no validation split was held out to detect overfitting.

License

Apache 2.0, matching the source dataset.