vishinvents/distil-qwen3-1.7b-posthog-prioritizer

distil-qwen3-1.7b-posthog-prioritizer

A 1.7B specialist that ranks a batch of deduplicated product findings by impact, frequency, and effort, returning strict JSON with a reason per item. One of three tools in the distil-posthog-traffic-analyser harness; trained on the Distil Labs platform.

Task contract

Input:

Findings (N):
[ {"id": "...", "kind": "bug|gap", "severity": 1-5,
   "occurrences": <int>, "title": "...", "evidence": "..."}, ... ]

Return JSON now. Every id above must appear in "ranked" exactly once.

Output — valid JSON only, matching:

{"ranked":[{"id":"<input id>","rank":1,"reason":"<=2 sentences citing the numbers"}]}

Hard constraint: every input id appears exactly once — no dropped, invented, or duplicated ids. Ranking judgment encoded in training: security issues lead regardless of frequency; revenue blockers rank above equally severe non-revenue issues; silent data corruption outranks loud errors; at equal severity, reach wins; cosmetic items rank last.

Training

• —Base model: Qwen3-1.7B (Apache 2.0)
• —Teacher: openai.gpt-oss-120b (Apache 2.0)
• —Seed data: 22 hand-authored, schema-validated ranking batches (single-item to 4-item, including equal-severity ties and security/revenue trade-offs), committed at `examples/seeds/prioritizer.jsonl` (18 train / 4 held-out test); a coverage validator guarantees every seed batch ranks each finding exactly once
• —Synthetic expansion: 10,004 examples generated and validated by the Distil Labs platform from the seed set
• —Method: platform-managed fine-tune (task type: question-answering, JSON output)

A 0.6B variant was trained first; it ranked held-out batches correctly but violated the exactly-once coverage constraint on a batch containing near-duplicate findings. The 1.7B student holds exact coverage on that same adversarial batch, which is why this size ships.

Evaluation

Held-out test set (n=4), scored by the platform's LLM judge (an answer fails outright on any coverage violation, making this judge deliberately strict):

The judge score saturates because both pass its format bar on most batches; live behavior separates them. This model, on live batches (this GGUF via Ollama, matching the hosted endpoint): exact id coverage on 6/6 — 4 held-out seed batches, one adversarial near-duplicate batch, and one live pipeline batch fed by the other two tools — with rank-1 choices matching the hand-written gold ranking on 3 of the 4 held-out batches and a defensible judgment call on the fourth.

Usage (Ollama)

ollama create posthog-prioritizer -f Modelfile   # FROM ./<this gguf>

Then in the harness .env:

TOOL_PRIORITIZER_MODEL=posthog-prioritizer