cnuland/llm-d-sc-complexity

llm-d-sc-complexity

A 384-dimensional sentence embedding model fine-tuned to separate prompt complexity tiers, for use as a routing signal by llm-d-sc, the semantic classification runtime for llm-d.

The model does not emit a class directly. It produces an embedding that is ranked against a set of labelled anchors (anchors.json, shipped with this repository). This keeps the taxonomy as data rather than as a frozen classification head: anchors can be replaced or extended without retraining.

Taxonomy

Intended use

Selecting a serving tier per request. A SIMPLE prompt does not need a frontier model; a REASONING prompt usually does. The classifier emits ranked evidence only. Routing, endpoint selection, and session affinity remain the caller's responsibility.

Evaluation

Evaluated by llm-d-sc on a held-out set of 80 prompts authored independently of the training corpus, 20 per tier, deliberately drawn from domains outside the training pipeline's domain-transfer list (sailing, agriculture, transit, broadcast, museums, orchestras). 20 of the 80 are boundary cases.

Classification method: cosine similarity against the anchors, mean of the top 3 per tier, argmax.

Per tier:

Both errors are COMPLEX predicted as MEDIUM, the same boundary the training-time report identified as the model's only remaining confusion. The base model's confidence is near-uniform (0.25-0.27 across four tiers), which is the expected signature of an embedding space that carries no complexity structure at all.

Latency on CPU (single thread, Apple M-series, embed plus rank): p50 9.2 ms, p99 19.0 ms.

These numbers were produced on a homelab and have not been independently reproduced.

Training

Fine-tuned from sentence-transformers/all-MiniLM-L6-v2 with BatchAllTripletLoss and group_by_label batch sampling on 871 synthetic examples generated and cross-verified by two separate LLMs. Pipeline: https://github.com/cnuland/hello-chris-sr-finetuned

Usage

from sentence_transformers import SentenceTransformer
import json, numpy as np

model = SentenceTransformer("cnuland/llm-d-sc-complexity")
anchors = json.load(open("anchors.json"))["anchors"]

def classify(text, top_k=3):
    q = model.encode(text, normalize_embeddings=True)
    scores = {}
    for tier, examples in anchors.items():
        sims = model.encode(examples, normalize_embeddings=True) @ q
        scores[tier] = float(np.sort(sims)[-top_k:].mean())
    return max(scores, key=scores.get), scores

print(classify("Prove that the square root of 3 is irrational."))
# ('REASONING', {...})

Limitations

• —English only.
• —Trained on synthetic data; no human-labelled validation set exists.
• —The MEDIUM / COMPLEX boundary is genuinely ambiguous and is where residual error concentrates.
• —Anchor quality directly determines accuracy. Replacing anchors.json changes behaviour without retraining.

License

Apache-2.0.