Tamkimd/tamev-large-qwen3.5-4b

⚡ TAMEV-Large-Qwen3.5-4B — Large System One Decision Model & LLM Router (MPS / MLX, Qwen3.5 + LoRA)

A self-hosted, open-source alternative to TypeSafe Jev: a large-tier System One decision model and LLM router. Give it a state plus typed questions and it returns calibrated probabilities over your candidate options (`choice` / `noul` / `score`) in one forward pass — no text generation, no JSON repair. Use it for LLM routing, tool-call gating, intent classification and multiple-choice scoring on a GPU or Apple Silicon. Ships safetensors, Metal MPS and MLX exports; the Qwen3.5 backbone is downloaded lazily on first use and the LoRA adapter is merged into it.

Task framing: this is an option-selection / decision model, not a fixed-label text classifier — the label set is supplied per call as the candidate list. It loads through transformers (AutoModel / AutoTokenizer, pipeline_tag: text-classification) and runs on CPU or on-device.

Open-source, Apache-2.0, self-hostable, and a drop-in protocol alternative to hosted decision APIs such as TypeSafe Jev. Related projects: Jared Palmer's Kev, Laya, and SemIf.

-3.48e-02-orange.svg) 

💡 What is TAMEV-Large-Qwen3.5-4B? A System One decision model and LLM router

A general-purpose LLM answers a routing or triage question by generating text that your application then parses. TAMEV-Large-Qwen3.5-4B answers it by scoring a candidate list:

• —⚡ 547.54 ms p50 / 1.4 req/s measured on Apple Silicon Metal MPS (BF16 backbone, FP16 pointer head).
• —🎯 Typed answers: choice (pick one of K options), noul (probability of a yes/no question), and score (expected value over an ordinal rubric) in one call.
• —Measured order sensitivity: the decision token attends to the packed option sequence, so reordering options changes the probabilities slightly — measured mean max drift 0.035 with a 8.00% decision-flip rate. The encoder tiers (Nano, Micro, Small) are exactly permutation-equivariant by construction; this tier is not. See the Evaluation note below.
• —🎯 Calibration you can inspect: calibrated probabilities, ECE 0.0346 and Brier 0.2413 on the 896-item development benchmark.
• —💰 Zero marginal cost: run it locally or in-process; no cloud round-trip and no per-token fee.

First run downloads the backbone and merges the training adapter. This repository ships only the trained pointer head; Qwen/Qwen3.5-4B (4.66B parameters) is fetched from the Hub in BF16 and the `jaredpalmer/kev-4b` LoRA is merged into it, because the head was trained on the merged backbone. Loading the plain base model instead of the merged one leaves the head out of distribution and drops top-1 sharply, so do not skip the merge. Use a GPU or Apple Silicon device.

📊 Model Overview

🏆 Evaluation

Source: the published weights in this repository, evaluated with the documented predict_decision helper over a 896-item development benchmark (data/processed/test.jsonl) built from the public datasets listed in the front matter, plus the leakage-free acceptance gate below. The dev suite is not a leakage-controlled held-out set: 390/896 items (43.5%) reuse a passage that also appears in training, 5/896 share an exact id with train.jsonl, and the same items have been reused across development rounds. The frozen 491-item acceptance set (runs/research/holdout_v1.manifest.json) is saturated (nano/micro/small top-1 = 1.000) and has no discriminative power. Suite composition (measured): agnews 270, yelp 158, boolq 82, banking77 80, mnli 59, imdb 55, sst5 54, trec 47, amazon 41, dbpedia14 39, plus 11 synthetic/game fixtures (compositional 7, legacypolicy 4) not listed in the front-matter datasets. The audit script is `scripts/audithf_artifacts.py`, shipped in the TAMEV source distribution.

🚦 Leakage-free acceptance gate (pubtest_v1, n=838)

The table above is the 896-item development benchmark. The acceptance authority for cross-tier claims is the leakage-free gate runs/research/pubtest_v1.jsonl (n=838): 0 exact-id overlap and 0 passage overlap with train.jsonl/the dev suite, drawn from the same 14 (source, qid) cells and templates. It is harder and differently calibrated than the dev suite, and it changes the ranking: the dev suite is a smoke-scale estimate, the gate is the acceptance gate.

This tier on the gate: top-1 0.7064 (95% CI 0.6754-0.7387), top-3 0.9749, ECE 0.1190 (95% CI 0.0964-0.1480), Brier 0.4093, on mps.

Pairwise McNemar on top-1: nano→micro p=0.0017, micro→small p=0.0028, small→medium p<0.001 (all CI-separated). Medium vs Large is not separated (top-1 .7136 vs .7064; b=55, c=49, p=0.6241), and Large's paired ECE is worse with a CI excluding zero (ΔECE = +0.0324 point difference; paired-bootstrap median +0.0243, 95% CI [0.0022, 0.0466]).

Medium is the best accuracy/calibration tier tested on the gate (highest top-1 .7136, lowest ECE .0866); Large is Pareto-dominated by Medium (strictly_dominated_by: ["tamev-medium-qwen3.5-0.8b"]) — 5.3× the total inference parameters (4,661,176,321 vs 873,963,585) for no top-1 gain and worse calibration. The large tier is the most expensive artifact, not the most accurate one. Source: runs/research/tier_compare_pubtest_v1.json (gate suite sha256 f16761d6…).

Device caveat (from the report itself): the pareto rows mix cpu and mps; compare parameters/top-1/ECE across devices and latency only within a device. Gate-run latency is load-contaminated and is not quoted on this card until the idle re-measure lands.

External public benchmarks on upstream test splits are published for Nano only (runs/research/pub_bench_tamev-nano-tinybert.md); no equivalent run exists for this tier.

How these numbers were produced. The published pointer head in this repository was run through the documented predict_decision helper over the 896-item development benchmark (data/processed/test.jsonl), one call per item, on Apple Silicon Metal MPS (BF16 backbone, FP16 pointer head), with the state/question packed into the trained delimiter sequence and truncated at max_state_len = 128 tokens per field. Top-1 and Top-3 do not depend on the path; the probabilities do: predict_decision / decide applies the calibrated temperature (2.406), which is what the ECE and Brier rows use, while forward(...)["logits"] are pre-temperature. The latency row is the whole documented helper path (tokenization plus one packed-sequence backbone pass), not a bare kernel time. Reproduce it with .venv/bin/python scripts/audit_hf_artifacts.py <model_dir> mps 896 from a TAMEV source checkout. Permutation drift is a real limitation here. The in-context causal tiers read a decision token that attends to the whole packed option sequence, so reordering options changes the probabilities slightly (measured mean max drift 0.035, 8.00% of items flip their top-1 answer). Only the encoder tiers (Nano, Micro, Small) are exactly permutation-equivariant by construction. If you need strict order independence, use an encoder tier.

These are the numbers from one measured run, not guarantees. All five tiers were scored on the same 896 items with their full option sets (K ∈ (2, 3, 4, 6, 14, 77)), so cross-tier accuracy is like-for-like on this suite — but an 896-item mixed development benchmark is a smoke-scale estimate, not a deployment estimate, and the per-tier 95% bootstrap interval on Top-1 is roughly ±0.03.

🔎 How option order affects this tier

• —Packed in-context sequence: the decision token attends over the whole packed sequence, so options can influence each other. Reordering them changes the probabilities — measured mean max drift 0.035, with 8.00% of items flipping their top-1 answer (dev-suite probe). The leakage-free gate probe measures 0.029 mean max drift with a 0.00% flip rate, so quote the probe you measured rather than a single number.
• —Shared pointer head: the head itself has one weight set used for every option slot, so it adds no positional asymmetry; the order sensitivity comes from the backbone's attention over the packed sequence.
• —Dedicated option budget: each option gets its own token window (up to 64 tokens), so a 77-option question is scored as 77 options instead of being truncated.
• —If you need strict order independence, use an encoder tier (`Tamkimd/tamev-nano-tinybert`, `Tamkimd/tamev-micro-minilm`, `Tamkimd/tamev-small-modernbert`), which are exactly permutation-equivariant by construction (≤ 9.7e-9 on the dev probe and ≤ 2.2e-8 on the gate probe, 0.00% flips).

📦 Artifacts in This Repository

⚠️ This repository does not contain the backbone. It contains the 1,311,233-parameter pointer head (5.00 MiB). The base model Qwen/Qwen3.5-4B (4.66B parameters) is downloaded from the Hugging Face Hub on first use, and the jaredpalmer/kev-4b LoRA adapter is merged into it with the optional peft package — the head was trained against the merged backbone, so skipping the merge would score it out of distribution (_ensure_backbone_weights() performs the merge). Budget the download and prefer a GPU or Apple Silicon device: on CPU this tier is much slower than Nano.

Download

hf download Tamkimd/tamev-large-qwen3.5-4b --local-dir ./tamev-large-qwen3.5-4b

🚀 Quick Start

Method 1: transformers with trust_remote_code=True

import torch
from transformers import AutoModel, AutoTokenizer

model_id = "Tamkimd/tamev-large-qwen3.5-4b"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True, fix_mistral_regex=True)
model = AutoModel.from_pretrained(model_id, trust_remote_code=True)

result = model.predict_decision(          # `model.decide(...)` is an alias
    state="My transaction on a Visa card was rejected while I was travelling in Tokyo.",
    question="Which service queue should handle this incident?",
    options=[
        "verify_travel_unblock",
        "file_fraud_dispute",
        "replace_damaged_card",
        "branch_appointment",
    ],
    tokenizer=tokenizer,
)

print(f"Selected: {result['best_option']} (confidence {result['confidence']:.2%})")
print("Probabilities:", result["probabilities"])
print("Permutation drift guarantee:", result["drift_guarantee"])  # this tier is order-sensitive: the field says equivariance does not apply

Requirements: torch>=2.4, transformers>=4.40, peft>=0.10, Python ≥ 3.10, plus the Qwen/Qwen3.5-4B backbone (4.66B parameters) downloaded on first use. peft is required because the `jaredpalmer/kev-4b` LoRA adapter is merged into the base backbone before the pointer head runs — the head was trained on the merged model. The optional causal_conv1d / flash-linear-attention kernels are not required, but without them the backbone falls back to transformers' reference PyTorch implementations of the chunked gated-delta-rule layers, which transformers itself reports as "much slower". Install them where your platform supports them before quoting this tier's latency.

Method 2: in-process serving with the tamev package

⚠️ `TypeSafeDirectClient` targets the encoder tiers. It builds an encoder-style TamevEngine, so it does not load TAMEV-Large-Qwen3.5-4B's trained pointer head — the snippet below runs the encoder path, not this tier. Use Method 1 above for this tier.

# From a TAMEV source checkout
uv venv && uv pip install -e ".[serve]"

from tamev import Choice, Noul, Score, TypeSafeDirectClient

# The default engine auto-loads the Nano checkpoint on CPU.
# `model_name` is only a display label; pass `checkpoint_path=`/`backbone=` to change the weights.
with TypeSafeDirectClient(model_name="TAMEV-Large-Qwen3.5-4B") as client:
    res = client.system_one(
        state="Customer reports a debit card block during an overseas ATM withdrawal.",
        questions={
            "action": Choice(
                instructions="Select the incident resolution playbook",
                criteria={
                    "travel_unblock": "Verify identity and lift the travel restriction",
                    "dispute_charge": "Open an unauthorized-transaction fraud case",
                    "branch_visit": "Direct the customer to the nearest branch",
                },
            ),
            "is_emergency": Noul(instructions="Is the customer stranded and in urgent need of cash?"),
            "urgency_score": Score(
                instructions="Rate the incident urgency",
                criteria=["Routine", "Elevated", "Critical"],
            ),
        },
    )

    # Server-side `confidence` is the winning probability normalized against uniform
    # chance for choice answers, (max(p) - 1/K) / (1 - 1/K); it is not a probability.
    print("Decision:", res.answers["action"].choice)
    print("Confidence:", res.answers["action"].confidence)
    print("P(emergency):", res.answers["is_emergency"].noul)
    print("Urgency score:", res.answers["urgency_score"].score)

Method 3: HTTP server

tamev serve builds an encoder-style engine, so it does not load this tier's pointer head — serving this repository with --checkpoint would silently run a backbone without the trained head. Use Method 1 (transformers) for in-context inference, or serve an encoder tier such as `Tamkimd/tamev-nano-tinybert`.

🎯 Intended Use

• —Agent workflow routing and tool-call gating: pick the next tool, subagent, or queue.
• —Safety and policy triage: classify an input into your own labelled categories and thresholds.
• —High-cardinality classification: intent catalogues such as Banking77-style 77-option menus.
• —Batch and offline scoring: label or route records with no network dependency.

Not intended for: open-ended text generation, factual question answering, multilingual deployment without validation, or acting as an unaudited safety control. The model scores the options you give it; it does not know your policy.

⚠️ Limitations

• —English-only training data. The suite is drawn from the datasets in the front matter, which are English (Banking77 excepted for its intent taxonomy). Other languages are unvalidated.
• —Calibration is per tier and not uniformly ≤ 0.05. This tier measures ECE 0.0346 on the 896-item development benchmark, which meets the repository's ≤ 0.05 target; on the 896-item development benchmark the 95% bootstrap interval is wider than the distance to the gate, so read the interval, not just the point estimate. On the leakage-free gate this tier measures ECE 0.1190 (95% CI 0.0964-0.1480), and no tier is below 0.05 on the gate — the best gate ECE is Medium .0866 and the worst is Small .1270. The "≤ 0.05" framing is a development-benchmark result only; on the gate the ECE order is Medium (.0866) < Nano (.1002) < Large (.1190) < Micro (.1201) < Small (.1270).
• —Accuracy is suite-specific and this suite is a development benchmark. 390/896 items (43.5%) reuse training passages, 5/896 share an exact id with train.jsonl, and the same 896 items were reused across development rounds, so this is a development benchmark, not leakage-controlled held-out data. The frozen 491-item acceptance set is saturated (nano/micro/small top-1 1.000) and cannot discriminate between tiers. Measured on the Nano tier: three retrains on the same corpus gained +4.5 to +6.4 points here while the external ARC/OpenBookQA/CommonsenseQA splits moved by at most +0.8 points with overlapping 95% CIs, and external ECE degraded (0.0652 to 0.0792-0.0893). This benchmark therefore cannot be used as a ship gate on its own. Validate on your own distribution.
• —The backbone is not included. First use downloads Qwen/Qwen3.5-4B (4.66B parameters).
• —Tokenizer note. tokenizer.json ships the pretokenize_regex declared by tokenizer_config.json, and config.json records the transformers version used at export. On older transformers you may still see the upstream pretokenize-regex warning for this tokenizer family — pass fix_mistral_regex=True to AutoTokenizer.from_pretrained (tokenizer only, not the model) to silence it. Tokenization of the evaluated suite is unchanged either way.
• —Teacher/KD coverage. Teacher probabilities exist only for the 3,500 synthetic game rows (snake_navigation 2,000, tetris_placement 1,500) of the 10,658-row merged training set; the 7,158 public-dataset rows carry none. Do not describe the public-dataset training as knowledge distillation.
• —Latency is hardware-specific. 547.54 ms p50 was measured on Apple Silicon Metal MPS (BF16 backbone, FP16 pointer head); your numbers will differ. MLX and Metal MPS exports are published here but their latency is not benchmarked in this repository.
• —Permutation equivariance is an architectural property, while the drift value above is measured by the benchmark suite (0.035); treat it as a measurement, not a contract.
• —Protocol compatibility is not an endorsement. TypeSafe-compatible means the /v1/systemone request/response shape, not TypeSafe's model or quality guarantees.

🥊 TAMEV Model Zoo

Medium/Large ship a pointer head and lazy-load the Qwen3.5 backbone; encoder tiers ship the full fused model. The dev-suite ranking is not the gate ranking: on the leakage-free acceptance gate Medium is the best accuracy/calibration tier tested (top-1 .7136, ECE .0866) and Large is dominated by Medium — 5.3× the total inference parameters for no top-1 gain and worse calibration. Large is the most expensive artifact, not the most accurate one. Accuracy across tiers also comes from different option budgets (4 vs 8), so compare with care.

📄 License & Citation

Apache License 2.0.

@misc{tamev2026,
  title={TAMEV: System One Decision Models for Edge AI, LLM Routing and Tool-Call Gating},
  author={TAMEV Contributors},
  year={2026},
  publisher={Hugging Face},
  url={https://huggingface.co/Tamkimd/tamev-large-qwen3.5-4b}
}