doofz/systemone-rlcd

<p align="center"> <img src="assets/banner.png" alt="System One by HAL-X" width="100%" /> </p>

System One — calibrated decisions in one forward pass

System One is HAL-X's non-autoregressive decision engine. You give it a state (a message, an email, a ticket, a JSON record or a conversation) and a set of typed questions. It returns a typed answer to every question, each with a calibrated probability, in a single encoder forward pass: about 9 ms on one RTX 4090. It never generates text, so there is nothing to parse and nothing to hallucinate. The answer space is defined at request time, so a new schema needs no retraining.

This release adds System One AZ, a checkpoint tuned for Azerbaijani. It comes with a language router that sends every request to the right checkpoint in about 20 µs.

Checkpoints

The repository bundles four checkpoints. The router picks one per request, and only the subfolder you ask for is downloaded.

Quickstart

# the `systemone` package ships in this repo; fetch only the code, weights download lazily on first use
hf download doofz/systemone-rlcd --include "systemone/*" "pyproject.toml" "setup.py" "README.md" --local-dir systemone-src
pip install "./systemone-src[server]"

from systemone import Router

router = Router(preload=True, device="cuda")      # all checkpoints resident; routing costs ~20 µs

state = {"body": "Salam, mart ayı üçün hesabımızdan iki dəfə pul çıxılıb. "
                 "Zəhmət olmasa artıq ödənişi bu gün geri qaytarın."}

questions = {
    "department": {"type": "choice", "instructions": "Which department should handle this request?",
                   "criteria": {"billing": "invoices, payments, refunds", "technical": "bugs, outages",
                                "sales": "pricing, new contracts", "other": "everything else"}},
    "urgency":    {"type": "score", "instructions": "How urgent is this request?",
                   "criteria": ["not urgent", "soon", "critical"]},
    "refund":     {"type": "noul",  "instructions": "Does the user explicitly request a refund?"},
}

res = router.predict(state, questions)
res["answers"]["department"]    # {'choice': 'billing', 'probabilities': {'billing': 0.979, ...}, 'confidence': ...}
res["routing"]                  # {'model': 'azerbaijani', 'reason': 'Azerbaijani (Latin script); ...'}

Questions can be written in English or in Azerbaijani, because the AZ checkpoint was trained on both. The three primitives are:

REST API

A production server ships in the package (FastAPI, bearer-token auth, every checkpoint preloaded):

SYSTEMONE_API_KEY=... CUDA_VISIBLE_DEVICES=0 python -m systemone.server --port 8095

curl -s localhost:8095/v1/decide -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" -d '{
  "state": "cox pis proqramdir, islemir",
  "questions": {"sentiment": {"type": "choice", "instructions": "Sentiment of this review?",
                              "criteria": ["positive", "negative"]}}}'
# -> {"answers": {"sentiment": {"choice": "negative", "probabilities": {"negative": 0.9997, ...}}},
#     "routing": {"model": "azerbaijani", ...}, "latency_ms": 10.4}

Why not vLLM? vLLM is built for autoregressive decoding (paged KV cache, continuous batching of generated tokens). System One generates no tokens. It is one bidirectional encoder pass, followed by a custom head that reads a score from each option's [MASK] position, and vLLM's pooling runner has no model class for that head. A single PyTorch process already answers in about 9 ms with no KV cache to manage. The throughput path is /v1/decide/batch, which packs requests into length-sorted batches.

How it works

Sequence construction

Every question becomes one sequence, and all the questions of a request are batched into one forward pass:

$$ \underbrace{\texttt{[CLS]}\;\langle t\rangle\;\text{question: } q\;\texttt{[SEP]}}{\text{instruction}}\; \underbrace{\texttt{[MASK]}\,o1\;\texttt{[MASK]}\,o2\;\cdots\;\texttt{[MASK]}\,ok\;\texttt{[SEP]}}{\text{options (head budget } \le 256\text{ tok)}}\; \underbrace{x\;\texttt{[SEP]}}{\text{state}} $$

The encoder output $H$ gets a type embedding $et$ added, then passes through two more transformer layers (the decision head). Each option $i$ is scored at the hidden state of its own marker $mi$:

$$ zi = w2^{\top}\,\mathrm{GELU}\!\left(W1\,\mathrm{LN}(\tilde h{mi})\right),\qquad \tilde H = \mathrm{Head}(H + et),\qquad pi = \frac{\exp(zi / Tb)}{\sum{j=1}^{k}\exp(zj / Tb)} $$

Here $T_b$ is a temperature fitted per bucket of (question type, option count). Because the options are part of the input, the label set can change on every request.

RLCD objective: strictly proper scoring

The policy reports a distribution $q$ over the options and is rewarded by a strictly proper scoring rule, so reporting honest probabilities is the only way to maximise the expected reward:

$$ R(q, y) \;=\; \underbrace{\log qy}{\text{log score}} \;+\; \lambda{\text{sph}}\,\underbrace{\frac{qy}{\lVert q\rVert2}}{\text{spherical}} \;-\; \lambda{\text{rps}}\,\mathbb{1}[t=\text{score}]\;\underbrace{\frac{1}{k-1}\sum{j=1}^{k}\Big(\textstyle\sum{i\le j} qi - \sum{i \le j} yi\Big)^{2}}_{\text{ranked probability score}} $$

with $\lambda{\text{sph}} = 0.5$ and $\lambda{\text{rps}} = 1$. The upstream base checkpoints were trained with REINFORCE: Gaussian exploration noise on the logits, a group-mean baseline and TD($\lambda$) for multi-turn states. System One AZ maximises the same reward by direct gradient ascent against one-hot targets. That is the deterministic-gradient form of the objective and has the same maximiser, $q^{\star} = p(y \mid x)$.

Calibration and confidence

Temperatures are fitted per bucket on held-out Azerbaijani validation data by minimising the NLL, $Tb^{\star}=\arg\min{T}\sum{n}-\log \mathrm{softmax}(z^{(n)}/T){y_n}$, and clamped to $[0.5, 5]$. Calibration is reported as

$$ \mathrm{ECE} = \sum{b=1}^{B}\frac{|Sb|}{N}\,\Big|\,\mathrm{acc}(Sb) - \mathrm{conf}(Sb)\,\Big|, \qquad \text{confidence} = 1 - \frac{H(p)}{\log k} $$

Benchmarks

All numbers below were measured by HAL-X on one RTX 4090 with bf16 weights. Every checkpoint answered byte-identical questions in the same run. Questions are always written in English (the developer's schema), and the state is in the target language.

<p align="center"><img src="assets/bench_azerbaijani.png" width="100%" alt="Azerbaijani accuracy"/></p>

The Router column is what Router().predict() returns on the same inputs. On Azerbaijani it matches the AZ checkpoint because 97–100% of the states are routed to it.

<p align="center"><img src="assets/bench_english.png" width="100%" alt="English retention"/></p>

The AZ checkpoint was trained with 4,000 English MASSIVE replay items, so its MASSIVE-en scores are in-domain. On the held-out SIB-200 task it still edges out both upstream checkpoints, so English was not traded away for Azerbaijani.

Calibration

<p align="center"><img src="assets/calibration.png" width="100%" alt="Reliability diagram and ECE"/></p>

The base multilingual checkpoint ships with all temperatures at 1.0 and is strongly over-confident on Azerbaijani: at 85% reported confidence it is right 46% of the time. After tuning and bucket-wise temperature fitting, the AZ checkpoint tracks the diagonal. Its ECE is 0.040 over 7,948 pooled Azerbaijani decisions, against 0.223 for the base. A confidence threshold is therefore meaningful: route low-confidence cases to a human or an LLM, and act automatically on the rest.

Latency

<p align="center"><img src="assets/latency.png" width="100%" alt="Latency"/></p>

Language detection in the router takes ~20 µs per request (pure Python). Peak VRAM per checkpoint is about 1.7–1.8 GB including activations.

Versus TypeSafe Jev

TypeSafe Jev is a closed, API-only decision model with the same interface idea: typed questions in, calibrated answers out. We have no Jev API access, so the Jev figures below are third-party published numbers reported by the upstream project and were not measured by HAL-X. The "System One" column covers the upstream EN and typed-decisions checkpoints, not the AZ checkpoint.

Jev still leads on very high-cardinality label sets and on matching soft teacher distributions. Latency is not an apples-to-apples comparison: Jev figures include the network round trip.

Router

state ──► script + language detection (~20 µs)
            ├─ Azerbaijani (ə, or ı/ş/ğ without Turkish function words, or az stop-words incl. ASCII spellings) ─► azerbaijani/
            ├─ confident English (English function words)                                                  ─► root (EN)
            ├─ other scripts / identified other languages (ru, tr, de, fr, es, hi, zh …)                   ─► multilingual/
            └─ short / unidentified Latin text  (az_first profile, default)                                ─► azerbaijani/

Routing is always overridable: router.predict(state, q, model="english") or lang="az". Set Router(az_first=False) to restore the upstream behaviour, where unidentified Latin text goes to the English checkpoint.

What to use it for

System One is a System 1 component. It makes the cheap, fast, high-volume decisions and leaves reasoning to a System 2 model (an LLM) only where its own confidence says it is needed.

• —LLM gateway and model routing: pick the right model, tool or agent for each request in about 10 ms, and escalate to a large model only when confidence is low. This is usually the biggest cost saver.
• —Guardrails and moderation: jailbreak, PII, toxicity and policy checks on every prompt and every response, in Azerbaijani, English and 100+ other languages, before the LLM sees the text.
• —Support and email triage: department, urgency, churn risk and refund intent for Azerbaijani banking, telecom and e-commerce inboxes, all in one pass.
• —Voice assistants and call centres: intent and slot-domain classification of ASR transcripts (the model was tuned on MASSIVE, which is spoken-assistant data).
• —App-store and social listening: Azerbaijani review sentiment and complaint detection at thousands of items per second.
• —Agent observability: score agent traces for risk, needs-review and outcome with calibrated probabilities you can threshold.
• —Human-in-the-loop automation: because probabilities are calibrated, "act automatically when $p \ge 0.9$" actually means about 90% precision.

Honest limits

• —Azerbaijani gains are mostly in-domain. MASSIVE and app-review scores use disjoint test splits of datasets whose train splits were used for tuning. The zero-shot signal is SIB-200 (0.672 → 0.716). Expect smaller gains on unseen Azerbaijani schemas than on the in-domain rows.
• —Business-triage `noul` questions in Azerbaijani are not yet tuned. On an Azerbaijani billing email, department and refund come out right, but "does the user threaten to cancel?" is under-detected (P = 0.10 on an explicit threat). Fine-tune on your own labelled tickets before relying on churn-type questions.
• —Ordinal `score` questions were not part of the AZ tuning. Their temperature bucket is unfitted (T = 1.0).
• —High-cardinality choice. With more than 20–30 options, raise agent.cfg["head_max_len"] or split the options into a coarse-to-fine hierarchy.
• —Very short text is ambiguous. "Super" or "ok" carries no language evidence; the az_first profile sends it to the AZ checkpoint.

Reproduce

Everything that produced these numbers is in this repo:

Data: MASSIVE (az-AZ, en-US), SIB-200 (azjLatn, engLatn), Azerbaijani app reviews (1–2★ vs 5★, balanced).

License and attribution

Apache 2.0. System One is developed by HAL-X. The EN, Multilingual and Typed-Decisions checkpoints and the core runtime are derived from Laya by Convai Innovations (Apache 2.0). The Azerbaijani checkpoint, the Azerbaijani-aware router, the batched runtime, the REST server and all benchmarks in this card are HAL-X work. See NOTICE.

@misc{halx2026systemone,
  title  = {System One: Calibrated Non-Autoregressive Decisions for Azerbaijani and 100+ Languages},
  author = {HAL-X},
  year   = {2026},
  url    = {https://huggingface.co/doofz/systemone-rlcd}
}