nullsilver/alpha-sys-1-1.6B

1---2license: other3license_name: lfm1.04license_link: LICENSE5base_model:6- LiquidAI/LFM2.5-VL-1.6B7base_model_relation: finetune8pipeline_tag: image-text-to-text9library_name: transformers10language:11- en12tags:13- calibration14- classification15- system-one16- jev-compatible17datasets:18- allenai/ai2_arc19- allenai/sciq20- allenai/openbookqa21- tau/commonsense_qa22- uoft-cs/cifar1023- mteb/stsbenchmark-sts24---25 26<div align="center">27  <img28    src="https://huggingface.co/buckets/nullsilver/main/resolve/nullsilver-banner-light-1.png"29    alt="alpha-sys-1 banner"30    style="width: 100%; max-width: 100%; height: auto; display: inline-block; margin-bottom: 0.5em; margin-top: 0.5em;"31  />32  <div style="display: flex; justify-content: center; gap: 0.5em; margin-bottom: 1em;">33    <a href="https://github.com/nullsilver-labs/alpha-sys-1/blob/main/docs/USAGE.md"><strong>docs</strong></a> •34    <a href="https://nullsilver.com"><strong>nullsilver.com</strong></a>35  </div>36</div>37 38# alpha-sys-1-1.6B39 40alpha-sys-1 is a multimodal, Jev-compatible **System One model**. It takes a state, which41may contain text, an image, or both, together with a question that has a fixed set of42answers, and returns a probability distribution over those answers in one forward pass. It43generates no text.44 45The model is trained for calibrated probabilities: across a large group of similar46examples where it assigns an answer a probability near 80%, that answer should be correct47in roughly 80% of cases. Calibration degrades when the input differs substantially from48the training data, so check the probabilities on data from the intended application.49 50| | |51|---|---|52| Base | [LiquidAI/LFM2.5-VL-1.6B](https://huggingface.co/LiquidAI/LFM2.5-VL-1.6B) |53| Tuning | LoRA rank 32, lr 1e-4, merged into the base weights |54| Checkpoint | `alpha-sys-1-260919`, revision `260919`, seed 1 of 3 |55| Input | text, one image, or both; English |56| Output | probabilities over the answer space |57| Sizes | [450M](https://huggingface.co/nullsilver/alpha-sys-1-450M) · [1.6B](https://huggingface.co/nullsilver/alpha-sys-1-1.6B) · [3B](https://huggingface.co/nullsilver/alpha-sys-1-3B) |58 59## Question types60 61Questions follow TypeSafe's System One format: a request contains one `state` and any62number of `questions`, so a question written for Jev runs here as is. `images` is an extra63field for multimodal inputs.64 65| type | answer space | returns |66|---|---|---|67| `choice` | named options, up to 26 | `probabilities` over the options, `choice` (argmax) |68| `noul` | a statement | `noul`, P(true) |69| `score` | ordered levels, lowest first | `probabilities` over the levels, `score` (expected level index) |70 71The answer is read from the next-token logits for the answer labels (`A`, `B`, … or72`No`/`Yes`), renormalised over the valid labels. Each question is answered independently:73one question's answer is never context for another.74 75> [!NOTE]76> `confidence` is `1 - H(p)/log(n)`, computed from the distribution. It is not a separate77> prediction.78 79> [!TIP]80> A `noul` probability near 0.5 means the model is uncertain.81 82## Usage83 84`alpha_sys_1.py` in this repository renders questions the way the model was trained on85them, batches the questions on one state, and returns answers in the System One shape.86 87```python88from huggingface_hub import hf_hub_download89import importlib.util, sys90spec = importlib.util.spec_from_file_location("alpha_sys_1", hf_hub_download("nullsilver/alpha-sys-1-1.6B", "alpha_sys_1.py", revision="260919"))91alpha_sys_1 = importlib.util.module_from_spec(spec); spec.loader.exec_module(alpha_sys_1)92 93m = alpha_sys_1.SystemOne("nullsilver/alpha-sys-1-1.6B", revision="260919")94m.system_one({95    "state": {"subject": "Duplicate charge on invoice #4411",96              "body": "We were billed twice for March. Refund the duplicate today or we cancel our plan."},97    "questions": {98        "department": {"type": "choice", "instructions": "Which department should handle this email?",99                       "criteria": {"billing": "invoices, payments, refunds", "technical": "bugs, outages",100                                    "sales": "pricing, new contracts", "other": "everything else"}},101        "urgency": {"type": "score", "instructions": "How urgent is this request?",102                    "criteria": ["not urgent", "soon", "critical deadline or blocking issue"]},103        "churn_risk": {"type": "noul", "instructions": "The user threatens to cancel or leave."}}})104# {"department": {"choice": "billing", "probabilities": {"billing": 0.98, "technical": 0.00, "sales": 0.01, "other": 0.01}, "confidence": 0.92},105#  "urgency": {"score": 1.60, "probabilities": [0.10, 0.20, 0.70], ...},106#  "churn_risk": {"noul": 0.59}}107```108 109Email triage is not one of the training environments; the output above is what this110checkpoint returns on it, not a tuned result.111 112Without the client, use this prompt format. A different format gives less reliable113probabilities.114 115```python116import string, torch117from transformers import AutoModelForImageTextToText, AutoProcessor118 119repo, rev = "nullsilver/alpha-sys-1-1.6B", "260919"120processor = AutoProcessor.from_pretrained(repo, revision=rev)121processor.tokenizer.padding_side = "left"122model = AutoModelForImageTextToText.from_pretrained(123    repo, revision=rev, dtype=torch.bfloat16, device_map="auto").eval()124 125def render(state, q):126    parts = [state] if state else []127    if q["type"] == "noul":128        c = q.get("criteria") or {}129        clar = "".join(f"\n{lab} means: {c[k]}" for lab, k in (("Yes", "true"), ("No", "false")) if c.get(k))130        parts.append(f"Statement: {q['instructions']}{clar}\nIs the statement true? Answer with Yes or No only.")131        return "\n\n".join(parts), ["No", "Yes"]132    crit = q["criteria"]133    items = list(crit.items()) if isinstance(crit, dict) else [(o, None) for o in crit]134    labels = list(string.ascii_uppercase[:len(items)])135    lines = [f"{lab}. {o}" + (f": {d}" if d else "") for lab, (o, d) in zip(labels, items)]136    parts.append(q["instructions"] + "\n" + "\n".join(lines) + "\nAnswer with the letter only.")137    return "\n\n".join(parts), labels138 139@torch.inference_mode()140def ask(q, state="", image=None):141    text, labels = render(state, q)142    content = ([{"type": "image", "image": image}] if image is not None else []) + [{"type": "text", "text": text}]143    inputs = processor.apply_chat_template(144        [[{"role": "user", "content": content}]], add_generation_prompt=True, tokenize=True,145        return_dict=True, processor_kwargs={"return_tensors": "pt"}).to(model.device)146    logits = model(**inputs, logits_to_keep=1).logits[0, -1].float()147    ids = [processor.tokenizer.encode(lab, add_special_tokens=False)[0] for lab in labels]148    return torch.softmax(logits[ids], -1).tolist()149 150p = ask({"type": "noul", "instructions": "The message conveys urgency"},151        state="Our API integration started returning 500 errors an hour before launch.")152urgent = p[1]  # P(Yes)153```154 155> [!NOTE]156> - A dict `state` is rendered one field per line, as `key: value`.157> - Training images smaller than 256 px were upscaled to 256 px.158> - For several questions on one state, batch them with `padding_side="left"`.159> - In bfloat16, probabilities move by up to a few hundredths with batch composition and160>   padding.161 162## Training163 164Training uses cross-entropy between the model's distribution and a target `y_soft`. The165target is one-hot when a dataset provides one answer, and the annotator distribution when166several annotations are available. Options are shuffled on every draw, the vision tower is167frozen, and environments are sampled in proportion to the square root of their size.168 169| environment | modality | type | label |170|---|---|---|---|171| mcq (ARC-Easy, SciQ, OpenBookQA, CommonsenseQA) | text | choice | one-hot |172| ChaosNLI (100-annotator items) | text | choice | annotator distribution |173| CivilComments-WILDS | text | noul | annotator share |174| STS-B | text | score | annotator mean |175| Folktables (ACS income, California 2014) | tabular as text | noul | outcome |176| CIFAR-10 | image | choice | one-hot |177| Camelyon17-WILDS | image | noul | outcome |178 179Three random seeds were trained. The released checkpoint is the seed with the lowest mean180development loss across environments.181 182## Evaluation183 184Each test split was read once per checkpoint. Reported intervals are 95% clustered185bootstrap intervals, clustered on the relevant dataset group: question, comment, hospital,186or state-year.187 188> [!IMPORTANT]189> Compare models on NLL and Brier score. ECE is reported alongside them and is misleading190> on its own: a model that always predicts the base rate can have a low ECE.191 192The tables carry two reference points. The **base rate** is the constant predictor: it193answers every question with the label frequencies of the training split (for example194"toxic" 14% of the time on CivilComments, whatever the comment says), or uniformly when195the options are shuffled. Any model should beat it. **Base + T** is the untuned196LFM2.5-VL-1.6B, read the same way as the tuned model, with its label logits divided by197one scalar temperature chosen to minimise NLL on the environment's development split.198 199**Trained environments.**200 201| environment | NLL | NLL, base + T | Brier | ECE | AUROC | acc |202|---|---|---|---|---|---|---|203| mcq | **0.427** | 0.527 | 0.218 | 0.012 | 0.877 | 0.847 |204| ChaosNLI | **0.784** | 0.835 | 0.171 | 0.022 | 0.670 | 0.680 |205| CivilComments | **0.323** | 0.629 | 0.044 | 0.064 | 0.922 | 0.933 |206| STS-B | **1.033** | 1.749 | 0.302 | 0.045 | 0.621 | 0.572 |207| Folktables | **0.448** | 0.612 | 0.296 | 0.013 | 0.761 | 0.778 |208| CIFAR-10 (+C) | **0.253** | 0.526 | 0.111 | 0.006 | 0.947 | 0.922 |209| Camelyon17 | **0.156** | 0.649 | 0.090 | 0.013 | 0.907 | 0.938 |210 211> [!NOTE]212> ChaosNLI, CivilComments and STS-B have soft labels from multiple annotations, so213> top-label ECE does not fully measure calibration. On these, use NLL and KL divergence to214> the annotator distribution.215 216**Unseen tasks.** Not in training.217 218| task | type | NLL | NLL, base + T | base rate |219|---|---|---|---|---|220| BoolQ | noul | 0.479 | **0.457** | 0.665 |221| Yelp review stars | score | **1.110** | 1.226 | 1.609 |222 223**Distribution shift.** CIFAR-10-C.224 225| | clean | sev. 1 | 2 | 3 | 4 | 5 |226|---|---|---|---|---|---|---|227| accuracy | 0.982 | 0.952 | 0.935 | 0.917 | 0.892 | 0.848 |228| mean confidence | 0.978 | 0.956 | 0.939 | 0.923 | 0.898 | 0.860 |229 230**Other System One models.** NLL on the text environments, same test splits, same231readout. The other alpha-sys-1 sizes on the table are their released seeds. `Qwen3.8-27B` is the open 27B232generalist, read at its first answer token with reasoning off, plus a dev-fitted233temperature.234 235| environment | alpha-sys-1-450M | alpha-sys-1-1.6B (this) | alpha-sys-1-3B | Qwen3.8-27B + T | base rate |236|---|---|---|---|---|---|237| mcq | 0.727 | 0.427 | 0.289 | **0.159** | 1.439 |238| ChaosNLI | 0.902 | 0.784 | 0.735 | **0.706** | 0.938 |239| CivilComments | 0.324 | 0.323 | **0.319** | 0.473 | 0.425 |240| STS-B | 1.107 | 1.033 | **0.961** | 1.347 | 1.727 |241| Folktables | **0.436** | 0.448 | 0.441 | 0.472 | 0.683 |242| BoolQ (unseen) | 0.661 | 0.479 | 0.405 | **0.316** | 0.665 |243| Yelp review stars (unseen) | 1.467 | 1.110 | 0.973 | **0.858** | 1.609 |244 245Per-hospital, per-state-year and per-identity-group tables, the three-seed gate tables and246the full comparison against other System One models (hosted and open) are in the247[repository](https://github.com/nullsilver-labs/alpha-sys-1) under `runs/`.248 249## Limitations250 251> [!WARNING]252> On a task that differs substantially from the training environments, do not assume this253> model stays calibrated; measure it against the base model's calibration. In254> leave-one-domain-out tests at 1.6B, a model tuned on the other environments beat the255> untuned base with a transferred temperature on one held-out environment out of three,256> and on the two unseen tasks above the trained-on-all checkpoints match the base model and do not beat it.257> With a few hundred labelled examples from your own task, fit a temperature on them:258> divide the label logits by one scalar chosen to minimise NLL on those examples259> (`alpha_sys_1.fit_temperature`), then pass it as `SystemOne(..., temperature=T)`.260 261- Knowledge depends on model size: on MCQ, the untuned 3B base outperforms this tuned262  model.263- When the model does not know an answer, its distribution is close to uniform.264- Under strong distribution shift, such as CIFAR-10-C at severity 5, confidence remains265  higher than accuracy.266- Reversing the option order changes the top answer on 11% of MCQ items, mostly267  among low-confidence examples.268- The answer space is capped at 26 options.269- Fine-tuning used English data only and at most one image per question.270 271## Related work272 273The interface follows TypeSafe's Jev (a hosted System One model, the `state` / `questions`274request shape). Reading an answer distribution from the label-token logits of one forward275pass is the readout of Kadavath et al. (2022, *Language Models (Mostly) Know What They276Know*) and of the LLM-as-a-Verifier line of work, which scores rubric levels from the277logits of letter tokens. That calibration improves with size, and that a temperature278fitted on one domain transfers badly to another, is Jiang et al. (2021, *How Can We Know279When Language Models Know?*). Training on a proper scoring rule against annotator280distributions is why a fixed answer space and calibration are non-conflicting (Kalai and281Vempala, 2024, *Calibrated Language Models Must Hallucinate*). Base models: Liquid AI's282LFM2.5-VL.283 284## License285 286This model is derived from LiquidAI/LFM2.5-VL-1.6B and is released under the [LFM Open License287v1.0](LICENSE).288 289## Citation290 291```bibtex292@misc{alphasys1,293  title  = {alpha-sys-1: a small calibrated System One model},294  author = {Nullsilver},295  year   = {2026},296  url    = {https://huggingface.co/collections/nullsilver/alpha-sys-1}297}298```299