Modelpublic

maxz411/SmolLM2-1.7B-packreadout-lmhead

sourceHugging Facecc-by-nc-4.0updated 1d agoView on Hugging Face

0likes12downloads

Model Card

SmolLM2-1.7B-packreadout-lmhead

A pack-readout classifier from HuggingFaceTB/SmolLM2-1.7B, from the paper Any Language Model Is a Classifier: Calibrated One-Pass Decisions over Arbitrary Label Sets from Pretrained LLMs (Max Zhang, 2026). Given a prompt and any list of options, the model scores every option in one forward pass (the options are packed after the prompt with a block attention mask, then read out) and returns a probability distribution over the options. It was trained once on a mixture of 18 public classification and multiple-choice tasks and is evaluated on 12 held-out tasks it never saw, with their own label sets.

LM-head readout; no scalar head.

This repository holds a LoRA adapter (rank 16, alpha 32, on every attention and MLP projection); the LM head of the base model is the readout. It contains no base-model weights: load it on HuggingFaceTB/SmolLM2-1.7B.

Training

Cross-entropy over the option set (the log score) with LoRA, 2,000 steps of 8 examples, learning rate 0.0002, 100 warm-up steps and cosine decay, at most 2,000 examples per training task, seed 0. The full recorded configuration is in config.json. The training mixture, the held-out tasks and every result are in the paper; the code is maxz411/classifier-from-llm (Python package packreadout).

Use

python

from huggingface_hub import snapshot_download
from transformers import AutoModelForCausalLM, AutoTokenizer
from packreadout.train import load_adapter

base = "HuggingFaceTB/SmolLM2-1.7B"
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype="bfloat16", device_map="auto")
model, readout, head = load_adapter(model, snapshot_download("maxz411/SmolLM2-1.7B-packreadout-lmhead"))

scripts/evaluate.py --model HuggingFaceTB/SmolLM2-1.7B --adapter <this directory> --tasks heldout in the code repository reproduces the paper's evaluation; README.md there shows how to score a prompt against a list of options.

License

The adapter weights are released under CC BY-NC 4.0 because the training mixture includes SciQ (CC BY-NC 3.0). The base model keeps its own license. The code is Apache-2.0.