specklabs/Speck2-140M-Instruct

[image]

Speck2-140M-Instruct

Speck2-140M-Instruct is a 140.7M parameter English instruction-tuned language model. It was initialized from Speck2-140M, a hybrid model pretrained from scratch on 20B tokens with a three-phase curriculum, then fully fine-tuned for one epoch on the 500,000-conversation SpeckChat2 mixture.

The Speck2 release updates the pretrained base while retaining the SpeckChat2 instruction-tuning data and one-epoch post-training recipe. The model uses a native chat template with optional system messages and was trained with assistant-only loss.

Summary

Architecture

The architecture is unchanged from Speck2-140M apart from 3 added role-token embeddings. It contains 18 residual blocks: 8 global attention + 10 gated causal convolution, each followed by a SwiGLU feed-forward.

Input/output embeddings (640-wide) are tied and connect to the 768-wide residual stream through learned projections.

Chat template

Three special tokens were added to the base 32k vocabulary:

A rendered conversation has the following form:

<s><|system|>
{optional system message}</s>
<|user|>
{user message}</s>
<|assistant|>
{assistant response}</s>

The system message is optional and may appear only first. Remaining roles must alternate between user and assistant. No default system prompt is injected. During generation, the prompt ends after <|assistant|>\n, and </s> terminates the response.

Usage

Speck2-140M-Instruct works with the Transformers Auto classes through its bundled custom model and tokenizer code. Set trust_remote_code=True when loading it.

pip install "transformers==5.1.0" torch sentencepiece safetensors

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "specklabs/Speck2-140M-Instruct"
device = "cuda" if torch.cuda.is_available() and torch.cuda.is_bf16_supported() else "cpu"
dtype = torch.bfloat16 if device == "cuda" else torch.float32

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=dtype,
).to(device)

messages = [
    {"role": "system", "content": "You are a concise assistant."},
    {"role": "user", "content": "What is the capital of France?"},
]

encoded = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt",
)
input_ids = encoded["input_ids"] if hasattr(encoded, "keys") else encoded
input_ids = input_ids.to(device)

output = model.generate(
    input_ids=input_ids,
    attention_mask=torch.ones_like(input_ids),
    max_new_tokens=64,
    do_sample=False,
    pad_token_id=tokenizer.eos_token_id,
)
generated = output[0, input_ids.shape[1] :]
print(tokenizer.decode(generated, skip_special_tokens=True))

The bundled generation path is validated for single-prompt greedy decoding. Direct forward passes support right-padded batches when use_cache=False.

Evaluation

The quality columns combine the Open SLM Leaderboard at revision 2eafcfc647b667e67f3b0288e9b67da497a78052 and BananaMind Base Bench 1.1 at revision d4aade51312889e8580963e1ce960c6eaef1a450. No chat template or generation was used for the seven Speck evaluations.

Benchmarks and speed

Open SLM Int Index means the chance-normalized Intelligence Index reported by the Open SLM Leaderboard. BananaMind Base Bench 1.1 Elo means the overall Elo reported by BananaMind Base Bench 1.1. Speed and memory values are local batch-1 measurements described below. Reference models saw 1.5-100x more pretraining tokens, so this is a parameter-adjacent comparison, not a compute-matched one. Asterisks mark architecture-equivalent Speck2 CPU measurements rather than new checkpoint-specific measurements.

Inference speed

Speed was measured locally at batch 1 with eager PyTorch, model-native caches, last-token logits, and tokenization excluded. Prefill uses 512 tokens. Decode measures 64 greedy cached steps after a 448-token prefix and includes argmax. CPU runs use FP32 with 16 threads; RTX 3090 runs use BF16. Reported throughput is calculated from the median duration.

The Speck2-140M-Instruct GPU measurements used PyTorch 2.9.1 and Transformers 5.1.0 on an NVIDIA GeForce RTX 3090. The run used 50 prefill warmups, 25 decode warmups, and 20 measurements. Across 12 CUDA probe rows at lengths 8, 448, and 512, the last-token projection preserved every full-logit greedy token selection with minimum cosine similarity 0.99951 and mean absolute difference below 0.075. CPU entries use the architecture-equivalent Speck2 measurements.

Memory is unique live BF16 model tensor storage plus cache/state tensor storage after a 2,048-token prefill at batch 1. Speck2-140M-Instruct uses 268.3 MiB for unique model tensors and 12.0 MiB for state, or 280.3 MiB combined. This excludes framework RSS, CUDA allocator reservations, and temporary operator workspace. FP32 CPU tensor memory is approximately twice the reported BF16 model-tensor memory. For another context length N, approximate memory as model tensor memory + State@2K x N / 2,048; Speck's small convolution history is fixed rather than context-scaled. The BF16 Safetensors weight file is 281.3 MB.

Limitations

• —At 140.7M parameters, the model has limited knowledge, reasoning, coding, and multilingual capacity.
• —Instruction following is inconsistent, especially for exact formatting, arithmetic, strict brevity, and system-prompt constraints.
• —Longer generations can become repetitive or incoherent.
• —It can hallucinate facts and produce incorrect code or calculations.
• —Mostly English: multilingual ability is weak and unvalidated.
• —No dedicated safety alignment, red-team evaluation, or misuse evaluation was performed.
• —Validated only up to 2,048 tokens despite a 4,096-token config.
• —Pretraining and instruction data are web-derived, user-authored, or synthetic and may contain bias, errors, unsafe content, personal information, or copyrighted text.
• —The bundled Transformers generation path currently supports single-prompt greedy decoding only.

Reproducibility

Full training and evaluation code: github.com/alkinun/speck

The released checkpoint is training step 8,534. The base model revision is 1201df613d9ee9d50909189f52e45c6fcefa3c01; the SpeckChat2 revision is 7b497b3e0c7f4653278cc67af27722b20a5c8d10.

Citation

@misc{alkinun2026speck2instruct,
  author       = {alkinun},
  title        = {Speck2-140M-Instruct: Instruction Tuning a Compact Hybrid Attention-Convolution Language Model},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/specklabs/Speck2-140M-Instruct}},
  url          = {https://github.com/alkinun/speck}
}