awilliamson/talkie-1930-13b-it-vllm

talkie-1930-13b-it (vLLM-servable repackage)

This is a HuggingFace + vLLM-ready repackage of `talkie-lm/talkie-1930-13b-it`. The original release ships as a raw torch state-dict (rl-refined.pt) plus a tiktoken vocab, with no config.json, tokenizer.json, or HF modeling code, so it can't be loaded by transformers or served by vLLM out of the box.

talkie-1930-13b-it is a 13B parameter instruction-tuned model from the talkie-lm project. The base model was pretrained on ~260B tokens of pre-1931 English text; the IT variant was instruction-tuned on a dataset built from pre-1931 reference works (etiquette manuals, encyclopedias, letter-writing guides) and refined with online DPO.

What this repo adds

Serving with vLLM

Tested with vLLM 0.19, transformers backend, on a single H100 (80 GB). bf16 only — fp8 is broken on this architecture.

vllm serve awilliamson/talkie-1930-13b-it-vllm \
    --model-impl transformers \
    --trust-remote-code \
    --dtype bfloat16 \
    --max-model-len 2048

Then hit it like any OpenAI-style chat endpoint:

curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "awilliamson/talkie-1930-13b-it-vllm",
        "messages": [{"role":"user","content":"Write one sentence about the year 1925."}],
        "temperature": 0.7,
        "max_tokens": 80
    }'

Sampling notes

• —Use `temperature ≥ 0.5` — greedy decoding (temperature=0) can collapse into single-token loops on this model.
• —top_p / top_k and repetition_penalty don't reliably help with that failure mode; temperature does.
• —Default max_position_embeddings=2048 matches the original IT training. The talkie-coder SWE recipe extends to 64K with NTK rope_theta=4e7, but loses ~14% on short evals (GSM8K) — only worth it for long-context agentic use.

Plain HuggingFace usage

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

tok = AutoTokenizer.from_pretrained("awilliamson/talkie-1930-13b-it-vllm", trust_remote_code=True)
m = AutoModelForCausalLM.from_pretrained(
    "awilliamson/talkie-1930-13b-it-vllm",
    trust_remote_code=True,
    dtype=torch.bfloat16,
).cuda().eval()

chat = tok.apply_chat_template(
    [{"role": "user", "content": "Write one sentence about the year 1925."}],
    tokenize=False, add_generation_prompt=True,
)
ids = tok([chat], return_tensors="pt").to("cuda")
out = m.generate(
    **ids, max_new_tokens=80, do_sample=True, temperature=0.7, top_p=0.9,
    pad_token_id=tok.pad_token_id,
    eos_token_id=[tok.convert_tokens_to_ids("<|end|>"), tok.eos_token_id],
)
print(tok.decode(out[0, ids.input_ids.shape[1]:], skip_special_tokens=True))

Provenance

• —Weights: rl-refined.pt from talkie-lm/talkie-1930-13b-it (bf16, vocab=65540, lm_head_gain.w_g=3.890625 baked in).
• —Tokenizer: built from vocab.txt from the same release (truncated to ranks < 65535, then the 5 chat specials appended at fixed ids).
• —Modeling code: adapted from ricdomolm/1930-coder/sft/modeling_talkie.py, with TalkieForCausalLM.lm_head switched from nn.Parameter to nn.Linear so the baked-in lm_head.weight loads cleanly for both HF and vLLM.

For the full inference reference and CLI, see the official talkie-lm/talkie repo.

License

Apache 2.0, matching the upstream talkie-lm/talkie-1930-13b-it release.