Hakureirm/rwkv7-0.1b-hf

RWKV-7 "Goose" 0.1B — converted for the in-tree transformers implementation

This is not a new model. It is a format conversion of `BlinkDL/rwkv-7-world` → RWKV-x070-World-0.1B-v2.8-20241210-ctx4096.pth, laid out as a transformers directory so that the rwkv7 implementation can load it with from_pretrained. All credit for the weights belongs to BlinkDL / the RWKV project; they are redistributed here under the Apache-2.0 licence they were released under.

Why this exists

RWKV-7 weights are published in two layouts, and until now neither loaded into the in-tree implementation:

• —the reference .pth — the same parameter names, but a bare torch.save of a flat dict rather than a transformers directory;
• —the fla layout — a transformers directory, but a trust_remote_code repo whose modelling code and tensor names come from flash-linear-attention, so it loads its own implementation.

The conversion is a rename, not a transformation. Every one of the 399 tensors in the source .pth is bit-identical here; the only additions are three all-zero placeholders for layer 0's value-residual LoRA, which that layer never reads (layer 0 produces v_first rather than mixing towards it) and which exist so the state dict is rectangular. bfloat16, the dtype the source is stored in.

Usage

The implementation ships inside this repo (auto_map remote code), so stock transformers is all you need — no fork, no extra package:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("RWKV/RWKV7-Goose-World2.8-0.1B-HF", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("Hakureirm/rwkv7-0.1b-hf", trust_remote_code=True, dtype=torch.bfloat16)

inputs = tokenizer("The Eiffel Tower is located in the city of", return_tensors="pt")
print(tokenizer.decode(model.generate(**inputs, max_new_tokens=20)[0]))

The same model class is also open as an in-tree PR (rwkv-rs/transformers-rwkv#2); this repo is the way to use it today on released transformers.

Runs on

Pure-PyTorch portable path, no CUDA-only code on it. Verified, greedy 20/20 against BlinkDL's own runtime in each case: CPU (fp32/bf16, transformers 5.12.1), Apple Silicon MPS (fp32 and fp16), CUDA (the full test suite). Optional Triton kernels engage only on CUDA and fall back to the portable path everywhere else.

RWKV-7 is attention-free and fully recurrent: the state is a fixed-size matrix per head, so there is no KV cache, memory is constant in context length, and each new token costs the same as the first.

Verification

Checked against BlinkDL's own runtime — the rwkv package at cpu fp32 with RWKV_V7_ON=1, not against the implementation being loaded here, which would be self-certifying, and not against fla, whose RWKV layer prints a warning on import saying it is potentially buggy and that results should be cross-checked against the official repository.

Greedy, 20 tokens, from "The Eiffel Tower is located in the city of":

Both produce " Paris, France. It is the tallest building in the world and is the world's tallest".

Logits agree with the reference to 7.06e-05 absolute and 1.97e-06 relative, with argmax agreement on every prompt position.

0.1B is a useful size to check against for a reason beyond its download: it is 768 wide with head_dim 64, i.e. twelve heads of width 64, so the head count and the head width differ. At larger widths where both are 64, a quantity indexed by the wrong one of them agrees by coincidence.

Reproducing the conversion

huggingface-cli download BlinkDL/rwkv-7-world RWKV-x070-World-0.1B-v2.8-20241210-ctx4096.pth --local-dir .
python src/transformers/models/rwkv7/convert_rwkv7_checkpoint_to_hf.py \
    --checkpoint RWKV-x070-World-0.1B-v2.8-20241210-ctx4096.pth \
    --flavour native --dtype bfloat16 --output_dir ./rwkv7-0.1b-hf

--flavour fla reads the safetensors layout instead. The converter compares the checkpoint's tensor shapes against the ones the config implies and refuses a disagreement, so a config that names a different model fails rather than producing something that loads and generates noise.

Citation

The model is RWKV-7 "Goose" by Bo Peng (BlinkDL) and the RWKV community. Reference implementation: BlinkDL/RWKV-LM.