wangyue114514/rwkv7-g1g-13.3b-hf

RWKV-7 G1G 13.3B — Hugging Face reference model

This repository is a self-contained Hugging Face conversion of rwkv7-g1g-13.3b-20260523-ctx8192.pth. Release v0.9.0 contains the complete readable, pure-PyTorch reference implementation next to the weights. Normal inference does not require rwkv7-hf, FLA, a custom CUDA wheel, JIT, or CUDA Graphs.

Install and use

python -m pip install torch transformers

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "wangyue114514/rwkv7-g1g-13.3b-hf"
revision = "v0.9.0"
tokenizer = AutoTokenizer.from_pretrained(
    model_id, revision=revision, trust_remote_code=True
)
model = AutoModelForCausalLM.from_pretrained(
    model_id, revision=revision, trust_remote_code=True, torch_dtype="auto"
).eval()
device = "cuda" if torch.cuda.is_available() else "cpu"
model = model.to(device)
inputs = tokenizer("The future of recurrent language models is", return_tensors="pt")
inputs = {key: value.to(device) for key, value in inputs.items()}
with torch.inference_mode():
    tokens = model.generate(**inputs, max_new_tokens=32)
print(tokenizer.decode(tokens[0], skip_special_tokens=True))

trust_remote_code=True loads the checked-in files configuration_rwkv7.py, cache_rwkv7.py, ops_rwkv7.py, modeling_rwkv7.py, and tokenization_rwkv7.py. The recurrent state has the canonical [batch, heads, key, value] layout. Cache lifecycle, loss, padding, generation, gradients, and the layer structure are visible in those files.

Fine-tuning and evaluation

The model follows the standard AutoModelForCausalLM contract and supports Trainer, Accelerate, PEFT LoRA, TRL SFT/DPO/GRPO, and lm_eval through the ordinary Transformers path. Set model.config.use_cache = False during training. Reproducible examples and evaluation manifests are in 123123213weqw/hf-adapter.

Reference versus optimized execution

The v0.9.0 model code intentionally favors readability and compatibility. It is the correctness/reference path, not a peak-throughput benchmark backend. Optional CUDA Graph and Triton work is maintained separately on perf/optional-native-backend-v0.10; older CUDA/JIT/quantization and KV-v2 experiments remain on perf/native-kernels-v0.8. Neither branch changes this model repository's public HF contract.

Model and provenance

• —Architecture: RWKV-7 recurrent causal language model
• —Checkpoint family: G1G
• —Parameters: 13,269,245,952
• —Layers: 61
• —Hidden size: 4096
• —Vocabulary size: 65536
• —Stored weight dtype: float16
• —Original checkpoint SHA256: 0aa686d3ca4bb486e83e3071f4798a210f960e1fc1f5042e6cb418cc463814d6
• —Reference source revision: b8438cab0dc7d11238942efca4c05135d53fcaf8
• —Release tag: v0.9.0

conversion_manifest.json retains the immutable original conversion and weight provenance. v0.9.0 changes the checked-in runtime code and config only; the safetensors bytes are not rewritten or re-uploaded.

License

Apache-2.0. See LICENSE.