RWKV-Red-Team/ARWKV-7B-Preview-0.1-NoG

<div align="center"> <img src="./figures/banner.jpg" style="border-radius: 10px; width: 100%; height: 100%; object-fit: cover; box-shadow: 10px 10px 20px rgba(0, 0, 0, 0.5); border: 2px solid white;" alt="ARWKV" /> </div>

<h1 align="center">ARWKV🪿</h1>

<p align="center"> <a href="https://arxiv.org/abs/2501.15570"><b>Paper Link</b>👁️</a> | <a href="https://github.com/yynil/RWKVInside"><b>Github</b>✅</a> </p>

ARWKV-7B-FROM-32B (Preview 0.1)

<img src="./figures/architecture.png" alt="ARWKV Hybrid Architecture" width="30%">

Preview version with RWKV-7 time mixing and Transformer MLP

This version doesn't have the parameter "g". It freezes the MLP and uses 7B logits for distillation alignment.

📌 Overview

ALL YOU NEED IS RWKV

This is an early preview of our 7B parameter RNN-based model, trained on 2k context length (only stage-2 applied, without SFT or DPO) through 3-stage knowledge distillation from Qwen2.5-7B-Instruct. While being a foundational version, it demonstrates:

• —✅ RWKV-7's efficient recurrence mechanism
• —✅ No self-attention, fully O(n)
• —✅ Constant VRAM usage
• —✅ Single-GPU trainability

Roadmap Notice: We will soon open-source different enhanced versions with:

• —🚀 16k+ context capability
• —🧮 Math-specific improvements
• —📚 RL enhanced reasoning model

How to use

pip3 install --upgrade rwkv-fla transformers

Before training: export WKV_MODE=chunk

from transformers import AutoModelForCausalLM, AutoTokenizer


model = AutoModelForCausalLM.from_pretrained(
    "RWKV-Red-Team/ARWKV-7B-Preview-0.1-NoG",
    device_map="auto",
    torch_dtype=torch.float16,
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained(
    "RWKV-Red-Team/ARWKV-7B-Preview-0.1-NoG"
)

🔑 Key Features

🏗️ Architecture Highlights

Core Modification Flow

Qwen2.5 Decoder Layer:
- Grouped Query Attention
+ RWKV-7 Time Mixing (Eq.3)
- RoPE Positional Encoding
+ State Recurrence
= Hybrid Layer Output