CHwensX/llama-chinese-35m

Llama-Chinese-35M

English | 中文

English

Model Description

A 35M-parameter Chinese LLaMA-style language model trained from scratch on Chinese Wikipedia data. This is an educational project demonstrating the complete pipeline of training a modern decoder-only Transformer.

Model Architecture

• —Architecture: LLaMA-style Decoder-Only Transformer
• —Layers: 5 transformer blocks
• —Hidden Size: 512
• —Attention Heads: 8 (no GQA)
• —Head Dimension: 64
• —FFN: SwiGLU (intermediate size: 1792)
• —Position Encoding: RoPE (theta=10000.0)
• —Normalization: RMSNorm (Pre-LN, eps=1e-6)
• —Weight Tying: lmhead.weight = embedtokens.weight
• —Vocabulary Size: 32,000
• —Max Sequence Length: 512

Training Details

• —Training Data: ~18MB Chinese Wikipedia
• —Epochs: 3
• —Batch Size: 8
• —Learning Rate: 3e-4
• —Optimizer: AdamW
• —Hardware: Apple M5 (MPS)
• —Training Time: ~26 minutes

Usage

import torch
from modeling_llama import LlamaModel

# Load config
config = {
    "vocab_size": 32000,
    "embed_dim": 512,
    "num_layers": 5,
    "num_heads": 8,
    "num_kv_heads": 8,
    "ffn_dim": 1792,
    "context_length": 512,
    "pad_token_id": 0,
}

# Initialize model
model = LlamaModel(**config)

# Load weights
from safetensors.torch import load_file
state_dict = load_file("model.safetensors")
model.load_state_dict(state_dict)
model.eval()

# Generate text
input_ids = torch.tensor([[2, 1, 2, 3, 4]])  # example input
with torch.no_grad():
    logits = model(input_ids)

Tokenizer

• —Type: ByteLevel BPE
• —Vocabulary Size: 32,000
• —Special Tokens: <pad>, <unk>, <bos>, <eos>

Load the tokenizer:

from tokenizers import Tokenizer
tokenizer = Tokenizer.from_file("tokenizer.json")

Files

• —model.safetensors - Model weights in safetensors format
• —config.json - Model configuration
• —modeling_llama.py - Model architecture code
• —tokenizer.json - Tokenizer vocabulary and merges
• —tokenizer_config.json - Tokenizer configuration
• —special_tokens_map.json - Special token mappings

中文

模型简介

从零训练的 35M 参数中文 LLaMA 风格语言模型，使用中文维基百科数据训练。这是一个教育项目，展示了训练现代 Decoder-Only Transformer 的完整流程。

模型架构

• —架构: LLaMA 风格 Decoder-Only Transformer
• —层数: 5 个 Transformer Block
• —隐藏维度: 512
• —注意力头: 8（无 GQA）
• —头维度: 64
• —前馈网络: SwiGLU（中间维度: 1792）
• —位置编码: RoPE（theta=10000.0）
• —归一化: RMSNorm（Pre-LN, eps=1e-6）
• —权重绑定: lmhead.weight = embedtokens.weight
• —词表大小: 32,000
• —最大序列长度: 512

训练详情

• —训练数据: ~18MB 中文维基百科
• —训练轮数: 3
• —批次大小: 8
• —学习率: 3e-4
• —优化器: AdamW
• —硬件: Apple M5 (MPS)
• —训练时间: 约 26 分钟

使用方法

import torch
from modeling_llama import LlamaModel

# 加载配置
config = {
    "vocab_size": 32000,
    "embed_dim": 512,
    "num_layers": 5,
    "num_heads": 8,
    "num_kv_heads": 8,
    "ffn_dim": 1792,
    "context_length": 512,
    "pad_token_id": 0,
}

# 初始化模型
model = LlamaModel(**config)

# 加载权重
from safetensors.torch import load_file
state_dict = load_file("model.safetensors")
model.load_state_dict(state_dict)
model.eval()

# 生成文本
input_ids = torch.tensor([[2, 1, 2, 3, 4]])  # 示例输入
with torch.no_grad():
    logits = model(input_ids)

分词器

• —类型: ByteLevel BPE
• —词表大小: 32,000
• —特殊标记: <pad>, <unk>, <bos>, <eos>

加载分词器：

from tokenizers import Tokenizer
tokenizer = Tokenizer.from_file("tokenizer.json")

文件说明

• —model.safetensors - 模型权重（safetensors 格式）
• —config.json - 模型配置
• —modeling_llama.py - 模型架构代码
• —tokenizer.json - 分词器词表和合并规则
• —tokenizer_config.json - 分词器配置
• —special_tokens_map.json - 特殊标记映射