tiny-random/hy3
031
1---2library_name: transformers3base_model:4- tencent/Hy35---6 7This tiny model is intended for debugging. It is randomly initialized using the configuration adapted from [tencent/Hy3](https://huggingface.co/tencent/Hy3).8 9| File path | Size |10|------|------|11| model.safetensors | 5.4MB |12 13 14### Example usage:15 16- vLLM17 18```bash19# Multi-token prediction is supported20model_id=tiny-random/hy321vllm serve $model_id \22 --tensor-parallel-size 2 \23 --speculative-config.method mtp \24 --speculative-config.num_speculative_tokens 1 \25 --tool-call-parser hy_v3 \26 --reasoning-parser hy_v3 \27 --enable-auto-tool-choice28```29 30- SGLang31 32```bash33# Multi-token prediction is supported34model_id=tiny-random/hy335python3 -m sglang.launch_server \36 --model $model_id \37 --tp 2 \38 --tool-call-parser hunyuan \39 --reasoning-parser hunyuan \40 --speculative-num-steps 1 \41 --speculative-eagle-topk 1 \42 --speculative-num-draft-tokens 2 \43 --speculative-algorithm EAGLE44```45 46- Transformers47 48```python49from transformers import AutoModelForCausalLM, AutoTokenizer50 51model_id = "tiny-random/hy3"52 53tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)54model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto", trust_remote_code=True)55messages = [56 {"role": "user", "content": "Write a short poem about AI."},57]58inputs = tokenizer.apply_chat_template(59 messages,60 tokenize=True,61 return_tensors="pt",62 add_generation_prompt=True,63 reasoning_effort='high',64)65print(inputs)66outputs = model.generate(**inputs.to(model.device), max_new_tokens=32)67output_text = tokenizer.decode(outputs[0])68print(output_text)69```70 71### Codes to create this repo:72 73<details><summary>Click to expand</summary>74 75```python76import json77from copy import deepcopy78from pathlib import Path79 80import torch81import torch.nn as nn82 83from huggingface_hub import file_exists, hf_hub_download84from transformers import (85 AutoConfig,86 AutoModelForCausalLM,87 AutoTokenizer,88 GenerationConfig,89 set_seed,90)91 92source_model_id = "tencent/Hy3"93save_folder = "/tmp/tiny-random/hy3"94 95processor = AutoTokenizer.from_pretrained(source_model_id, trust_remote_code=True)96processor.save_pretrained(save_folder)97 98with open(hf_hub_download(source_model_id, filename='config.json', repo_type='model'), 'r', encoding='utf-8') as f:99 config_json = json.load(f)100config_json.update({101 'expert_hidden_dim': 32,102 'moe_intermediate_size': 32,103 'head_dim': 32,104 'hidden_size': 8,105 'intermediate_size': 32,106 'num_attention_heads': 8,107 'num_hidden_layers': 4,108 'num_key_value_heads': 4,109})110with open(f"{save_folder}/config.json", "w", encoding='utf-8') as f:111 json.dump(config_json, f, indent=2)112 113config = AutoConfig.from_pretrained(114 save_folder,115 trust_remote_code=True,116)117print(config)118torch.set_default_dtype(torch.bfloat16)119set_seed(42)120model = AutoModelForCausalLM.from_config(config, trust_remote_code=True).eval().cpu()121if file_exists(filename="generation_config.json", repo_id=source_model_id, repo_type='model'):122 model.generation_config = GenerationConfig.from_pretrained(123 source_model_id, trust_remote_code=True,124 )125 model.generation_config.top_k = 40 # original value in source model is -1 , which is invalid126 127# mtp128mtp = deepcopy(model.model.layers[-1])129mtp.eh_proj = nn.Linear(config.hidden_size * 2, config.hidden_size, bias=False)130mtp.enorm = nn.RMSNorm(config.hidden_size)131mtp.hnorm = nn.RMSNorm(config.hidden_size)132mtp.final_layernorm = nn.RMSNorm(config.hidden_size)133model.model.layers.append(mtp)134 135# init weights136set_seed(42)137model = model.cpu().eval()138n_params = sum(p.numel() for p in model.parameters())139with torch.no_grad():140 for name, p in sorted(model.named_parameters()):141 torch.nn.init.normal_(p, 0, 0.2)142 print(name, p.shape, p.dtype, f'{p.numel() / n_params * 100: .2f}%')143 144# expert bias is in float32145for i in range(config.first_k_dense_replace, config.num_hidden_layers + 1, 1):146 model.model.layers[i].mlp.e_score_correction_bias = nn.Parameter(torch.randn_like(147 model.model.layers[i].mlp.e_score_correction_bias148 ).float() * 0.002)149 150model.save_pretrained(save_folder)151print(model)152torch.set_default_dtype(torch.float32)153```154 155</details>156 157### Printing the model:158 159<details><summary>Click to expand</summary>160 161```text162HYV3ForCausalLM(163 (model): HYV3Model(164 (embed_tokens): Embedding(120832, 8, padding_idx=120002)165 (layers): ModuleList(166 (0): HYV3DecoderLayer(167 (self_attn): HYV3Attention(168 (q_proj): Linear(in_features=8, out_features=256, bias=False)169 (k_proj): Linear(in_features=8, out_features=128, bias=False)170 (v_proj): Linear(in_features=8, out_features=128, bias=False)171 (o_proj): Linear(in_features=256, out_features=8, bias=False)172 (q_norm): HYV3RMSNorm((32,), eps=1e-05)173 (k_norm): HYV3RMSNorm((32,), eps=1e-05)174 )175 (mlp): HYV3MLP(176 (gate_proj): Linear(in_features=8, out_features=32, bias=False)177 (up_proj): Linear(in_features=8, out_features=32, bias=False)178 (down_proj): Linear(in_features=32, out_features=8, bias=False)179 (act_fn): SiLUActivation()180 )181 (input_layernorm): HYV3RMSNorm((8,), eps=1e-05)182 (post_attention_layernorm): HYV3RMSNorm((8,), eps=1e-05)183 )184 (1-3): 3 x HYV3DecoderLayer(185 (self_attn): HYV3Attention(186 (q_proj): Linear(in_features=8, out_features=256, bias=False)187 (k_proj): Linear(in_features=8, out_features=128, bias=False)188 (v_proj): Linear(in_features=8, out_features=128, bias=False)189 (o_proj): Linear(in_features=256, out_features=8, bias=False)190 (q_norm): HYV3RMSNorm((32,), eps=1e-05)191 (k_norm): HYV3RMSNorm((32,), eps=1e-05)192 )193 (mlp): HYV3MoE(194 (gate): HYV3TopKRouter()195 (experts): HYV3Experts(196 (act_fn): SiLUActivation()197 )198 (shared_experts): HYV3MLP(199 (gate_proj): Linear(in_features=8, out_features=32, bias=False)200 (up_proj): Linear(in_features=8, out_features=32, bias=False)201 (down_proj): Linear(in_features=32, out_features=8, bias=False)202 (act_fn): SiLUActivation()203 )204 )205 (input_layernorm): HYV3RMSNorm((8,), eps=1e-05)206 (post_attention_layernorm): HYV3RMSNorm((8,), eps=1e-05)207 )208 (4): HYV3DecoderLayer(209 (self_attn): HYV3Attention(210 (q_proj): Linear(in_features=8, out_features=256, bias=False)211 (k_proj): Linear(in_features=8, out_features=128, bias=False)212 (v_proj): Linear(in_features=8, out_features=128, bias=False)213 (o_proj): Linear(in_features=256, out_features=8, bias=False)214 (q_norm): HYV3RMSNorm((32,), eps=1e-05)215 (k_norm): HYV3RMSNorm((32,), eps=1e-05)216 )217 (mlp): HYV3MoE(218 (gate): HYV3TopKRouter()219 (experts): HYV3Experts(220 (act_fn): SiLUActivation()221 )222 (shared_experts): HYV3MLP(223 (gate_proj): Linear(in_features=8, out_features=32, bias=False)224 (up_proj): Linear(in_features=8, out_features=32, bias=False)225 (down_proj): Linear(in_features=32, out_features=8, bias=False)226 (act_fn): SiLUActivation()227 )228 )229 (input_layernorm): HYV3RMSNorm((8,), eps=1e-05)230 (post_attention_layernorm): HYV3RMSNorm((8,), eps=1e-05)231 (eh_proj): Linear(in_features=16, out_features=8, bias=False)232 (enorm): RMSNorm((8,), eps=None, elementwise_affine=True)233 (hnorm): RMSNorm((8,), eps=None, elementwise_affine=True)234 (final_layernorm): RMSNorm((8,), eps=None, elementwise_affine=True)235 )236 )237 (norm): HYV3RMSNorm((8,), eps=1e-05)238 (rotary_emb): HYV3RotaryEmbedding()239 )240 (lm_head): Linear(in_features=8, out_features=120832, bias=False)241)242```243 244</details>245 246### Test environment:247 248- torch: 2.11.0+cu128249- transformers: 5.13.0