Modelpublic

nagohachi/Llama-3.2-tiny-lm-japanese-500m-base-v2

sourceHugging Facellama3.2updated 2mo agoView on Hugging Face

1likes83downloads

Model Card

Llama-3.2-tiny-lm-japanese-500m-base-v2

Built with Llama. This model is a derivative of Meta's Llama-3.2-3B, produced by structured (width) pruning followed by continued pre-training in Japanese.

A ~476M-parameter Japanese base model created by width-pruning Llama-3.2-3B down to a 500M-class model, swapping in the Japanese llm-jp/llm-jp-3-440m tokenizer, and recovery-pretraining on ~10B tokens of educational Japanese web text. This is the base model — no instruction tuning, no chat template. See Llama-3.2-tiny-lm-japanese-500m-sft-v2 and Llama-3.2-tiny-lm-japanese-500m-dpo-v2 for the instruction-tuned / preference-aligned variants.

It is the pruning-based counterpart to the from-scratch tiny-lm-japanese-500m-base-v1; at an equal ~10B-token budget it reaches the from-scratch model's final quality in roughly half the tokens and scores higher on downstream tasks (see Evaluation).

The model uses a custom Llama-style implementation shipped with the repo (trust_remote_code=True required).

Model details


Parameters	476M
Source model	meta-llama/Llama-3.2-3B
Pruning	width: hidden 3072→1024, heads 24→8, KV 8→4 (head_dim 128 kept), 28 layers
Context length	2048
Vocab size	99,574
Tokenizer	llm-jp/llm-jp-3-440m (embeddings remapped mean-of-pieces from Llama)

Training

—Data: hotchpotch/fineweb-2-edu-japanese (sample_10BT)
—Schedule: width-prune Llama-3.2-3B → recovery pre-training on ~10B tokens

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "nagohachi/Llama-3.2-tiny-lm-japanese-500m-base-v2"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, dtype="bfloat16")

inputs = tok("日本の首都は", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=64, do_sample=True, temperature=0.7, top_p=0.9)
print(tok.decode(out[0], skip_special_tokens=True))

Evaluation

llm-jp-eval (v2.1.5)

We evaluated the models using 100 examples from the test split with greedy decoding. The llm-jp baselines were re-evaluated with the same harness. Base models use the 4-shot setting; SFT/DPO models and the instruct baseline are prompted through their chat template. The -v1 rows are the from-scratch counterparts for reference.

Model name	average	CR	EL	FA	HE-EN	HE-JA	MR	MT	NLI	QA	RC
Llama-3.2-tiny-lm-japanese-500m-base-v2	0.232	0.270	0.322	0.061	0.170	0.100	0.013	0.657	0.398	0.137	0.190
Llama-3.2-tiny-lm-japanese-500m-sft-v2	0.224	0.320	0.229	0.058	0.153	0.090	0.017	0.665	0.405	0.112	0.190
Llama-3.2-tiny-lm-japanese-500m-dpo-v2	0.222	0.320	0.225	0.064	0.153	0.060	0.017	0.664	0.395	0.108	0.210
---	---	---	---	---	---	---	---	---	---	---	---
tiny-lm-japanese-500m-base-v1	0.198	0.223	0.130	0.035	0.207	0.130	0.007	0.504	0.483	0.072	0.190
tiny-lm-japanese-500m-sft-v1	0.191	0.310	0.102	0.036	0.187	0.070	0.013	0.523	0.380	0.082	0.210
tiny-lm-japanese-500m-dpo-v1	0.194	0.310	0.130	0.036	0.187	0.070	0.013	0.531	0.345	0.082	0.240
---	---	---	---	---	---	---	---	---	---	---	---
llm-jp/llm-jp-3-440m	0.249	0.280	0.266	0.059	0.170	0.080	0.027	0.760	0.525	0.144	0.180
llm-jp/llm-jp-3-440m-instruct3	0.254	0.317	0.289	0.066	0.223	0.110	0.017	0.744	0.520	0.155	0.100

License

This model is a derivative of Meta Llama 3.2 and is distributed under the Llama 3.2 Community License (see also the Acceptable Use Policy). Attribution is provided in the NOTICE file. Built with Llama.