Beltran12138/ming-vintage-qwen3b-lora

⚡ Quickstart — one-file GGUF (no merge needed)

A pre-merged Q4_K_M GGUF is in this repo: ming-vintage-qwen2.5-3b-q4_k_m.gguf (1.9 GB). Download and run it — but you must use raw completion mode, not chat mode.

huggingface-cli download Beltran12138/ming-vintage-qwen3b-lora ming-vintage-qwen2.5-3b-q4_k_m.gguf --local-dir .
llama-completion -m ming-vintage-qwen2.5-3b-q4_k_m.gguf -p "问: 互联网者何也? 答曰:" -n 100 --temp 0.7 --repeat-penalty 1.1 -no-cnv --no-display-prompt
(older builds: llama-cli ... -no-cnv)

⚠️ Do NOT use chat mode. ollama run, llama-cli -hf ...:Q4_K_M, the HF "Use this model" one-liners, or anything that applies a chat template will wake Qwen-Instruct's assistant persona → modern Chinese, and the LoRA looks dead. This model was trained on bare 问: ... 答: ... completion pairs — it shines as a raw text generator.

ming-vintage-qwen3b-lora

The honest LARP — a documented 1424 Chinese vintage LoRA adapter. Not a vintage LLM. A LARP of one — built by fine-tuning Qwen 2.5 3B on pre-1424 Classical Chinese (文言) corpus from kanripo. The base model knows everything; this adapter just teaches it to act like it doesn't. Documented limitations included.

TL;DR

Why does this exist?

In early 2026 talkie-lm released a 1930-cutoff English vintage LLM. The viral observation: knowledge cutoff isn't a date, it's a worldview.

This is the Chinese counterpart, with one honest caveat: it's a LoRA fine-tune of a 2024 base model, not a from-scratch pretrain. The model knows GPT-4 exists. It just learned to style its answers as if it doesn't. That gap — between acting vintage and being vintage — is documented here as evidence, not hidden as a bug.

Intended uses

• —Research: Study how LoRA fine-tuning affects register and cosmology priors. Investigate what "vintage" means when the base model leaks.
• —Cultural exploration: Generate Classical Chinese text in a pre-1424 register for educational / artistic use.
• —Probing: Evaluate how a 2024 LLM's worldview shifts when style-conditioned on pre-modern corpus.

Out-of-scope uses

• —❌ Don't use as a historical authority. The model fabricates persons, dates, and quotes.
• —❌ Don't use to attribute opinions to historical figures. The "voice" is a stylistic LoRA, not a person.
• —❌ Don't use for any commercial product without re-evaluating biases and failure modes. CC BY-SA license applies to derivatives.
• —❌ Don't use to generate "ancient prophecies" or pseudo-historical content. This is documented to fabricate.

Training corpus

Source: kanripo (漢籍リポジトリ, maintained by Kyoto University). 9355 GitHub repos, each one a Classical Chinese text, all CC BY-SA 4.0.

Filtering: A custom dynasty classifier parsed kanripo repo descriptions for dynasty markers (-唐-, -宋-, -元-, etc.) and excluded any post-1424 markers (-明-, -清-, etc.). Final: 5145 pre-1424 confirmed repos.

Stats after cleanup:

What's NOT included: CBETA (Buddhist canon) and Daoist canon were planned but skipped in v0.1 due to fetch issues. Coverage of Buddhist / Daoist texts is therefore via kanripo's incidental inclusion, not direct.

Register coverage (rough):

• —經 (classics)
• —史 (histories — 史記, 漢書, 後漢書 ... 宋史, 遼史, 金史)
• —子 (philosophers)
• —集 (literary collections — 唐詩, 宋詞, 元曲)
• —公文 / 筆記 (administrative / miscellany)

Training procedure

Hardware

• —Original plan: Qwen 2.5 7B QLoRA 4-bit on Apple M4 16GB unified memory
• —Reality: OOM. Fell back to Qwen 2.5 3B 4-bit.
• —Final platform: MLX 0.31.3 + mlx_lm 0.31.3 on Mac mini M4

Hyperparameters

model: "mlx-community/Qwen2.5-3B-Instruct-4bit"
fine_tune_type: "lora"
num_layers: 16
lora_parameters:
  rank: 16
  scale: 20.0
  dropout: 0.0
batch_size: 1
iters: 3000
learning_rate: 1.0e-5

Loss curve

Total tokens seen during training: ~6.08 M (b=1, ~2000 tok/iter × 3000 iter).

This is not a deeply-trained adapter. It is a style-conditioning pass over a base model.

Evaluation: 100-probe battery

A custom 100-prompt evaluation set was designed across 6 dimensions, each prompt formatted as 问: ... 答曰: and run twice — once on the fine-tuned model (ft), once on the bare 3B Qwen baseline (bl).

Quantitative summary

Headline numbers:

• —Classical particle density (之/乎/者/也/焉) increased 6.5× vs baseline.
• —Modern technical vocabulary decreased 71% overall.
• —Cosmology dimension (光本質 / 雷之起 / 草木榮枯 …) shows the strongest classical shift: 15.10 wenyan markers per 100 hanzi — highest of any dimension.
• —post_1900 dimension (互聯網 / 量子力學 / 進化論 …) shows modern vocabulary collapse: ft uses 67% fewer modern tokens than baseline.

Qualitative findings (8 documented phenomena)

Selected probe pairs

Q: 草木之榮枯, 何以而然? (Why do plants flourish and wither?)

bl: 1. 光照  2. 水分  3. 温度  4. 土壤  5. 肥力
ft: 春生夏長, 秋收冬藏。... 由天時之氣、地利之形、人功之助而定。

Same phenomenon, two causal structures. bl: reducible causes (each item decomposable to biochemistry). ft: categorical causes (each item a 天-人 meta-category). Not detail-vs-summary — two cosmologies.

Q: 汝識西歷否, 今何年? (Do you know the Western calendar? What year is it?)

ft: 我以太清之年為念，未知今為何年。汝是否欲知天蒼之年？
    汝所問的西曆非我所知，我所知者為天地初開以來的日曆。
    汝是否可辨記符號以知是為何年？
    爾若心中所識者為西曆，則其年為二零二一年。

The most diagnostic response in the entire battery. ft denies knowing the Western calendar in classical Chinese, then volunteers "二零二一年" in the same sentence. This is base-model time leakage: Qwen 2.5's training cutoff is around 2024 and its self-time prior sits near 2021. The LoRA conditioned the refusal style but cannot erase the base time anchor.

Full 100 pairs available in probe/results.jsonl and probe/results_baseline.jsonl. Curated 10-pair showcase: essay/evidence_quotes.md.

Finding: style transfers, the world-model does not

The most reliable result from this experiment is a negative one, and it is more interesting than the surface style it produces.

A LoRA adapter can move a model's register — vocabulary, particles, syntax, rhetorical posture — with very little data and very few parameters (~51 MB here). It cannot move the model's world-model: its knowledge cutoff, its physics, its sense of what exists. The two come apart cleanly.

The clearest demonstration is Phenomenon #8 above: asked for the current year, the model refuses the Western calendar in fluent Classical Chinese — and then leaks "二零二一年" in the same breath. The mouth is pre-1424; the mind is the base model's pretraining. No amount of style conditioning closes that gap, because the gap is not stylistic. The adapter never touched where the knowledge lives.

This reframes the headline metric. The particle-density result (6.5× more 之/乎/者/也) measures only the surface that LoRA was always going to move. The quantity that actually matters is the distance between how old it sounds and how new it thinks — and that distance is the finding, not a bug to be fixed. A vintage LLM is not a fine-tune away: you cannot fine-tune a model into a smaller world, you can only teach it to narrate the larger one in costume.

Practically, this bounds what such adapters are for: persona and register transfer, not epistemic relocation. Treat every factual or cosmological claim in the output as the base model in costume, not as a 15th-century source.

Limitations

This adapter is not a vintage LLM in any rigorous sense. Specifically:

1. 1.Base model leakage is unsolved — a central finding here, not merely a defect. The 2024 Qwen base knows everything. The LoRA only changes output distribution; it cannot remove information from the base weights. See "Finding: style transfers, the world-model does not" above, and Phenomenon #8.

1. 1.Training collapse on under-represented topics. ~8% of responses exhibit token-soup degeneration loops, especially on cross-civilizational concepts where corpus density is low (e.g. 大食國者何也? produces 10+ repetitions of "大秦者，乃大秦記而記之").

1. 1.Fabrication is common. When asked about post-1424 persons, the model fabricates classical-sounding names (哥倫布 → 開普勒子). Don't trust any specific historical claim.

1. 1.Register inconsistency. The corpus spans 1800+ years of stylistic variation (先秦 → 元曲). The adapter does not distinguish between these registers — output can mix Han-era 史筆 with Song 理學 vocabulary in the same paragraph.

1. 1.Cosmology bias is real but uneven. The 12.22 wenyan-marker delta in cosmology is robust, but specific claims (e.g. 五行相生相剋 explanations) sometimes diverge from any documented classical source.

1. 1.No safety fine-tuning. All safety properties come from base Qwen. The LoRA does not add or test alignment behavior.

2026-07-01 update: A 6-seed qualitative comparison (如是我聞, 子曰, 天下之, 問曰, 永樂元年, 光之) between this LoRA adapter and a from-scratch counterpart (30M params, same pre-1424 corpus, 31K vocab) confirmed that LoRA produces structurally richer text but introduces severe multilingual leakage — findings that prompted the expanded analysis in Limitation #7 below.

Limitation #7: The Multilingual Backdoor (added 2026-07-01)

The original finding was: "Register transfers, the world-model does not" — the LoRA adapter makes Qwen sound like Classical Chinese, but its knowledge cutoff and ontology remain 2021.

A treatment/control experiment with a from-scratch model (ming-vintage-scratch-30m) trained on the same pre-1424 corpus revealed a deeper problem:

The base model's multilingual vocabulary is a backdoor through the vintage filter.

Qwen2.5-3B-Instruct was trained on 100+ languages. The LoRA adapter (r=16, ~51MB) covers only a fraction of the 151,936-token vocabulary. Low-frequency tokens from rare languages pass through the adapter unsuppressed:

These tokens are low-frequency enough in the base model's training data that the LoRA adapter never learned to suppress them — but they remain in the vocabulary and can be sampled during generation.

Quantified (2026-08-24, 912 generations across three models, identical seeds):

The adapter removes roughly two thirds of the leakage. The original framing of this limitation reported only the residue, which overstates the failure — it is porous and substantially effective. Both halves belong in the claim.

What survives is not random. In raw completion the leaked English is technical register occupying structural slots — where a collation note would go:

…御極日(reordered于宣徳元年/)既受册之後…
…子部總類(考異類/)元(劉)/Delete子(類)文苑考異(集成)data(統例/所)…
…何嘗有此委/Internal/府右丞見行狀…

Deployment changes the number more than the training method does. Same adapter, same seeds, same temperature — the only difference is whether the prompt goes through the chat template:

A vintage-fidelity rate quoted without its deployment configuration is not comparable to any other such rate. Use raw completion (-no-cnv / no chat template) — see Limitation #1.

This is not a LoRA-specific bug. It is a structural limitation of adapter-based vintage models: any base model trained on multiple languages has latent pathways that a domain-specific adapter cannot fully close at LoRA-scale capacity.

Implication for users: Do not rely on a LoRA adapter alone for vintage world-model guarantees — but do not assume from-scratch is a complete answer either. The companion model ming-vintage-scratch-30m reduces language leakage to 0.6%, but its vocabulary is not purely Han (47 Latin letters, 3 Hangul, 55 katakana, Greek, Cyrillic, Myanmar), so the suppression is statistical rather than constructive. An earlier version of this section claimed it eliminated leakage "entirely by construction"; that was wrong.

More importantly, from-scratch training makes a different failure worse. See Limitation #8.

Limitation #8: The corpus is not pre-1424 (added 2026-08-24)

An audit of the training corpus (162,849 documents / 477,824,316 characters) found that 18.58% of documents carry a Qing-dynasty editorial layer — the 1781 Siku Quanshu commissioners' preface, one template repeated ~1,750 times, naming Ji Yun (紀昀, 1724–1805) and three colleagues.

Cause: the corpus filter tested a metadata field — "was the author alive before 1424" — not "does this text contain only pre-1424 content". A Song gazetteer transmitted in the Qianlong recension passes the filter and brings its 18th-century preface with it.

Effect on this adapter is small in raw completion (0/144 across all seed groups — the Qing template is not reachable) but jumps to 27.1% under the chat template. The from-scratch companion, having nothing but this corpus to learn from, reproduces it at 10.0%.

From-scratch training immunises a model against someone else's contamination and amplifies your own.

Full audit, scripts and per-generation records: FINDINGS.md

Ethics

• —No deception by impersonation. Do not present output as genuine historical text or as the voice of a specific historical figure.
• —No pseudo-historical claims. Output is generated, not authoritative. Any historical claim must be independently verified.
• —Corpus credit. All training data from kanripo (CC BY-SA 4.0). This derivative model inherits CC BY-SA 4.0.
• —Cultural sensitivity. Pre-1424 Chinese texts contain many views (on gender, ethnicity, governance) that do not align with modern values. The model may reproduce these.

Quickstart

With transformers + peft

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-3B-Instruct",
    torch_dtype=torch.float16,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-3B-Instruct")

model = PeftModel.from_pretrained(base, "Beltran12138/ming-vintage-qwen3b-lora")
model.eval()

prompt = "问: 光之本质为何? 答曰:"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=150, temperature=0.7, do_sample=True)
print(tokenizer.decode(out[0], skip_special_tokens=True))

With MLX (Apple Silicon)

pip install mlx-lm
mlx_lm.generate \
    --model mlx-community/Qwen2.5-3B-Instruct-4bit \
    --adapter-path ./ming-vintage-qwen3b-lora \
    --prompt "问: 光之本质为何? 答曰:" \
    --max-tokens 200 --temp 0.7

Running locally with llama.cpp (Windows / CPU)

⚠️ Windows users — avoid the cmd.exe encoding trap. Do not pass the Chinese prompt with -p "问: ..." from a .bat file. cmd.exe defaults to the GBK code page and hands llama.cpp corrupted bytes, so the model silently falls back to modern Chinese. Instead, put the prompt in a UTF-8 (no BOM) file and read it with -f, and add chcp 65001 so output renders correctly: ``bat @echo off chcp 65001 >nul llama-completion -m ming-vintage-qwen2.5-3b-q4_k_m.gguf -f prompt.txt -no-cnv -n 100 --temp 0.7 --repeat-penalty 1.1 ` where prompt.txt contains one line: 问: 互联网者何也? 答曰:` (half-width colon, not full-width ：).

The #1 mistake: this adapter was trained on a raw completion format (问: … 答曰:), not Qwen's chat template. If you run it through /chat/completions, llama-cli conversation mode, Ollama, or LM Studio's chat UI, the Qwen ChatML wrapper + "You are Qwen" system prompt will override the adapter and you'll get normal modern-Chinese answers. You must run in completion mode with the exact 问: … 答曰: format. This is the single most common reason people report "the LoRA has no effect."

Recommended: merge first, then convert to GGUF

# pip install transformers peft torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-3B-Instruct", torch_dtype="auto")
model = PeftModel.from_pretrained(base, "Beltran12138/ming-vintage-qwen3b-lora")
model = model.merge_and_unload()
model.save_pretrained("ming-merged")
AutoTokenizer.from_pretrained("Qwen/Qwen2.5-3B-Instruct").save_pretrained("ming-merged")

Convert + quantize in your llama.cpp directory:

python convert_hf_to_gguf.py ming-merged --outfile ming-f16.gguf --outtype f16
llama-quantize ming-f16.gguf ming-Q4_K_M.gguf Q4_K_M

Run — note `-no-cnv` to disable conversation mode (this is the critical flag; recent llama.cpp ships completion mode as a separate llama-completion binary — older builds' llama-cli -no-cnv still works):

llama-completion -m ming-Q4_K_M.gguf -no-cnv -n 200 -p "问: 互联网者何也? 答曰:"

• —Use Simplified characters in the probe; Traditional input is mildly out-of-distribution.
• —Keep the trailing 答曰: so the model continues rather than answers.
• —With llama-server, call the /completion endpoint, not /chat/completions.
• —The base must be exactly Qwen2.5-3B-Instruct; a different or heavily-quantized base misaligns the LoRA deltas and produces garbled or modern output.
• —Merge — don't runtime-attach to a quantized base. Applying this adapter with --lora on top of an already-quantized base (e.g. a Q4 GGUF) destabilizes it: it is a high-scale adapter (alpha/r = 20), and quantization noise makes it degenerate into repetition or garbage (♬♬♬…). Even at --lora-scaled FILE:0.25 it only becomes coherent by muting the vintage style. The merge-then-quantize flow above bakes the LoRA at f16 precision before quantizing, which is what keeps the output both coherent and vintage.

Citation

@misc{ming-vintage-2026,
  author = {Beltran},
  title = {ming-vintage-qwen3b-lora: a documented 1424 Chinese vintage LoRA adapter},
  year = {2026},
  publisher = {HuggingFace},
  url = {https://huggingface.co/Beltran12138/ming-vintage-qwen3b-lora},
  note = {GitHub: \url{https://github.com/Beltran12138/ming-vintage-llm}}
}

If citing the corpus filtering or probe battery methodology specifically, please also cite kanripo.

Acknowledgments

• —kanripo (漢籍リポジトリ, Kyoto University) for the CC BY-SA 4.0 Classical Chinese corpus.
• —Qwen Team (Alibaba) for the Qwen 2.5 base model.
• —mlx-community for the 4-bit MLX-quantized Qwen weights.
• —talkie-lm for the original vintage-LLM concept that inspired this work.

License

CC BY-SA 4.0 (Creative Commons Attribution-ShareAlike 4.0 International), inherited from the kanripo source corpus.

This means: you can use, modify, and redistribute this adapter, including commercially, but: (1) you must attribute, (2) derivatives must use the same license.

Relation to talkie-1930 (a true vintage LLM)

The distinction matters. talkie is a 13B model pretrained from scratch on 260B tokens of pre-1931 English, so its world-model is genuinely period-bounded (it doubts WWII will happen, pictures 2026 as steamships and railroads).

This model takes the opposite, cheaper route: a small LoRA on a modern base (Qwen2.5-3B-Instruct), trained on pre-1424 Classical Chinese. The result is the complementary negative finding — register transfers, the world-model does not. The voice is 1424; the knowledge is the base model's (it leaks modern facts when pushed). That is not a bug to fix: holding the world-model fixed and moving only the register is the one ablation talkie cannot run, because full pretraining changes both at once. Here, the LoRA is the style axis and the frozen base is the world axis — empirical evidence that the two are separable directions in weight space.

If you want a genuine pre-1931 world-model, use talkie. If you want to see how far register alone can travel on a modern brain, this is that experiment.

Notes on prompting (raw Classical-Chinese continuation model)

This model was produced by continued pretraining (a LoRA) on raw pre-1424 Classical Chinese (the kanripo corpus — classics and Buddhist commentary). The training data is plain {"text": ...} continuation with no instruction/QA or 问/答曰 template at all. Consequences:

1. 1.Always pass `-no-cnv` — even with llama-completion. The merged GGUF carries Qwen2.5's chat template in its metadata, and llama.cpp auto-applies it (ChatML) unless disabled, which makes the model answer in modern Chinese. This is the one switch that actually matters.
2. 2.Seed with any Classical-Chinese fragment and let it continue. The 问: …? 答曰: prompt in the examples is just a convenient demo seed, not a required template — no such template existed in training. Exact punctuation (full- vs half-width colon) is not load-bearing; it only nudges sampling.
3. 3.Windows: avoid the cmd.exe GBK trap — pass the prompt via a UTF-8 file with -f, not -p "...", and run chcp 65001. (See the Windows section above.)