zhoumiaosen/minimind-64m-pretrain

MiniMind 64M — Pretrained on a Single RTX 3060

A small, dense language model trained from scratch for one pretraining epoch on a consumer NVIDIA RTX 3060 with 12 GB VRAM. This release is a base model for text continuation and further fine-tuning. It has not undergone instruction tuning, preference optimization, or alignment training.

The checkpoint is exported to the standard Transformers Qwen3ForCausalLM architecture using MiniMind's conversion utility. Qwen3 describes the compatible architecture; no pretrained Qwen weights were used. The tokenizer comes from MiniMind.

Model at a glance

Quick start

Tested with Python, PyTorch 2.6.0 and Transformers 4.57.6. Install a PyTorch build suitable for your platform, then:

pip install 'transformers==4.57.6' safetensors

After publication, load the model directly from the Hub. For a local download, set model_id to its directory:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "zhoumiaosen/minimind-64m-pretrain"
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.float16 if device == "cuda" else torch.float32
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, torch_dtype=dtype
).to(device).eval()

# Use a continuation prompt: this checkpoint is not an instruction model.
inputs = tokenizer("机器学习是一种", return_tensors="pt").to(device)
with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=64,
        do_sample=False,
        eos_token_id=tokenizer.eos_token_id,
        pad_token_id=tokenizer.pad_token_id,
    )
print(tokenizer.decode(output[0], skip_special_tokens=True))

No trust_remote_code=True is required. A tokenizer chat template may be present because the upstream tokenizer includes one; that does not make this checkpoint instruction-tuned. CPU inference is supported by the exported architecture. Raspberry Pi performance has not been measured.

Training data and procedure

The run used pretrain_t2t_mini.jsonl from jingyaogong/minimind_dataset, containing 1,270,238 records. This is a Chinese-oriented training setup; language coverage and English capability have not been independently measured. Consult the upstream dataset card for its provenance and terms. The dataset is not redistributed here.

Each record is tokenized independently, truncated to 766 text tokens, wrapped with beginning/end tokens, and padded to 768 tokens. Padding labels are ignored. The objective is causal next-token prediction. Exact non-padding training token counts were not recorded.

The machine restarted during pretraining. The run resumed from microbatch 306,000, repeating the unsaved tail. Resume restored model and optimizer state, but did not preserve partially accumulated gradients; this was not a bit-for-bit uninterrupted run.

This release preserves the upstream out/pretrain_768.pth artifact exactly through format conversion. The upstream trainer saves at the final microbatch before its trailing partial-accumulation optimizer step, so that later in-memory update is not included in the saved artifact.

Training loss

[image]

These are individual training-microbatch losses, not held-out validation results or full-epoch averages. The curve keeps the last logged reading when a step was repeated after recovery. The smoothed line averages up to 50 logged readings. The machine-readable readings are included in pretraining-loss.csv.

No held-out perplexity, standardized benchmark, factuality score, or safety evaluation is reported. Falling training loss does not establish useful question-answering ability. Any sample continuations in sample-generations.json are generation smoke checks, not quality benchmarks. Chat samples produced by a separate fine-tuned checkpoint are not results for this release.

Observed base-model continuations

A CPU smoke check used greedy decoding (do_sample=False, max_new_tokens=64) with the exported model:

These two probes show poor continuation quality with these prompts and settings. The export loads and generates successfully, but this checkpoint should be treated as a training experiment rather than a usable assistant. Both outputs and generation settings are preserved in sample-generations.json.

The exported tokenizer's model_input_names was set to input_ids and attention_mask to match the Qwen3 generation interface; token vocabulary and model weights were not changed.

Intended uses and limitations

This checkpoint is intended for studying small-model pretraining, experimenting with inference, and serving as a starting point for supervised fine-tuning. It is not a reliable assistant. Outputs may be repetitive, incoherent, factually incorrect, biased, or inappropriate. Code completions may be invalid. Long-context behavior, multilingual performance, and Raspberry Pi throughput remain untested.

Reproducibility and attribution

• —Training implementation: jingyaogong/minimind.
• —Source revision: `a3c7b01cc004d5de86aea961f20bf1e638e7c09e`.
• —Training dataset SHA256: 6dd6716c84ab36897bdbfc7f88e04f4441c48c1ab7ecee88ce0b0e7d4685560c.
• —Training environment: PyTorch 2.6.0+cu124, Transformers 4.57.6, Datasets 3.6.0.
• —Original checkpoint hash and export checks are recorded in verification.json.

Command run from the upstream repository's trainer directory:

python train_pretrain.py --epochs 1 --batch_size 4 \
  --accumulation_steps 64 --max_seq_len 768 --num_workers 0 \
  --dtype bfloat16 --device cuda:0 --from_resume 1 \
  --log_interval 100 --save_interval 1000

The upstream project is distributed under Apache 2.0; its license is included in LICENSE. This release uses the same license. Credit for the architecture implementation, tokenizer, training utilities, and dataset preparation belongs to the upstream contributors. This is an independently trained checkpoint, not an official upstream release.