igalk474/MiniFrontier-150M-Modern-3B-token-mixture

MiniFrontier 150M-Modern 5B-token mixture

Training-mixture export from MiniFrontier - AI-LLM-Transformers-Edu-Model, an educational+modern, from-scratch decoder-only language model. Each row is one admitted document (post-filter, post-dedup, pre-tokenization) with its full provenance: text, source, revision, license, language, record_id, content_hash, path, source_type, split, parent_content_hash, transform.

License is per-example, not one blanket license for the dataset. The GitHub-code component alone aggregates many different real permissive licenses (Apache-2.0, BSD-2-Clause, BSD-3-Clause, CC0-1.0, ISC, MIT, Unlicense) depending on which repository each row came from -- check the row's own license and source columns before reusing any individual example, and preserve that attribution the same way the original repository's own license requires.

Mixture composition

Weights and real token counts are measured directly from each source's own metadata.json (written by scripts/prepare_data.py) at export time, not estimated.

Repository layout

Each source is its own Hugging Face config (a separate entry in the Dataset Viewer's dropdown, and load_dataset(repo_id, "github-code") loads just that one). Place each source's scripts/prepare_data.py --export-parquet-dir output directly at the repo root, in a folder named exactly like the source, before uploading:

MiniFrontier 150M-Modern 3B-token mixture/
├── README.md          (this file)
├── dclm-edu/
│   ├── train-00000-of-00001.parquet
│   └── validation-00000-of-00001.parquet
├── fineweb-edu/
│   ├── train-00000-of-00001.parquet
│   └── validation-00000-of-00001.parquet
├── github-code/
│   ├── train-00000-of-00001.parquet
│   └── validation-00000-of-00001.parquet
├── finemath/
│   ├── train-00000-of-00001.parquet
│   └── validation-00000-of-00001.parquet
├── cosmopedia-v2/
│   ├── train-00000-of-00001.parquet
│   └── validation-00000-of-00001.parquet

The train-*/validation-* filenames are Hugging Face's own auto-detected split naming convention -- with a folder per source and this naming, the Hub's Dataset Viewer and load_dataset both work correctly with no loading script, just the configs: block above.

MiniFrontier (AI-LLM-Transformers-Edu-Model) is a clean, from-scratch PyTorch decoder-only transformer designed to train and run on a single consumer GPU (8–24 GB range). It is an educational + research-oriented implementation that deliberately stays readable while incorporating several 2024–2026 modern LLM techniques.

Key characteristics:

Strengths (what makes it good)

1. 1.Excellent pedagogical design The code is unusually well-commented and structured for learning. model.py, attention.py, rope.py, etc. contain clear “beginner’s maps.” There are dedicated labs (attention math, RoPE, MHA vs GQA, QK-Norm, KV cache, Muon vs AdamW, MTP, fused cross-entropy, etc.). This is one of the better “read the source to understand modern transformers” repos I’ve seen.

1. 1.Modern techniques done correctly
2. 2.Grouped-Query Attention (GQA)
3. 3.QK-Norm
4. 4.Hybrid local/global attention (sliding window + occasional global layers)
5. 5.RoPE (with experimental global NoPE)
6. 6.SwiGLU
7. 7.KV cache with proper handling
8. 8.Optional Multi-Token Prediction (MTP)
9. 9.Muon optimizer experiments
10. 10.Cautious AdamW
11. 11.FIM / code data paths
12. 12.Speculative decoding scaffolding
13. 13.GGUF / Hugging Face export paths

These are not just name-dropped; they are implemented and tested against each other.

1. 1.Solid engineering hygiene
2. 2.TOML configs for different sizes/presets
3. 3.Proper checkpointing, EMA, packing, sharding
4. 4.CPU test suite (191 tests claimed)
5. 5.Real home-GPU evidence (RTX 2070 Super 8 GB FP16 runs)
6. 6.Data preparation pipelines (FineWeb-Edu, FineMath, Cosmopedia, GitHub code mixes, etc.)
7. 7.Evaluation harnesses (language modeling, FIM, code, degeneration checks)
8. 8.Release / audit / reproducibility tooling

1. 1.End-to-end usability Training + inference + chat + export paths exist. You can actually train it on your home PC and run it.

Limitations / honest assessment

• —Scale: 150 M parameters is firmly in the “educational / research toy / local assistant” regime. It will not compete with 7 B–70 B models on knowledge, reasoning, or coding quality. Expect it to be fluent on common patterns but limited in depth and world knowledge.
• —Inference stack: Pure PyTorch + some llama.cpp adapter work. It is functional but not as optimized as vLLM / SGLang / TensorRT-LLM / custom CUDA kernels for maximum tokens/sec.
• —Data & evaluation: Solid local pipelines, but not yet at the level of the large curated mixes + rigorous public benchmarks that frontier labs use. GPQA-diamond style local tasks exist, which is good.

Overall verdict

For a home-built, single-GPU educational LLM that you can both train and run inference on, this is above average and closer to the high end of open educational efforts. The code quality, modern feature coverage, test discipline, and documentation put it in a different league from most “I trained a small transformer” GitHub projects.

It succeeds at its stated goal: making the path from classic Transformer fundamentals to a useful set of modern techniques visible, testable, and measurable on consumer hardware.

more ways to use the dataset to train model

Yes — Hugging Face Transformers + PEFT / TRL (especially with Unsloth) is one of the most common and practical ways people do continued pre-training or LoRA on custom datasets like yours when they need a clean path to GGUF for llama.cpp.

It is flexible, well-documented, works on a single consumer GPU on Windows, and the final conversion step is straightforward.

Install via pip install unsloth or plain transformers + peft + trl + datasets + accelerate (Unsloth’s Windows wheels are the least painful).

Load any similar-size base (SmolLM2-135M, Qwen2.5-0.5B, etc.), then do continued pre-training or LoRA on the parquet data (Hugging Face datasets loads the parquets directly).

After training, merge LoRA → convert to GGUF with the official converthfto_gguf.py from llama.cpp.

Works well on a single consumer GPU; you stay in the HF ecosystem until the final GGUF step.

Best lazy already ready , base model to use with this dataset

Primary recommendation: `HuggingFaceTB/SmolLM2-135M` (the Base version)

Why this one fits best:

• —Size is almost perfect match (~135M vs the dataset’s intended 150M target).
• —Very high quality for its size (trained on 2T tokens with strong educational/synthetic data that overlaps with Cosmopedia-style text in your mixture).
• —Llama-style architecture → converts cleanly and reliably to GGUF with the official convert_hf_to_gguf.py.
• —Excellent Unsloth support (fastest + lowest VRAM on Windows).
• —Easy to continue pre-training on raw text (your parquet text column) or do LoRA.

Strong alternatives (pick based on your GPU):

Avoid starting from Instruct versions if your goal is continued pre-training on the raw text mixture — use the Base checkpoints. You can always do a short SFT/chat stage later if needed.

Typical workflow (high level)

1. 1.Install (Unsloth is the least painful on Windows):

   pip install unsloth
   # or the longer plain way: transformers peft trl datasets accelerate

1. 1.Load the base + your dataset:

   from unsloth import FastLanguageModel
   from datasets import load_dataset

   model, tokenizer = FastLanguageModel.from_pretrained(
       model_name = "HuggingFaceTB/SmolLM2-135M",
       max_seq_length = 2048,   # or 1024 if VRAM is tight
       dtype = None,
       load_in_4bit = True,     # or False for full precision if you have VRAM
   )

   dataset = load_dataset("igalk474/MiniFrontier-150M-Modern-3B-token-mixture", split="train")
   # or load the local parquet files directly

1. 1.Add LoRA (recommended) or do full continued pre-training, then train with TRL’s SFTTrainer (or the continued-pretrain style).

1. 1.After training:
2. 2.Merge LoRA into the base (Unsloth has a one-liner for this).
3. 3.Convert the resulting Hugging Face folder to GGUF:

     python convert_hf_to_gguf.py ./your-merged-model --outfile my-model-f16.gguf --outtype f16

• —(Optional) Quantize further with llama.cpp’s quantize tool to Q80 / Q6K / Q5KM / Q4KM.

1. 1.Serve with llama.cpp:

   llama-server -m my-model-q5_k_m.gguf -c 2048 --port 8080

Then point the Vercel AI SDK at http://localhost:8080/v1.