Modelpublic

ML-is-Fun/Fleck-M-500K

sourceHugging Facemitupdated 23d agoView on Hugging Face

0likes680downloads

Model Card

Fleck-M-500K

[Fleck-M-500K Instruct](https://huggingface.co/ML-is-Fun/Fleck-M-500K) — the independently instruction-tuned child of `Fleck-M-500K-Base`.

—497,288 parameters
—BF16 SafeTensors weights
—Fine-tuned from the Base-100M model on Dolly-15k
—Intended for local conversational experiments on Apple Silicon

Model Details


Architecture	Decoder-only Transformer
Parameters	497,288
Hidden size	128
FFN size	336
Physical blocks	2
Effective depth	4 (`A → B → A → B`)
Attention	GQA — 4 query heads, 2 KV heads, head dimension 32
Normalization	RMSNorm
Embedding	Factorized tied embedding, rank 64
Vocabulary	2,048
Context length	2,048 tokens
Canonical dtype	BF16
Base model	`ML-is-Fun/Fleck-M-500K-Base`

Training

Pretraining

The Base parent was initialized from scratch and pretrained for exactly 100,000,000 real tokenizer tokens on a 70/30 FineWeb-Edu/FineWeb mixture.

Instruction Tuning


Base model	`Fleck-M-500K-Base`
Dataset	`databricks/databricks-dolly-15k`
Dataset license	CC BY-SA 3.0
Requested budget	250K supervised tokens
Realized supervised tokens	216,070
Optimizer	AdamW
Instruction tuning	Assistant-response supervision
Hardware	Apple M2 (10-core GPU)

Benchmark Results

Corrected zero-shot evaluation, no chat template, FP32 evaluation, and the same benchmark aggregation protocol were used for both variants.

Task	Metric	Base	Instruct	Δ
HellaSwag	`acc_norm`	25.04%	24.67%	-0.38pp
PIQA	`acc_norm`	51.31%	50.98%	-0.33pp
ARC-Easy	`acc_norm`	26.05%	26.09%	+0.04pp
ARC-Challenge	`acc_norm`	24.15%	24.23%	+0.09pp
LAMBADA OpenAI	`acc`	0.04%	0.08%	+0.04pp
WinoGrande	`acc`	51.30%	50.99%	-0.32pp
BoolQ	`acc`	37.83%	37.83%	0.00pp
MMLU (57-subject macro)	`acc`	23.14%	23.20%	+0.06pp
Eight-task mean	—	29.86%	29.76%	-0.10pp

Tokenizer and Chat Format

—Fleck-Tokenizer-2048
—Byte-level BPE
—Vocabulary size: 2,048

The training chat format is:

text

<bos><|user|>{user}<|eot|><|assistant|>{response}<|eot|><eos>

Special Tokens

Token	ID	Role
`<bos>`	0	sequence start
`<eos>`	1	sequence end
`<pad>`	2	padding
`<unk>`	3	unknown token
`<\	system\	>`	4	system turn
`<\	user\	>`	5	user turn
`<\	assistant\	>`	6	assistant turn
`<\	eot\	>`	7	end of turn

Usage

The bundle includes a self-contained inference.py; it does not import the Fleck-LM checkout. The accompanying config.json, generation_config.json, and tokenizer_config.json describe the custom architecture and generation/tokenizer defaults; standard transformers.AutoModel loading is not supported. The chat_template.jinja file contains the Instruct chat template used by compatible HF tooling. Install the three runtime dependencies:

bash

python -m pip install torch safetensors tokenizers

By default the CLI starts an interactive chat. /exit quits and /clear resets the conversation history:

bash

python inference.py --device cpu --max-tokens 32

For one prompt without interactive mode, pass --no-chat and --prompt:

bash

python inference.py \
  --ckpt model.safetensors \
  --tokenizer tokenizer.json \
  --no-chat \
  --prompt "Explain what a tokenizer does." \
  --max-tokens 32 \
  --device cpu

The script reads and runs the BF16 checkpoint without an FP32 model copy, validates every SafeTensors key, shape, and dtype, and uses FP32 only for attention score/softmax and tied-logit accumulation. It reproduces the factorized tied embedding/logits, effective-depth execution A → B → A → B, half-split RoPE, GQA, physical KV caches, RMSNorms, and greedy generation without repository-local imports. Generation stops on <|eot|> or <eos>.

Limitations

This model is extremely small and is intended for research and local experimentation rather than reliable general-purpose assistance. It may produce repetitions, malformed text, weak factual answers, or incoherent responses. Instruction tuning improves conversational behavior but does not overcome the limits of a 497,288 parameter model.

License

MIT License.

Files

The public bundle contains these files:

—README.md — model card and usage documentation
—inference.py — standalone strict loader and interactive/single-prompt inference CLI
—model.safetensors — BF16 model weights
—tokenizer.json — standalone tokenizer
—config.json — custom architecture configuration
—generation_config.json — greedy generation defaults
—tokenizer_config.json — tokenizer defaults and special-token mapping
—chat_template.jinja — Instruct chat template No training data, optimizer state, or other training outputs are included.