Modelpublic

llaa33219/MicroMixer-3-500K-discord-dialogues

sourceHugging Faceapache-2.0updated 3mo agoView on Hugging Face

0likes17downloads

Model Card

MicroMixer-3-500K-discord-dialogues

<table> <tr> <td align="center" style="padding: 20px;"> <strong style="color: #007BFF; font-size: 1.2em;">Micro Language Model</strong><br/> <em>Attention-Free • MLP-Only • Byte-Level • Factorized State-Content</em> </td> </tr> </table>

![GitHub](https://github.com/llaa33219/MicroMixer-3)

</div>

📋 Overview

MicroMixer-3-500K-discord-dialogues is a ~515K parameter Factorized State-Content MLP-Mixer (FSC-Mixer) language model trained on Discord conversation data. The 500K variant shares the same 8-layer block structure and full state-dilation schedule (1,2,4,8,16,32,32,32) as the 1M flagship, but at half the hidden width — a mid-tier trade-off between speed and capacity.

</div>

🏗️ Architecture

mermaid

graph TD
    A[Byte Input] --> B[Embed 256→96 NoPE]
    B --> C[FSC-Mixer Block × 8]
    C --> D[RMSNorm]
    D --> E[LM Head Tied with Embed]
    E --> F[Byte Output]

    subgraph "FSC-Mixer Block"
        X[Input 96] --> Split
        Split --> Cc[Content 48]
        Split --> Cs[State 48]

        Cc --> RN1[RMSNorm] --> CTM[CausalDSConv1d k=3 dil=1]
        CTM --> CCM[Channel MLP 4×]
        CCM --> Cc2[Content Out]

        Cs --> RN2[RMSNorm] --> STM[CausalDSConv1d k=3 dil=d_l]
        STM --> SCM[Channel MLP 2×]
        SCM --> Cs2[State Out]

        Cc2 --> GateRecomb
        Cs2 --> GateRecomb
        GateRecomb["g⊙c + (1-g)⊙W_s@s"] --> Out[96 concat]
    end

    style A fill:#007BFF,color:#fff
    style F fill:#00D620,color:#fff
    style GateRecomb fill:#AE00FF,color:#fff
    style CTM fill:#FF6600,color:#fff
    style STM fill:#FF6600,color:#fff

</div>

Model Configuration

<table> <tr> <th style="background-color: #007BFF; color: white;">Parameter</th> <th style="background-color: #AE00FF; color: white;">Value</th> </tr> <tr><td>Total Parameters</td><td><code>515,040</code></td></tr> <tr><td>Hidden Dimension (dmodel)</td><td><code>96</code></td></tr> <tr><td>Content Dimension (dcontent)</td><td><code>48</code></td></tr> <tr><td>State Dimension (d_state)</td><td><code>48</code></td></tr> <tr><td>Number of Layers</td><td><code>8</code></td></tr> <tr><td>State Dilation Schedule</td><td><code>(1, 2, 4, 8, 16, 32, 32, 32)</code></td></tr> <tr><td>Content Dilation</td><td><code>1</code> (local)</td></tr> <tr><td>State Receptive Field</td><td><code>255 bytes</code> by layer 8</td></tr> <tr><td>Content Channel MLP Expansion</td><td><code>4×</code></td></tr> <tr><td>State Channel MLP Expansion</td><td><code>2×</code></td></tr> <tr><td>Max Sequence Length</td><td><code>1024</code></td></tr> <tr><td>Vocabulary Size</td><td><code>256</code> (Byte-level)</td></tr> <tr><td>Position Encoding</td><td><code>NoPE</code> (causal structure provides implicit position)</td></tr> <tr><td>Activation</td><td><code>GELU</code></td></tr> <tr><td>Normalization</td><td><code>RMSNorm</code></td></tr> </table>

Core Components

┌────────────────────────────────────────────────────┐
│              FSC-Mixer Block (×8)                   │
│  ┌──────────────────────────────────────────┐      │
│  │  Content Branch                          │      │
│  │  RMSNorm → CausalDSConv1d(k=3,d=1) → +  │      │ ← Local morphology
│  │  Channel MLP (4×) → +                    │      │
│  ├──────────────────────────────────────────┤      │
│  │  State Branch                            │      │
│  │  RMSNorm → CausalDSConv1d(k=3,d=d_l) → + │      │ ← Long-range syntax
│  │  Channel MLP (2×) → +                    │      │   (dilations exponentially)
│  ├──────────────────────────────────────────┤      │
│  │  State-Gated Recombination               │      │
│  │  g = σ(Linear_s(s))                      │      │ ← Attention equivalent
│  │  out = g⊙c + (1-g)⊙(W_s@s)               │      │   (linear + sigmoid)
│  └──────────────────────────────────────────┘      │
└────────────────────────────────────────────────────┘

</div>

1️⃣ Causal Depthwise-Separable Conv (Token Mixing)

—Content branch: dilation=1, captures local morphology (3-byte window)
—State branch: dilations grow exponentially (1,2,4,8,16,32,32,32), reaching a 255-byte receptive field by layer 8
—Pure convolution → fully parallel across the time dim, no Python loops

2️⃣ Channel MLPs

—Content: Linear → GELU → Linear with 4× expansion
—State: Linear → GELU → Linear with 2× expansion (smaller, because state is meant to be a "summary")

3️⃣ State-Gated Recombination (MLP-Mixer "Attention Equivalent")

—g = σ(Linear_s(s)) — gate computed from the state branch
—out = g ⊙ c + (1-g) ⊙ (W_s @ s) — state modulates content via a learned, content-dependent gate
—No Q·K^T scores, no O(n) state update — all linear + sigmoid (true MLP)

🎯 Generation Examples

Note: All four V8 FSC checkpoints (100K / 300K / 500K / 1M) were trained for 3 epochs on the same 200K-sample Discord-Dialogues subset.

[Prompt] User: hey anyone here?
Assistant:
[Output] *ASE FROM PLS SING THE MCQuickers
Usell bonus
Um in my graves. He's just so cool to achieve everything was the worst and if its js shape of him reviewed an opinion

And you like mixed or display is to

[Prompt] User: what's the best way to learn python?
Assistant:
[Output]
UwU. How you did make company with my mind?
Use she can't remember that

ASSISTANt: I feel like alot of tests doesn't
Ustraling higher name?
Uma who is too much better

Attempted because of

Again th

[Prompt] User: lol that was hilarious
Assistant:
[Output]
Ult, you can put a machine
Usens advice for a server but im gonna wake up in like one?
UsEr: at least i don't lie i see yo boring

Also is going to all its pay on a free and i need food, just with a

</div>

What the Generations Show

—Multi-speaker dialogue structure: Use, UseR:, UsEr:, ASSISTANt:, Asser: — the model has learned speaker-turn formatting
—Contractions: don't, I've, I'm, can't
—Conjunctions: Also, And, But
—SVO fragments: I + verb + object constructions
—No repetition loops: rep-3 / rep-4 are essentially 0% across all generations (V7 had severe loops)

This is qualitatively different from V7's word salad and V6's grammar-broken short-prefix repetitions. Even at 3 epochs, V8 produces grammatical multi-speaker dialogue.

🌊 Long-Context Generation (1024 tokens)

A key property of V8's factorized state branch is that the state receptive field grows exponentially with depth (255 bytes by layer 8). The result: even at the model's full 1024-token generation length, grammatical accuracy is preserved across the entire output — speaker turns, contractions, and SVO structure hold up at the 1024th token, not just the first 100.

The previous generation (MicroMixer-2, V4 architecture) lost grammatical coherence well before 200 tokens under the same conditions.

[Prompt] User: hey anyone here?
Assistant:
[Output, 1024 tokens, rep-3: 0.0% | rep-4: 0.0%]
*ASE FROM PLS SING THE MCQuickers
Usell bonus
Um in my graves. He's just so cool to achieve everything was the worst and if its js shape of him reviewed an opinion

And you like mixed or display is to
[… full 1024 tokens, multi-speaker dialogue with consistent grammar throughout …]

Long-Context Properties

—Speaker turns remain formatted through all 1024 tokens: Use, UseR:, UsEr:, ASSISTANt:, Asser: — no formatting collapse
—Contractions preserved end-to-end: don't, I've, I'm, it's, don't
—Conjunctions distributed throughout: And, Also, But, Actually
—Zero repetition at the full 1024-token horizon (rep-3, rep-4 = 0.0%)
—Sub-word noise (tht, ofthe, js, Ustraling) is byte-level tokenizer artifact, not grammatical failure
—Semantic incoherence still grows with length (expected at sub-1M), but the syntactic skeleton holds

</div>

📊 Training Results

Metric	Value
Train Loss (final)	1.2329
Train PPL (final)	3.43
Val Loss	1.2229
Val PPL	3.40
Epochs Trained	3
Global Steps	35,625
Best Val Loss	1.2229
Throughput	~298,000 tok/s
Optimizer	AdamW
Scheduler	WSD (warmup-stable-decay)
Learning Rate	3e-3
Weight Decay	0.01
Warmup Steps	500
Max Grad Norm	1.0
Batch Size	16
Hardware	RTX 4060 Ti
Training Time (3 epochs)	~25 min

</div>

V8 Family Comparison (3 epochs, same data)

Size	Params	Val PPL	Val Loss	Tok/s	Epoch Time	Total Time
100K	110,016	3.80	1.3351	~500k	~6 min	~14 min
300K	277,120	3.52	1.2592	~365k	~9 min	~19 min
500K	515,040	3.40	1.2229	~298k	~10 min	~25 min
1M	899,712	3.32	1.1992	~285k	~10 min	~26 min

Scaling is monotonic: more parameters → better PPL, with the 1M checkpoint reaching the strongest validation perplexity of the family.

📊 Training Data

Dataset: Discord-Dialogues

—7.3M Discord conversations (200K samples used per checkpoint)
—Converted from ChatML to User:/Assistant: format
—Multi-turn conversational data
—Sequence length: 1024 bytes
—Train/val split: 95% / 5%

</div>

🔧 Usage

Files in this repository

—epoch_{0,1,2}.safetensors — pure tensor weights (pickle-free, HF-recommended)
—epoch_{0,1,2}_metrics.json — per-epoch training metrics (loss, PPL, etc.)
—config.json — model hyperparameters (vocabsize, dmodel, dilations, …)
—config.txt — human-readable config summary

Load and generate (safetensors — no pickle)

python

import json
import torch
from safetensors.torch import load_file
from src.model_v8_fsc import MicroMixerV8FSC, V8Config
from src.tokenizer import ByteTokenizer

# Clone the repository first:
# git clone https://github.com/llaa33219/MicroMixer-3.git
# cd MicroMixer-3

# 1. Load config from JSON (no pickle)
with open("checkpoints/discord-v8fsc-500k-1024/config.json") as f:
    cfg = V8Config(**json.load(f))

# 2. Load weights from safetensors (no pickle)
model = MicroMixerV8FSC(cfg)
state = load_file("checkpoints/discord-v8fsc-500k-1024/epoch_2.safetensors")
model.load_state_dict(state)
model.eval()

# 3. Generate
tokenizer = ByteTokenizer()
input_ids = torch.tensor(
    [tokenizer.encode("User: hello\nAssistant: ")]
)
with torch.no_grad():
    output = model.generate(
        input_ids,
        max_new_tokens=200,
        temperature=0.8,
        top_k=40,
        top_p=0.9,
        repetition_penalty=1.2,
        no_repeat_ngram_size=4,
    )
print(tokenizer.decode(output[0].tolist()))

Load from Hugging Face Hub (no clone required)

python

import json
import torch
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from src.model_v8_fsc import MicroMixerV8FSC, V8Config
from src.tokenizer import ByteTokenizer

REPO = "llaa33219/MicroMixer-3-v8fsc-discord-500K"

cfg_path   = hf_hub_download(REPO, "config.json")
ckpt_path  = hf_hub_download(REPO, "epoch_2.safetensors")

cfg = V8Config(**json.load(open(cfg_path)))
model = MicroMixerV8FSC(cfg)
model.load_state_dict(load_file(ckpt_path))
model.eval()

# ... generate as above

CLI (loads from the local clone)

bash

uv run python infer_v8_fsc.py --ckpt-dir checkpoints/discord-v8fsc-500k-1024 --epoch 2

⚠️ Limitations

Limitation	Description
Sub-1M Parameters	Capacity-limited; ~2M bits of learnable knowledge (Allen-Zhu 2024)
Byte-Level Noise	256-vocab byte tokenizer makes PPL noisier than BPE baselines
Word-Level Incoherence	Generations show grammatical structure but garbled semantics
Long-Range (≥256 bytes)	State branch's 255-byte receptive field is the effective context horizon
3-Epoch Training Only	V8 keeps improving with more epochs; expect PPL ~3.0 with 5-10 epochs
Research Use Only	Designed for architecture experimentation, not production deployment

</div>

🧬 Lineage: Why V8 Exists

Version	Val PPL	Outcome	Why it failed / succeeded
V6 (multi-scale Toeplitz)	4.08 (after 91h)	Grammar-broken outputs; short repetitive prefixes at long context	Muon+WD orthogonalized (3, 4096) Toeplitz kernel to L2 ≈ 0.013 — mixer effectively collapsed
V7 (7-technique stack)	11.99 (after 3.8h)	Word salad (real words, broken grammar)	All 7 techniques competed for the same hidden capacity — no channel dedicated to syntax
V8 FSC-Mixer	3.40 (after 25 min)	Multi-speaker dialogue with grammar	Dedicate 50% of every layer to an explicit, long-range syntactic state pathway

The single architectural insight that made V8 work: V7 lacked a dedicated channel for syntactic state. V8's state branch (d_s per layer, dilated causal conv, state-gated recombination) gives the model an explicit place to encode "what syntactic context am I in" — separate from "what byte comes next."

![GitHub](https://github.com/llaa33219/MicroMixer-3)

<sub>Part of the <a href="https://github.com/llaa33219/MicroMixer-3">MicroMixer-3</a> research project — V8 (FSC-Mixer) family</sub>

</div>