Modelpublic

llaa33219/MicroMixer-3-1M-discord-dialogues

sourceHugging Faceapache-2.0updated 3mo agoView on Hugging Face

0likes21downloads

Model Card

MicroMixer-3-1M-discord-dialogues

<table> <tr> <td align="center" style="padding: 20px;"> <strong style="color: #007BFF; font-size: 1.2em;">Micro Language Model</strong><br/> <em>Attention-Free • MLP-Only • Byte-Level • Factorized State-Content</em> </td> </tr> </table>

![GitHub](https://github.com/llaa33219/MicroMixer-3)

</div>

📋 Overview

MicroMixer-3-1M-discord-dialogues is a ~900K parameter Factorized State-Content MLP-Mixer (FSC-Mixer) language model trained on Discord conversation data. This is the flagship model in the family — the largest variant that still fits under the strict sub-1M parameter budget, with 8 FSC-Mixer blocks and a full 255-byte state-branch receptive field.

</div>

🏗️ Architecture

mermaid

graph TD
    A[Byte Input] --> B[Embed 256→128 NoPE]
    B --> C[FSC-Mixer Block × 8]
    C --> D[RMSNorm]
    D --> E[LM Head Tied with Embed]
    E --> F[Byte Output]

    subgraph "FSC-Mixer Block"
        X[Input 128] --> Split
        Split --> Cc[Content 64]
        Split --> Cs[State 64]

        Cc --> RN1[RMSNorm] --> CTM[CausalDSConv1d k=3 dil=1]
        CTM --> CCM[Channel MLP 4×]
        CCM --> Cc2[Content Out]

        Cs --> RN2[RMSNorm] --> STM[CausalDSConv1d k=3 dil=d_l]
        STM --> SCM[Channel MLP 2×]
        SCM --> Cs2[State Out]

        Cc2 --> GateRecomb
        Cs2 --> GateRecomb
        GateRecomb["g⊙c + (1-g)⊙W_s@s"] --> Out[128 concat]
    end

    style A fill:#007BFF,color:#fff
    style F fill:#00D620,color:#fff
    style GateRecomb fill:#AE00FF,color:#fff
    style CTM fill:#FF6600,color:#fff
    style STM fill:#FF6600,color:#fff

</div>

Model Configuration

<table> <tr> <th style="background-color: #007BFF; color: white;">Parameter</th> <th style="background-color: #AE00FF; color: white;">Value</th> </tr> <tr><td>Total Parameters</td><td><code>899,712</code></td></tr> <tr><td>Hidden Dimension (dmodel)</td><td><code>128</code></td></tr> <tr><td>Content Dimension (dcontent)</td><td><code>64</code></td></tr> <tr><td>State Dimension (d_state)</td><td><code>64</code></td></tr> <tr><td>Number of Layers</td><td><code>8</code></td></tr> <tr><td>State Dilation Schedule</td><td><code>(1, 2, 4, 8, 16, 32, 32, 32)</code></td></tr> <tr><td>Content Dilation</td><td><code>1</code> (local)</td></tr> <tr><td>State Receptive Field</td><td><code>255 bytes</code> by layer 8</td></tr> <tr><td>Content Channel MLP Expansion</td><td><code>4×</code></td></tr> <tr><td>State Channel MLP Expansion</td><td><code>2×</code></td></tr> <tr><td>Max Sequence Length</td><td><code>1024</code></td></tr> <tr><td>Vocabulary Size</td><td><code>256</code> (Byte-level)</td></tr> <tr><td>Position Encoding</td><td><code>NoPE</code> (causal structure provides implicit position)</td></tr> <tr><td>Activation</td><td><code>GELU</code></td></tr> <tr><td>Normalization</td><td><code>RMSNorm</code></td></tr> </table>

Core Components

┌────────────────────────────────────────────────────┐
│              FSC-Mixer Block (×8)                   │
│  ┌──────────────────────────────────────────┐      │
│  │  Content Branch                          │      │
│  │  RMSNorm → CausalDSConv1d(k=3,d=1) → +  │      │ ← Local morphology
│  │  Channel MLP (4×) → +                    │      │
│  ├──────────────────────────────────────────┤      │
│  │  State Branch                            │      │
│  │  RMSNorm → CausalDSConv1d(k=3,d=d_l) → + │      │ ← Long-range syntax
│  │  Channel MLP (2×) → +                    │      │   (dilations exponentially)
│  ├──────────────────────────────────────────┤      │
│  │  State-Gated Recombination               │      │
│  │  g = σ(Linear_s(s))                      │      │ ← Attention equivalent
│  │  out = g⊙c + (1-g)⊙(W_s@s)               │      │   (linear + sigmoid)
│  └──────────────────────────────────────────┘      │
└────────────────────────────────────────────────────┘

</div>

1️⃣ Causal Depthwise-Separable Conv (Token Mixing)

—Content branch: dilation=1, captures local morphology (3-byte window)
—State branch: dilations grow exponentially (1,2,4,8,16,32,32,32), reaching a 255-byte receptive field by layer 8
—Pure convolution → fully parallel across the time dim, no Python loops

2️⃣ Channel MLPs

—Content: Linear → GELU → Linear with 4× expansion
—State: Linear → GELU → Linear with 2× expansion (smaller, because state is meant to be a "summary")

3️⃣ State-Gated Recombination (MLP-Mixer "Attention Equivalent")

—g = σ(Linear_s(s)) — gate computed from the state branch
—out = g ⊙ c + (1-g) ⊙ (W_s @ s) — state modulates content via a learned, content-dependent gate
—No Q·K^T scores, no O(n) state update — all linear + sigmoid (true MLP)

🎯 Generation Examples

Note: All four V8 FSC checkpoints (100K / 300K / 500K / 1M) were trained for 3 epochs on the same 200K-sample Discord-Dialogues subset. The 1M is the strongest of the four.

[Prompt] User: hey anyone here?
Assistant:
[Output] Usesselheld time i will be great! (that's why I live in bullet, now ofc

Also a bit at all.
UsEr: Basically I'm still

Any shape ruining is an idea of the colours

Atk, sorry. Service I missed

[Prompt] User: i'm so tired today
Assistant:
[Output] how to upgrade more
Use team fr? I've been watching the entiring

Air but mods of flex together. if you don't like a working, bullying)

And see what races

Also I was going. - damn this actual pl

[Prompt] User: guess what i just found
Assistant:
[Output]
Use all of you

but in first?
Usin the can do on top of

And im not so sad

Who are yosu

Also I juSt build some office for it?

ASSistent far those movies, connection =)
Usophen but
Uscread

</div>

What the Generations Show

—Multi-speaker dialogue structure: Use, UseR:, UsEr:, ASSISTANt:, Asser: — the model has learned speaker-turn formatting
—Contractions: don't, I've, I'm, can't
—Conjunctions: Also, And, But
—SVO fragments: I + verb + object constructions
—No repetition loops: rep-3 / rep-4 are essentially 0% across all generations (V7 had severe loops)

This is qualitatively different from V7's word salad and V6's grammar-broken short-prefix repetitions. Even at 3 epochs, V8 produces grammatical multi-speaker dialogue.

🌊 Long-Context Generation (1024 tokens)

A key property of V8's factorized state branch is that the state receptive field grows exponentially with depth (255 bytes by layer 8). The result: even at the model's full 1024-token generation length, grammatical accuracy is preserved across the entire output — speaker turns, contractions, and SVO structure hold up at the 1024th token, not just the first 100.

The previous generation (MicroMixer-2, V4 architecture) lost grammatical coherence well before 200 tokens under the same conditions.

[Prompt] User: tell me a story about a brave knight
Assistant:
[Output, 1024 tokens, rep-3: 0.0% | rep-4: 0.0%]
how do you trade finality
Usei: ok

Any rook who know the best way to deal with an idea but never getting considering, it will be more long

Am in another version? Im annoying. That sounds funny, bunnies in oxable lines too

Actually it's helpful instead of cutties
UsEr: Oh, I didn't knew it

Who's probably both

Also its my opinion

Ago that's, though

Dont defend from it.

A cool good event things and not enoug/pull throug dumb tower arounD computer thumbs position downloaded account on week building
Usual for tech players thn

assintaningly got out.. thd shelden response back, college french is fried coconut mania could just add th sidiseph togetheaters
Usse yo disappeared wishes only tht sticker. I regret it?

At mine widge which doesn't mean shouls tries, bro investments are runners.

The costs so I can't.
How abt dont, I've plot less weird dota is) founs our time
Usapas as take away

Ain"thani has it, because was gonna beat myself after

Are yeah he language lung
I started talking

Aft

[Prompt] User: how does a computer work?
Assistant:
[Output, 1024 tokens, rep-3: 0.0% | rep-4: 0.0%]
buff
Use pony, good holy enemy player fingers too

And you shouldn't be able to make me wonderful i dont do this then solo seeds, in a red blood
Usablier has one shake

Also yo it's going to

Some of those threat. Trying, if yogurtade hates yoyo road.

Agreeing?
UsEr: Im not real cute or earliest invasion release as almost 3 so i got copy yo

Ohh because its mindset pancakes

A doubles texture tbh lamutations

There'll begin if? Are yoo correct yonko fans, I might back to, mosquater

ASS was supposedly based off trials we talking
Ussel since tracked with people are somehow. That" is amazingly rare

I guess what can I do

He just made, i feel like iffield literally having! Mostly anyways

Is tf isn't.

She's junk, whenever I woulD need to. You caught

Aquest)

At learning. (idk hope all that

Charge team will help me, honestlied is
Usreach it rn

Actio tournament whos years it's taken ofc

Am at 11k fish is, man bein time
Usint a few titanium coulbach

Aim missed. Tiller istendous

Goodbye gives him

Long-Context Properties

—Speaker turns remain formatted through all 1024 tokens: Use, UseR:, UsEr:, ASS:, Ussel, Usreach — no formatting collapse
—Contractions preserved end-to-end: don't, I've, I'm, don't, you've, woulD
—Conjunctions distributed throughout: And, Also, But, Actually, Some of, Ago that's, though
—Zero repetition at the full 1024-token horizon (rep-3, rep-4 = 0.0%)
—Sub-word noise (tht, ofthe, lamutations) is byte-level tokenizer artifact, not grammatical failure
—Semantic incoherence still grows with length (expected at sub-1M), but the syntactic skeleton holds

</div>

📊 Training Results

Metric	Value
Train Loss (final)	1.2071
Train PPL (final)	3.34
Val Loss	1.1992
Val PPL	3.32
Epochs Trained	3
Global Steps	35,625
Best Val Loss	1.1992
Throughput	~285,000 tok/s
Optimizer	AdamW
Scheduler	WSD (warmup-stable-decay)
Learning Rate	3e-3
Weight Decay	0.01
Warmup Steps	500
Max Grad Norm	1.0
Batch Size	16
Hardware	RTX 4060 Ti
Training Time (3 epochs)	~26 min

</div>

V8 Family Comparison (3 epochs, same data)

Size	Params	Val PPL	Val Loss	Tok/s	Epoch Time	Total Time
100K	110,016	3.80	1.3351	~500k	~6 min	~14 min
300K	277,120	3.52	1.2592	~365k	~9 min	~19 min
500K	515,040	3.40	1.2229	~298k	~10 min	~25 min
1M	899,712	3.32	1.1992	~285k	~10 min	~26 min

Scaling is monotonic: more parameters → better PPL, with the 1M checkpoint reaching the strongest validation perplexity of the family.

📊 Training Data

Dataset: Discord-Dialogues

—7.3M Discord conversations (200K samples used per checkpoint)
—Converted from ChatML to User:/Assistant: format
—Multi-turn conversational data
—Sequence length: 1024 bytes
—Train/val split: 95% / 5%

</div>

🔧 Usage

Files in this repository

—epoch_{0,1,2}.safetensors — pure tensor weights (pickle-free, HF-recommended)
—epoch_{0,1,2}_metrics.json — per-epoch training metrics (loss, PPL, etc.)
—config.json — model hyperparameters (vocabsize, dmodel, dilations, …)
—config.txt — human-readable config summary

Load and generate (safetensors — no pickle)

python

import json
import torch
from safetensors.torch import load_file
from src.model_v8_fsc import MicroMixerV8FSC, V8Config
from src.tokenizer import ByteTokenizer

# Clone the repository first:
# git clone https://github.com/llaa33219/MicroMixer-3.git
# cd MicroMixer-3

# 1. Load config from JSON (no pickle)
with open("checkpoints/discord-v8fsc-1m-1024/config.json") as f:
    cfg = V8Config(**json.load(f))

# 2. Load weights from safetensors (no pickle)
model = MicroMixerV8FSC(cfg)
state = load_file("checkpoints/discord-v8fsc-1m-1024/epoch_2.safetensors")
model.load_state_dict(state)
model.eval()

# 3. Generate
tokenizer = ByteTokenizer()
input_ids = torch.tensor(
    [tokenizer.encode("User: hello\nAssistant: ")]
)
with torch.no_grad():
    output = model.generate(
        input_ids,
        max_new_tokens=200,
        temperature=0.8,
        top_k=40,
        top_p=0.9,
        repetition_penalty=1.2,
        no_repeat_ngram_size=4,
    )
print(tokenizer.decode(output[0].tolist()))

Load from Hugging Face Hub (no clone required)

python

import json
import torch
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from src.model_v8_fsc import MicroMixerV8FSC, V8Config
from src.tokenizer import ByteTokenizer

REPO = "llaa33219/MicroMixer-3-v8fsc-discord-1M"

cfg_path   = hf_hub_download(REPO, "config.json")
ckpt_path  = hf_hub_download(REPO, "epoch_2.safetensors")

cfg = V8Config(**json.load(open(cfg_path)))
model = MicroMixerV8FSC(cfg)
model.load_state_dict(load_file(ckpt_path))
model.eval()

# ... generate as above

CLI (loads from the local clone)

bash

uv run python infer_v8_fsc.py --ckpt-dir checkpoints/discord-v8fsc-1m-1024 --epoch 2

⚠️ Limitations

Limitation	Description
Sub-1M Parameters	Capacity-limited; ~2M bits of learnable knowledge (Allen-Zhu 2024)
Byte-Level Noise	256-vocab byte tokenizer makes PPL noisier than BPE baselines
Word-Level Incoherence	Generations show grammatical structure but garbled semantics
Long-Range (≥256 bytes)	State branch's 255-byte receptive field is the effective context horizon
3-Epoch Training Only	V8 keeps improving with more epochs; expect PPL ~3.0 with 5-10 epochs
Research Use Only	Designed for architecture experimentation, not production deployment

</div>

🧬 Lineage: Why V8 Exists

Version	Val PPL	Outcome	Why it failed / succeeded
V6 (multi-scale Toeplitz)	4.08 (after 91h)	Grammar-broken outputs; short repetitive prefixes at long context	Muon+WD orthogonalized (3, 4096) Toeplitz kernel to L2 ≈ 0.013 — mixer effectively collapsed
V7 (7-technique stack)	11.99 (after 3.8h)	Word salad (real words, broken grammar)	All 7 techniques competed for the same hidden capacity — no channel dedicated to syntax
V8 FSC-Mixer	3.32 (after 26 min)	Multi-speaker dialogue with grammar	Dedicate 50% of every layer to an explicit, long-range syntactic state pathway

The single architectural insight that made V8 work: V7 lacked a dedicated channel for syntactic state. V8's state branch (d_s=64 per layer, dilated causal conv, state-gated recombination) gives the model an explicit place to encode "what syntactic context am I in" — separate from "what byte comes next."

![GitHub](https://github.com/llaa33219/MicroMixer-3)

<sub>Part of the <a href="https://github.com/llaa33219/MicroMixer-3">MicroMixer-3</a> research project — V8 (FSC-Mixer) family</sub>

</div>