Datasetpublic

RASMUS/commitmoe-qwen35-fp8-layers-v2

CommitMoE — Qwen3.5-35B-A3B-FP8 expert-routing traces, columnar by layer Per-token MoE routing decisions from Qwen/Qwen3.5-35B-A3B-FP8, laid out one directory per layer so a predictor for a single layer reads only what it needs instead of scanning interleaved shards. The model has 40 MoE layers, 256 experts each, top-8 routing, hidden size 2048. Every row is one (prompt, decode token, layer) triple. What this is for Predicting which experts a layer will route to a… See the full description on the dataset page: https://huggingface.co/datasets/RASMUS/commitmoe-qwen35-fp8-layers-v2.

sourceHugging Faceapache-2.0updated 1mo agoView on Hugging Face

0likes329downloads

Dataset Card

CommitMoE — Qwen3.5-35B-A3B-FP8 expert-routing traces, columnar by layer

Per-token MoE routing decisions from Qwen/Qwen3.5-35B-A3B-FP8, laid out one directory per layer so a predictor for a single layer reads only what it needs instead of scanning interleaved shards.

The model has 40 MoE layers, 256 experts each, top-8 routing, hidden size

1.Every row is one (prompt, decode token, layer) triple.

What this is for

Predicting which experts a layer will route to a couple of layers ahead, so their weights can be prefetched over PCIe into a small GPU-resident cache before the layer runs. A prediction that misses costs a stall; the metric that matters is recall at the cache size C, and under a W = C admission policy the stall count per token per layer is exactly 8 · (1 − recall@C).

Layout

layers_v1/<chunk>/
  prompts.jsonl                       # id + prompt_idx, one row per prompt
  labels/part_*.safetensors           # top_indices[8], prompt_idx, tok, layer, gen_token_id
  hidden/layer=NN/part_*.safetensors  # h[2048] (pre-MoE hidden), prompt_idx, tok
  logits/layer=NN/part_*.safetensors  # logits[256] (router), prompt_idx, tok

labels covers all 40 layers in one table — it is small (~2.6 GB per 10k prompts) and window features read every layer of it. hidden is the expensive part at about 10.0 GB per layer per 10k prompts; logits is 1.27 GB.

Chunks and prompt ids

prompt_idx is globally unique across chunks, so they can be pooled without collision:

chunk	prompts	`prompt_idx` range
`chunk_10k`	10,000	0 – 9,999
`chunk00`	4,000	10,000 – 13,999
`chunk01`	6,000	14,000 – 19,999

chunk00 lives in the companion repo `commitmoe-qwen35-fp8-layers`. Each chunk's prompts.jsonl starts at its own base, and readers derive that base from the file's first row rather than assuming zero.

A note on the ids, because it is easy to get wrong. The raw shards store prompt_idx as a shard-local slot (0–31, an index into that shard's own 32-entry prompt list), not a global id. Writing it through unchanged collapses every shard's prompts onto the same 32 ids — silently, since the output still looks well-formed. The files here are built by resolving each shard's prompt-id strings through the chunk's index, and every upload is checked by comparing the written prompt_idx set against the set resolved from the shards.

How the shards were laid out upstream

Each raw shard is a decode-step slice of a 32-prompt batch — 32 prompts × ~26 decode steps × 40 layers ≈ 32,768 rows — so roughly ten consecutive raw shards share the same 32 prompts. That is invisible in this columnar form but matters if you rebuild from the raw traces.

Splits

Not stored. The split is derived from the prompt-id string, so it stays consistent across chunks and rebuilds.