hanspeterlyngsoeraaschoujensen/phi-tiny-moe-fineweb-edu-topk-router-traces
Phi-tiny-MoE FineWeb-Edu top-k router traces Native top-2 routing traces from microsoft/Phi-tiny-MoE-instruct over a deterministic 10,000,000-token sample of HuggingFaceFW/fineweb-edu (sample-10BT). Documents are deterministically shuffled, assigned to document-isolated train/eval streams, tokenized without prompt formatting, separated by EOS tokens, and packed into sequences of at most 4,096 tokens. Run:… See the full description on the dataset page: https://huggingface.co/datasets/hanspeterlyngsoeraaschoujensen/phi-tiny-moe-fineweb-edu-topk-router-traces.
Phi-tiny-MoE FineWeb-Edu top-k router traces
Native top-2 routing traces from microsoft/Phi-tiny-MoE-instruct over a deterministic 10,000,000-token sample of HuggingFaceFW/fineweb-edu (sample-10BT). Documents are deterministically shuffled, assigned to document-isolated train/eval streams, tokenized without prompt formatting, separated by EOS tokens, and packed into sequences of at most 4,096 tokens.
- Run:
phi-tiny-fineweb-edu-topk_and_next_layer_logits-10000000-2026-07-23-00-29-45 - Total tokens: 10,000,000
- Model revision:
2fe50e88d0e2a5a132563815686ea0dcc8e252b5 - Dataset revision:
87f09149ef4734204d70ed1d046ddc9ca3f2b8f9 - Attention backend:
flash_attention_2 - Stored routing shape:
[1, sequence, 32, 2]
Each .pt shard contains input_ids, topk_indices, topk_probs, next_layer_router_logits, trajectory_id, instance_id, source_token_count, resolved, token_start, and token_end. Expert IDs are INT16 and routing weights are BF16. topk_probs stores Phi's native SparseMixer weights, whose sum is not required to equal one. next_layer_router_logits[:, :, i] stores the full BF16 expert logits produced by layer i + 1's router from layer i's router input, so its layer axis has length 31. No hidden states are stored.