hobbylm
Datasets
All datasets matching “hobbylm”hobbylm-routing-dynamics
HobbyLM-1B Routing Dynamics
Router activation data captured from 20 checkpoints spanning a 100B-token sparse
Mixture-of-Experts pretraining run, with fully documented data provenance for
every input token. Base model: harims95/hobbylm-1b-hf.
What this is
For each of 20 training checkpoints, the model was run over an identical frozen
probe set and its router's per-token, per-layer decisions were recorded: which
experts were selected, the router's raw scores before… See the full description on the dataset page: https://huggingface.co/datasets/harims95/hobbylm-routing-dynamics.hobbylm-mix100b-gpt2hobbylm-longctx-2p5b
HobbyLM-1B Long-Context Extension Corpus
2.5B GPT-2 tokens staged for extending harims95/hobbylm-1b-hf's
context window from 1024 tokens to 4096, then 8192.
Format
Each <source>_NNNNNN.bin file is the same flat-binary format used throughout
HobbyLM training: a 256 x int32 header (magic=20240520, version=1,
num_tokens), followed by the token stream as uint16 GPT-2 token ids.
The flat .bin files do not encode document boundaries on their own. Each
source also ships… See the full description on the dataset page: https://huggingface.co/datasets/harims95/hobbylm-longctx-2p5b.