flexitok
Datasets
All datasets matching “flexitok”training_dataSpider-FLEXITOKENS-FP8
Spider-FLEXITOKENS FP8 Training
FP8 training pipeline for Spider-FLEXITOKENS on NVIDIA Blackwell GPUs (sm_120) using torchao Float8Linear and optional TileKernels fused MoE routing.
Architecture
Spider is a Recurrent-Depth Transformer (RDT) with:
1B parameters (996M), hidden_size=2048
Byte-level vocab: 272 tokens (256 UTF-8 bytes + 16 specials: BOS=257, EOS=258, PAD=256)
6 recurrent layers with MoE (32 experts, top-2 routing) + MLA attention
2 prelude + 2 coda dense… See the full description on the dataset page: https://huggingface.co/datasets/CLIWorks/Spider-FLEXITOKENS-FP8.mod-arithmetic
Modular Arithmetic Dataset
Synthetic dataset of modular-arithmetic problems of the form a mod b,
paired with the result and a hypothesis about the most suitable tokenizer.
Tokenizer hypothesis
For a mod b where b = 2^k × 5^j (no other prime factors), only the
rightmost max(k, j) digits of a determine the answer, because
10^max(k,j) ≡ 0 (mod b). A tokenizer that groups digits right-to-left
in chunks of that size exposes the relevant information as a single token.
For all… See the full description on the dataset page: https://huggingface.co/datasets/flexitok/mod-arithmetic.multilingual-addition
Multilingual Addition Dataset
Synthetic dataset of addition problems of the form a+b=answer, where a
and b are written-form representations of integers in 21 languages, plus
a 22nd split using raw digit strings.
Task format
Each sample contains:
field
type
description
a_str
str
written-form (or digit) representation of a
a_digit
int
integer value of a
b_str
str
written-form (or digit) representation of b
b_digit
int
integer value of b
answer
str… See the full description on the dataset page: https://huggingface.co/datasets/flexitok/multilingual-addition.