evalstate/tiny-memorization-synthetic
Tiny memorization synthetic datasets Uniformly-random token datasets replicating the methodology of "How much do language models memorize?" (arXiv:2505.24832). Each variant /<tag>/dataset_tokens.npy is an (N, 65) int64 array: a BOS token (id 2048) followed by 64 uniform iid tokens in [0, 2048). The exact entropy is H = N * 64 * log2(2048) = N * 704 bits (see each generation_params.json). Tokens are sampled with numpy.random.default_rng(seed); the seed is recorded per variant.
Tiny memorization synthetic datasets
Uniformly-random token datasets replicating the methodology of "How much do language models memorize?" (arXiv:2505.24832).
Each variant /<tag>/dataset_tokens.npy is an (N, 65) int64 array: a BOS token (id 2048) followed by 64 uniform iid tokens in [0, 2048). The exact entropy is H = N * 64 * log2(2048) = N * 704 bits (see each generation_params.json). Tokens are sampled with numpy.random.default_rng(seed); the seed is recorded per variant.