eltociear/scaling-tasks-v1
scaling-tasks-v1 35 quantitative scaling tasks for the scaling-env RL environment, on the topics of Google DeepMind's free book How To Scale Your Model: rooflines and arithmetic intensity, transformer FLOPs and parameter counts, KV-cache and optimizer-state memory, collective communication volume, and decode throughput. field meaning task_id sc-000 … sc-034 category roofline / flops / memory / sharding / inference prompt the question, every parameter it needs, and… See the full description on the dataset page: https://huggingface.co/datasets/eltociear/scaling-tasks-v1.
scaling-tasks-v1
35 quantitative scaling tasks for the `scaling-env` RL environment, on the topics of Google DeepMind's free book [How To Scale Your Model](https://jax-ml.github.io/scaling-book/): rooflines and arithmetic intensity, transformer FLOPs and parameter counts, KV-cache and optimizer-state memory, collective communication volume, and decode throughput.
Categories: roofline 7, flops 8, memory 6, sharding 7, inference 7. No dependencies — every task is answered with math.
Original problems, and every hardware number is stated
Nothing here is copied from the book. And no task requires knowing a real chip's peak FLOPs or HBM bandwidth: the parameters are given in the task.
That second point is the main design decision. An environment that required recalling a spec sheet would measure memorisation of numbers that change with each accelerator generation — and its answer keys would silently become wrong as hardware moves while still looking authoritative. Stating the parameters makes every problem self-contained, permanently valid, and a test of the reasoning rather than the recall.
What the answers look like
The arithmetic that decides real scaling choices, where the result is often counterintuitive: the same matmul is compute-bound at batch 1024 (intensity 683) and memory-bound at batch 1 (intensity 0.9995); attention is only 16% of a layer's FLOPs at S=8192; Adam state is 6× the weights, so a 70B model needs 11 chips of 96 GiB just to be held; a 64-way gradient all-reduce takes 6.1× longer than the step it overlaps; batch-1 decode of a 70B model is 11.4 tokens/s, fixed by bandwidth alone.
Grading
Floats at a 1e-9 relative tolerance; strings, booleans and None exactly. The tolerance exists for one thing — the order of multiplications and divisions, since 6ND/(chips·peak·mfu) and ((6N)/chips)·(D/peak)/mfu differ in their last bits. It is orders of magnitude too tight to hide a wrong formula: a missing (N-1)/N, a factor of 2 and an error of one part in a million all fail, and each of those is an asserted unit check.
Verify with python environments/scaling_env/build_tasks.py --verify (35/35). Source: <https://github.com/eltociear/my-molt-agent/tree/main/environments/scaling_env>