CoolFace
Datasetpublic

eltociear/pytorch-tasks-v1

pytorch-tasks-v1 Task dataset for a PyTorch RL / eval environment, in the shape used by the Prime Intellect Environments Hub. 45 PyTorch tasks across 5 categories. Each task gives the model one or more input tensors and an instruction; the answer is the tensor left in result, graded with torch.allclose against a reference. Grading is deterministic — no LLM judge, no external API, CPU only. Category Tasks Covers tensor_ops 16 reshape, transpose, axis reductions, clamp… See the full description on the dataset page: https://huggingface.co/datasets/eltociear/pytorch-tasks-v1.

sourceHugging Facemitupdated 2mo agoView on Hugging Face
0likes23downloads
Dataset Card

pytorch-tasks-v1

Task dataset for a PyTorch RL / eval environment, in the shape used by the Prime Intellect Environments Hub.

45 PyTorch tasks across 5 categories. Each task gives the model one or more input tensors and an instruction; the answer is the tensor left in result, graded with torch.allclose against a reference. Grading is deterministic — no LLM judge, no external API, CPU only.

CategoryTasksCovers
tensor_ops16reshape, transpose, axis reductions, clamp, boolean masking, sort/argsort, cat/stack, matmul, L2 row-normalise, unsqueeze, cumsum, dtype cast, elementwise max
nn_modules12Linear and Conv2d with explicit weights, relu/gelu/sigmoid, softmax and log_softmax, LayerNorm, max/avg pooling, flatten, Embedding
autograd6backward on scalar losses, torch.autograd.grad, gradient through sin, clamp-ed exp, detach semantics, second derivatives with create_graph
losses6MSE, cross-entropy (mean and per-sample), binary cross-entropy, L1, smooth L1
optim5one SGD step, SGD with momentum, one Adam step, three chained SGD steps, clip_grad_norm_

Fields

FieldDescription
task_idstable id, e.g. torch-017
categoryone of the five above
promptthe natural-language instruction shown to the model
input_dataJSON object mapping tensor name (x, m, b, i, s, l) to a serialised tensor
expected_outputthe serialised reference result

Tensors serialise as {"data": <nested list>, "dtype": <torch dtype without the prefix>, "shape": [...]}.

How it was built, and why you can trust the answer key

Tasks are defined as (deterministic input tensors, instruction, reference solution). The expected output is computed by executing the reference, never written by hand, so the key cannot drift from the instruction.

Every task is then independently verified:

  • —the reference runs and returns a real-valued tensor
  • —it is deterministic (executed twice, compared exactly)
  • —the result is finite — no NaN or inf in the answer key
  • —the result is non-empty and not identical to its input
  • —both the result and every input tensor survive the serialisation round-trip exactly, dtype and shape included

All 45 pass on torch 2.13.0+cpu. Builder and verifier: `build_tasks.py`.

Why the nn layers use explicit weights

Linear, Conv2d and Embedding here are filled with arange/linspace/constant weights rather than manual_seed plus default initialisation. Default initialisers and RNG stream details are implementation choices that have changed between PyTorch releases; arange cannot. A dataset whose labels silently change under pip install -U torch would be worse than no dataset.

A note on grading tolerance

Grading uses torch.allclose (rtol 1e-5, atol 1e-7) rather than exact equality: matmul, convolution and pooling dispatch to BLAS/oneDNN kernels whose last bits legitimately differ across builds and architectures. Shape is compared exactly, and an integer-valued reference requires an integer answer, so the tolerance never launders a wrong-shaped or wrong-typed result.