Datasetpublic

NagaYu/litmus-kernels

Litmus Kernel Verification Corpus Correct and deliberately-broken Triton kernels, each broken one shipped with the input that exposes it. The corpus exists to measure one thing: how much of what a fixed-shape torch.rand() allclose test calls "correct" actually is. On this corpus the answer is that 88% of the planted bugs pass that test. Columns column meaning name kernel identifier family elementwise / reduction / softmax / layernorm / matmul /… See the full description on the dataset page: https://huggingface.co/datasets/NagaYu/litmus-kernels.

sourceHugging Faceapache-2.0updated 1mo agoView on Hugging Face

0likes32downloads

Dataset Card

Litmus Kernel Verification Corpus

Correct and deliberately-broken Triton kernels, each broken one shipped with the input that exposes it.

The corpus exists to measure one thing: how much of what a fixed-shape torch.rand() allclose test calls "correct" actually is. On this corpus the answer is that 88% of the planted bugs pass that test.

Columns

column	meaning
`name`	kernel identifier
`family`	elementwise / reduction / softmax / layernorm / matmul / attention
`bug_class`	`none` for correct kernels, otherwise one of six planted defect families
`triton_source`	the kernel, annotated with its buffer contract
`correct`	ground-truth label
`numerically_observable`	`false` when the defect has no numerical signature at any shape
`why_standard_misses`	why the fixed-shape `allclose` test lets this bug through
`witness_inputs`	JSON arrays: the exposing input (buggy kernels only)
`witness_dims`	the shape the witness uses
`witness_max_abs_error`	deviation from the float64 oracle on the witness
`standard_test_detects`	did `allclose(rtol=1e-2)` on the fixed shape catch it?
`litmus_verdict`	`accept` / `reject` / `unknown` at build time

Bug families

—`dtype_promotion` — 2 kernels
—`mask_leak` — 4 kernels
—`no_max_subtract` — 2 kernels
—`race` — 3 kernels
—`reduction_init` — 3 kernels
—`stride` — 2 kernels

Caveat

The corpus is deliberately bug-heavy (16 buggy, 9 correct), so the absolute "fraction correct" is a property of this corpus, not of kernels in general. The finding is the gap between the standard test's pass rate and the truth.

Reproduce

bash

git clone https://github.com/NagaYu/litmus && cd litmus
pip install -e ".[dev]"
python benchmarks/run.py        # regenerates the numbers above

Built by Litmus -- error bounds, symbolic GPU safety checks and targeted falsification for Triton kernels. Interactive demo: Space.