Datasetpublic

plawanrath/Linalg-Spec-30

Linalg-Spec-30 Accepted to NeurIPS 2026 (Evaluations & Datasets Track)Paper (arXiv) · Code (GitHub) · All six datasets (collection) Hand-authored NL→MLIR pairs for linalg named ops under memref semantics (n=30). This dataset is one of six NL→MLIR benchmarks released with the NeurIPS 2026 Evaluations & Datasets Track paper Cross-Dialect Generalization Without Retraining: Benchmarks and Evaluation of Schema-Derived Constrained Decoding for MLIR (arXiv:2607.18254). The full suite… See the full description on the dataset page: https://huggingface.co/datasets/plawanrath/Linalg-Spec-30.

sourceHugging Faceapache-2.0updated 18h agoView on Hugging Face

0likes18downloads

Dataset Card

Linalg-Spec-30

Accepted to NeurIPS 2026 (Evaluations & Datasets Track) Paper (arXiv) · Code (GitHub) · All six datasets (collection)

Hand-authored NL→MLIR pairs for linalg named ops under memref semantics (n=30).

This dataset is one of six NL→MLIR benchmarks released with the NeurIPS 2026 Evaluations & Datasets Track paper Cross-Dialect Generalization Without Retraining: Benchmarks and Evaluation of Schema-Derived Constrained Decoding for MLIR (arXiv:2607.18254). The full suite — `MLIR-Spec-150`, `Linalg-Spec-30`, `StableHLO-Spec-30`, `StableHLO-Held-Out-200`, `StableHLO-OutOfGrammar-25`, `MLIR-Functional-Reference-30` — totals 465 instances across three MLIR dialects.

Composition

—Instances: 30
—Format: one JSON record per line in data/test.jsonl
—Schema: fields = dialect, difficulty, id, mlir, nl, notes
—Verifier: mlir-opt --verify-diagnostics against pinned LLVM 19.1.7
—License: Apache-2.0 (SPDX: Apache-2.0). No third-party IP restrictions.

Loading

python

from datasets import load_dataset
ds = load_dataset("plawanrath/Linalg-Spec-30", split="test")
print(ds[0])

Each record is a self-contained natural-language→MLIR pair; verify-valid pass-rate under the dialect's verifier is the primary evaluation metric.

Source format

The JSONL file at data/test.jsonl is the canonical HuggingFace interface. It is exported 1-to-1 from the source records (examples/*.json) in `eval/benchmarks/linalg_spec_30/` of the code repository.

Datasheet

Key points (full Gebru-style datasheet: `docs/datasheets/datasheet.md`):

—All reference MLIR programs are verifier-clean at the time of release.
—Hand-authored (no crowdsourcing, no LLM-authored references).
—Test-only — fine-tuning on these benchmarks contaminates future evaluation and is explicitly out of scope.

Croissant metadata

`croissant.json` in this repository is the dataset's MLCommons Croissant 1.1 metadata, including the Responsible AI (rai:) and provenance (prov:) fields required by the NeurIPS 2026 Evaluations & Datasets Track. (Hugging Face's auto-generated Croissant endpoint covers the core fields only.)

Code

Generation, verification and evaluation code: github.com/plawanrath/slm-to-mlir-constrained-emitter.

Citation

bibtex

@inproceedings{rath2026crossdialect,
  title     = {Cross-Dialect Generalization Without Retraining: Benchmarks and Evaluation of Schema-Derived Constrained Decoding for {MLIR}},
  author    = {Rath, Plawan Kumar},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS), Evaluations and Datasets Track},
  year      = {2026},
  eprint    = {2607.18254},
  archivePrefix = {arXiv}
}