EngineeringSoftware/PLSemanticsBench
The 43rd International Conference on Machine Learning (ICML 2026), Seoul, South Korea LLMs Lean on Priors, Not Programming Language Semantics by Aditya Thimmaiah1, Jiyang Zhang1, Jayanth Srinivasa2, Junyi Jessy Li1, Milos Gligoric1 1The University of Texas at Austin 2Cisco Research TLDR: Frontier LLMs execute programs with up to 90–100% accuracy when symbols retain their usual… See the full description on the dataset page: https://huggingface.co/datasets/EngineeringSoftware/PLSemanticsBench.
Cleaning up auxiliary task dataset splits
Delete rule2nl
Delete nl2rule
Updating dataset
Delete rule2nl
Delete predtrace
Delete predstate
Delete predrule
Delete nl2rule
Update README.md
Update README.md
Update README.md
Adding note to README
Update dataset layout
Uploading new version of dataset
Delete predtrace
Delete predstate
Delete predrule
Delete notation-understanding
Update README.md
Update README.md
Update README.md
Adding new dataset version
updating dataset
Upload dataset
Upload dataset
Upload dataset
Upload dataset
Upload dataset
Upload dataset
Upload dataset
Upload dataset
Upload dataset
Upload dataset
Upload dataset
Upload dataset
Update README.md
Fixing typo
Adding back dataset split info and updating license
Update README.md
Upload dataset
Upload dataset
Upload dataset
Upload dataset
Upload dataset
Upload dataset
Upload dataset
Upload dataset
Upload dataset
Upload dataset