stanford-crfm/helm-scenarios

HELM Scenarios

This repository contains mirrors of datasets that are used as scenarios by `crfm-helm`.

Scenarios

TURL Column Type Annotation

The subfolder turl-column-type-annotation contains files for the table column type annotation task from the TURL paper. No modifications were made to these files.

The TURL dataset by Xiang Deng, Huan Sun, Alyssa Lees, You Wu, and Cong Yu is licensed under CC BY 4.0. The TURL dataset was modified from the TabEL dataset by Chandra Sekhar Bhagavatula, Thanapon Noraset, Doug Downey, which is licensed under CC BY 4.0.

Bird-SQL Mini-Dev

BIRD-SQL Mini-Dev is a light development dataset that is based on the full BIRD-SQL dataset for evaluating text-to-SQL models. This mirror contains only questions, and not databases.

It was downloaded from this URL following instructions from GitHub issue #1. and post-processed to remove duplicate questions identified in GitHub issue #17.

The BIRD-SQL Mini-Dev dataset by Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Rongyu Cao, Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li, Kevin C.C. Chang, Fei Huang, Reynold Cheng and Yongbin Li is licensed under CC BY 4.0.

BoolQ

BoolQ is a question answering dataset for yes/no questions containing 15942 examples. These questions are naturally occurring ---they are generated in unprompted and unconstrained settings. Each example is a triplet of (question, passage, answer), with the title of the page as optional additional context.

It was downloaded from the Google Cloud Storage URLs specified on the GitHub repository README.md, which are no longer functional.

The dataset by Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, Kristina Toutanova is licensed under CC BY-SA 3.0.