cpllab/syntaxgym
1
1---2title: SyntaxGym3emoji: 🏋️4colorFrom: pink5colorTo: yellow6sdk: gradio7sdk_version: 3.0.138app_file: app.py9pinned: false10tags:11- evaluate12- metric13description: >-14 Evaluates Huggingface models on SyntaxGym datasets (targeted syntactic evaluations).15---16 17# Metric Card for SyntaxGym18 19## Metric Description20 21[SyntaxGym][syntaxgym] is a framework for targeted syntactic evaluation of language models. This metric can be combined with the [SyntaxGym dataset][syntaxgym-dataset] to evaluate the syntactic capacities of any Huggingface causal language model.22 23## How to Use24 25The metric takes a SyntaxGym test suite as input, as well as the name of the model that should be evaluated:26 27```python28import datasets29import evaluate30import numpy as np31 32dataset = datasets.load_dataset("cpllab/syntaxgym", "subordination_src-src")33metric = evaluate.load("cpllab/syntaxgym")34result = metric.compute(dataset=dataset["test"], model_id="gpt2")35 36# Compute suite accuracy. Mean success over items, where "success" is the conjunction37# of all boolean prediction results.38suite_accuracy = result["subordination_src-src"].accuracy39```40 41### Run the entire SyntaxGym dataset42 43You can load and evaluate all suites at once by omitting the dataset configuration name (second argument):44 45```python46import datasets47import evaluate48import numpy as np49 50dataset = datasets.load_dataset("cpllab/syntaxgym")51metric = evaluate.load("cpllab/syntaxgym")52result = metric.compute(dataset=dataset["test"], model_id="gpt2")53 54# Compute suite accuracy. Mean success over items, where "success" is the conjunction55# of all boolean prediction results.56suite_accuracies = {suite_name: suite_results.accuracy57 for suite_name, suite_results in result.items()}58overall_accuracy = np.mean(list(suite_accuracies.values()))59```60 61```python62>>> suite_accuracies63{'center_embed': 0.9285714285714286,64 'center_embed_mod': 0.8571428571428571,65 'cleft': 1.0,66 'cleft_modifier': 0.925,67 'fgd_hierarchy': 0.0,68 'fgd_object': 0.9583333333333334,69 'fgd_pp': 0.875,70 'fgd_subject': 0.5,71 'mvrr': 0.7857142857142857,72 'mvrr_mod': 0.75,73 'npi_orc_any': 0.9736842105263158,74 'npi_orc_ever': 1.0,75 'npi_src_any': 0.5789473684210527,76 'npi_src_ever': 0.9210526315789473,77 'npz_ambig': 0.9166666666666666,78 'npz_ambig_mod': 0.875,79 'npz_obj': 1.0,80 'npz_obj_mod': 1.0,81 'number_orc': 0.631578947368421,82 'number_prep': 0.7894736842105263,83 'number_src': 0.7894736842105263,84 'reflexive_orc_fem': 0.47368421052631576,85 'reflexive_orc_masc': 0.8421052631578947,86 'reflexive_prep_fem': 0.21052631578947367,87 'reflexive_prep_masc': 0.7894736842105263,88 'reflexive_src_fem': 0.15789473684210525,89 'reflexive_src_masc': 0.631578947368421,90 'subordination': 1.0,91 'subordination_orc-orc': 1.0,92 'subordination_pp-pp': 1.0,93 'subordination_src-src': 1.0}94>>> overall_accuracy950.779383943730293696```97 98### Inputs99 100- **dataset** (`Dataset`): SyntaxGym test suite, represented as a Huggingface dataset. See the [dataset reference][syntaxgym-dataset].101- **model_id** (str): Model used to calculate probabilities of each word. (This is only well defined for causal language models. This includes models such as `gpt2`, causal variations of BERT, causal versions of T5, and more. The full list can be found in the [`AutoModelForCausalLM` documentation][causal].)102- **batch_size** (int): Maximum batch size for computations103- **add_start_token** (bool): whether to add the start token to each sentence. Defaults to `True`.104- **device** (str): device to run on, defaults to `cuda` when available105 106### Output Values107 108The metric returns a dict of `SyntaxGymMetricSuiteResult` objects, mapping test suite names to test suite performance. Each inner object has three properties:109 110- **accuracy** (`float`): Model accuracy on this suite. This is the accuracy of the conjunction of all boolean predictions per item in the suite.111- **prediction_results** (`List[List[bool]]`): For each item in the test suite, a list of booleans indicating whether each corresponding prediction came out `True`. Typically these are combined to yield an accuracy score (but you can simply use the `accuracy` property).112- **region_totals** (`List[Dict[Tuple[str, int], float]`): For each item, a mapping from individual region (keys `(<condition_name>, <region_number>)`) to the float-valued total surprisal for tokens in this region. This is useful for visualization, or if you'd like to use the aggregate surprisal data for other tasks (e.g. reading time prediction or neural activity prediction).113 114```python115>>> print(result["subordination_src-src"]["prediction_results"][0])116[True]117>>> print(result["subordination_src-src"]["region_totals"][0])118{('sub_no-matrix', 1): 14.905603408813477,119 ('sub_no-matrix', 2): 39.063140869140625,120 ('sub_no-matrix', 3): 26.862628936767578,121 ('sub_no-matrix', 4): 50.56561279296875,122 ('sub_no-matrix', 5): 7.470069408416748,123 ('no-sub_no-matrix', 1): 13.15120792388916,124 ('no-sub_no-matrix', 2): 38.50318908691406,125 ('no-sub_no-matrix', 3): 27.623855590820312,126 ('no-sub_no-matrix', 4): 48.8316535949707,127 ('no-sub_no-matrix', 5): 1.8095952272415161,128 ('sub_matrix', 1): 14.905603408813477,129 ('sub_matrix', 2): 39.063140869140625,130 ('sub_matrix', 3): 26.862628936767578,131 ('sub_matrix', 4): 50.56561279296875,132 ('sub_matrix', 5): 26.532146453857422,133 ('no-sub_matrix', 1): 13.15120792388916,134 ('no-sub_matrix', 2): 38.50318908691406,135 ('no-sub_matrix', 3): 27.623855590820312,136 ('no-sub_matrix', 4): 48.8316535949707,137 ('no-sub_matrix', 5): 38.085227966308594}138 ```139 140 ## Limitations and Bias141 142 TODO143 144 ## Citation145 146 If you use this metric in your research, please cite:147 148```bibtex149@inproceedings{gauthier-etal-2020-syntaxgym,150 title = "{S}yntax{G}ym: An Online Platform for Targeted Evaluation of Language Models",151 author = "Gauthier, Jon and Hu, Jennifer and Wilcox, Ethan and Qian, Peng and Levy, Roger",152 booktitle = "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations",153 month = jul,154 year = "2020",155 address = "Online",156 publisher = "Association for Computational Linguistics",157 url = "https://www.aclweb.org/anthology/2020.acl-demos.10",158 pages = "70--76",159 abstract = "Targeted syntactic evaluations have yielded insights into the generalizations learned by neural network language models. However, this line of research requires an uncommon confluence of skills: both the theoretical knowledge needed to design controlled psycholinguistic experiments, and the technical proficiency needed to train and deploy large-scale language models. We present SyntaxGym, an online platform designed to make targeted evaluations accessible to both experts in NLP and linguistics, reproducible across computing environments, and standardized following the norms of psycholinguistic experimental design. This paper releases two tools of independent value for the computational linguistics community: 1. A website, syntaxgym.org, which centralizes the process of targeted syntactic evaluation and provides easy tools for analysis and visualization; 2. Two command-line tools, {`}syntaxgym{`} and {`}lm-zoo{`}, which allow any user to reproduce targeted syntactic evaluations and general language model inference on their own machine.",160}161```162 163 If you use the [SyntaxGym dataset][syntaxgym-dataset] in your research, please cite:164 165 ```bibtex166@inproceedings{Hu:et-al:2020,167 author = {Hu, Jennifer and Gauthier, Jon and Qian, Peng and Wilcox, Ethan and Levy, Roger},168 title = {A systematic assessment of syntactic generalization in neural language models},169 booktitle = {Proceedings of the Association of Computational Linguistics},170 year = {2020}171}172 ```173 174[syntaxgym]: https://syntaxgym.org175[syntaxgym-dataset]: https://huggingface.co/datasets/cpllab/syntaxgym176[causal]: https://huggingface.co/docs/transformers/master/en/model_doc/auto#transformers.AutoModelForCausalLM