cpllab/syntaxgym

support loading all suites by default; represent multiple suites within dataset with suite_name feature

extract metric code for SyntaxGym metric space

add pytest comparison between gpt2 outputs from this code and from syntaxgym-core

fix surprisal indexing issues

ignore padding tokens in surprisal aggregation

improve punctuation rendering (no extraneous spaces)

type hint improvements

Download test suite materials dynamically rather than distributing JSONs

Implementation nice and clean, but need to verify results w/ reference syntaxgym-core still

Get prediction evaluation working

Get token aggregation working; just need to support predictions

restructure to accord with HF Hub expectations, I think

scaffold test script, partway there for actual prediction evaluation

gitignore

working example with half-implemented evaluation

stage working data loader, no metric/eval

stage test suites

initial commit