kylelovesllms/grammar-number-agreement-frames-rb
Grammar V1 + Number Agreement — RB Populated Frames The 212 depth-1 syntactic frames of grammar-number-agreement-frames populated with randomly sampled lexicalizations, 18 base lexicalizations per (frame, task) (seeded RNG, seed=0; grammar source: experiments/01_grammar_v1/src/cfg/grammar_number_agreement.py). Grammar V1 with subject-verb number agreement: nouns and verbs are singular or plural (N_sg / N_pl, V_trans_sg / V_trans_pl, ...), the matrix verb agrees with the matrix… See the full description on the dataset page: https://huggingface.co/datasets/kylelovesllms/grammar-number-agreement-frames-rb.
Grammar V1 + Number Agreement — RB Populated Frames
The 212 depth-1 syntactic frames of grammar-number-agreement-frames populated with randomly sampled lexicalizations, 18 base lexicalizations per (frame, task) (seeded RNG, seed=0; grammar source: experiments/01_grammar_v1/src/cfg/grammar_number_agreement.py).
Grammar V1 with subject-verb number agreement: nouns and verbs are singular or plural (N_sg / N_pl, V_trans_sg / V_trans_pl, ...), the matrix verb agrees with the matrix subject, a subject-gap RC verb with the head noun and an object-gap RC verb with the embedded subject, while objects take either number. Matrix Number is the number of the matrix subject, RC Number that of the noun the RC verb agrees with (None without an RC), and n_attractors counts nouns of the other number between a verb and the noun it agrees with (only center-embedded sentences have any). Every sentence is grammatical under agreement by construction: each frame slot fixes the number of the word drawn for it.
The multiplier was solved so that the ce-rb train token budget is closest to the reference budget of Grammar V1 ce-rb, multiplier 100 (78,080 tokens).
Partition (README.md / DataGenFlow.png): ambiguous (in-distribution) frames contribute copy and translate examples split 90/10 into train/validation (per frame and task, so every ambiguous frame appears in both); unambiguous (out-of-distribution) frames contribute copy examples to train and translate examples to test. Copy and translate rows of a frame never share a sentence, so no test translate input appears as a train copy input.
Copy-task variant rb: copy simple (no RC) and right branching (object-attached RC) structures only
Train copy rows are kept only for frames whose relative clause attaches at one of these sites: None, Object (None = no relative clause). The validation and test splits are identical across the ce-rb, ce and rb datasets (same base lexicalizations), so only the training data differs; validation therefore still contains copy rows of structures whose train copy rows were dropped. To keep the train token budget equal to ce-rb's (non-pad tokens the model is trained on: <bos> src <task> <sep> tgt <eos> = 2 * n_tokens + 4 per row), every remaining train (frame, task) group is boosted by the same factor with additional distinct lexicalizations (rounded per group):
- train: 4688 rows = 1800 translate (ambiguous) + 1800 copy (ambiguous) + 1088 copy (unambiguous)
- validation: 240 rows = 120 translate + 120 copy (ambiguous only)
- test: 2736 rows (translate, unambiguous)
Train rows per structure
HF Sentence is the task target: the HI sentence itself for copy rows, the hierarchical (Head Final) translation for translate rows. Frames use indexed bracketed notation where [N_3] means the terminal filled by HI token 3.