jiyounglee0523/TransEnV_hellaswag

Version 2 (2026-08): Dialect configs regenerated with a stronger pipeline

The 18 dialect configs (AAVE, AppE, AuE, AuEV, BahE, EAngE, IrE, Manx, NZE, NEng, NfE, OzE, SEAmE, SEEng, SWEng, ScE, TdCE, WaE) were regenerated with an upgraded Trans-EnV pipeline. The ESL configs (A/B_) are unchanged (v1). Previous versions of all files remain available via git revisions of this repo.

What changed

• —Transformation model: google/gemma-2-27b-it → google/gemma-4-31B-it, with a naturalness constraint added to the actionable prompt (discourse insertions such as focus markers are applied at most once per sentence; systematic morphological changes are still applied consistently).
• —Higher feature coverage: the stronger model applies eWAVE dialect rules far more reliably (e.g., GSM8K dialect transformation rate rose from ~50% to ~85% of items on average) while preserving item semantics.
• —Independent semantic verification (new): every transformed item was checked by an independent judge model (Qwen3-32B, temperature 0) using the paper's semantic-equivalence criterion — 375,112 pairs in total. Items judged to alter meaning (2.4%) were rolled back to the original SAE text, so the released dialect configs contain no verified meaning-altering transformation.
• —Task-integrity fixes: WinoGrande items now always retain the _ blank (v1 had lost it in many rows); stray wrapping quotation marks introduced by the transformation model were removed; HellaSwag ctx_a/ctx no longer duplicate ctx_b.
• —Deterministic integrity checks (numbers, proper nouns, question structure) pass on all configs; schema and row order are identical to v1.

Per-config rollback statistics are available in the repository discussion or on request. Transformation code: https://github.com/jiyounglee-0523/TransEnV

This dataset is introduced in the paper Trans-EnV: A Framework for Evaluating the Linguistic Robustness of LLMs Against English Varieties. It comprises transformed versions of existing datasets, designed to evaluate the performance of Large Language Models (LLMs) across a diverse range of English varieties.

For more comprehensive details, please refer to the:

• —Full Dataset Collection
• —GitHub Repository