Laborator/austrian-german-benchmark

AT-Bench: Austrian German Benchmark 300 multiple-choice questions testing whether an LLM actually understands Austrian German — not just German. The problem Every German benchmark treats German as one language. But ask GPT what "Obers" means and half the time it guesses wrong. Ask it about the Bezirksgericht and it describes the German court system. Austrian German is an official language variety spoken by 9 million people, and models consistently get it wrong.… See the full description on the dataset page: https://huggingface.co/datasets/Laborator/austrian-german-benchmark.