CoolFace
Datasetpublic

collectioncard/TinyTownQA

Tiny Town QA We present Tiny Town QA, a multiple choice question and answer dataset designed to provide insight into how LLMs understand tile-based game worlds. Tiny town maps and question sets were originally generated with Our Tiny Town Dataset Generator. Question data then underwent a multiple pass validation check to ensure quality. As of March 2025, models are capable of the following results when tested via our QA Analysis tool: Model Name World Representation… See the full description on the dataset page: https://huggingface.co/datasets/collectioncard/TinyTownQA.

sourceHugging Facecc-by-4.0updated 1y agoView on Hugging Face
1likes5downloads
Dataset Card

Tiny Town QA

We present Tiny Town QA, a multiple choice question and answer dataset designed to provide insight into how LLMs understand tile-based game worlds.

Tiny town maps and question sets were originally generated with Our Tiny Town Dataset Generator. Question data then underwent a multiple pass validation check to ensure quality.

As of March 2025, models are capable of the following results when tested via our QA Analysis tool:

Model NameWorld RepresentationAccuracyRun File
Qwen 2.5 1.5BWorld Facts332/642 correct: 51.7%Qwen2.5-1.5B WF
Deepseek R1 Distill 1.5BWorld Facts376/642 correct: 58.6%deepseek-1.5B Distill WF
Qwen 2.5 7BWorld Facts504/642 correct: 78.5%Qwen2.5-7B WF
Gemini-flash 2.0 (Images + Building Labels ONLY)Images + POI412/642 correct : 64.17%Gemini-flash2.0 IMG+POI
Gemini-flash 2.0 (base map array)TileData Array + POI430/642 correct: 67.0%Gemini-flash2.0 TD+POI
Gemini-flash 2.0 (images + Base map array)TileData Array + Images432/642 correct : 67.6%Gemini-flash2.0 TD+IMG
Gemini-flash 2.0 (Text Only)World Facts604/642 correct: 94%Gemini-flash2.0 WF
GPT 4o mini (Images + building Labels Only)Images + POI330/462 correct: 51.4%GPT-4o-mini IMG+POI
GPT 4o mini (Images + Text)World Facts + Images583/642 correct: 90.8%GPT-4o-mini WF+IMG
GPT 4o mini (Text Only)World Facts585/642 correct : 91%GPT-4o-mini WF
GPT 4o (Text Only)World Facts606/642 correct: 94.4%GPT-4o WF
GPT 4o (Images + Text)World Facts + Images624/642 correct: 97.2%GPT-4o WF+IMG
Deepseek R1World Facts614/642 correct : 95.6%Deepseek-R1 WF