fabiogr/social_i_qa_pt

SocialIQa dataset v1.4 (PT)

This is translation to the portuguese language of the dataset allenai/socialiqa. Translations were done using three independent models:

• —Helsinki-NLP/opus-mt-tc-big-en-pt
• —unicamp-dl/translation-en-pt-t5
• —facebook/nllb-200-distilled-1.3B

Translations were evaluated using the evaluation metric GEMBA - GPT Estimation Metric Based Assessment (from the article Large Language Models Are State-of-the-Art Evaluators of Translation Quality) using GPT3.5-turbo. The highest ranked translation set was selected based on this metric and used in the dataset.

To use the metric, the prompt was modified to not use human reference in the evaluation and to evaluate the translations of the three models of this work simultaneously.

The description below is from the source dataset:

Dataset Summary:

Social IQa (Social Interaction QA) is a question-answering benchmark designed to test social commonsense intelligence. Unlike many prior benchmarks that focus on physical or taxonomic knowledge, Social IQa emphasizes reasoning about people's actions and their social implications. For example, given an action like "Jesse saw a concert" and a question like "Why did Jesse do this?", humans can easily infer that Jesse wanted "to see their favorite performer" or "to enjoy the music, " rather than "to see what's happening inside" or "to see if it works." The dataset contains over 37,000 QA pairs, evaluating models' abilities to reason about social implications in everyday events and situations.

Data Fields:

• —context: A string feature representing the context.
• —question: A string feature representing the question.
• —answerA, answerB, answerC: String features representing answer options.
• —label: A string feature indicating the correct answer option.

Lines in each file are json encoded QA instances, using unicode characters (not escaped chars).

{"context":"Tracy não foi para casa naquela noite e resistiu aos ataques de Riley.",
"question":"O que Tracy precisa fazer antes disso?","answerA":"Faça um novo plano",
"answerB":"Vá para casa e veja Riley","answerC":"Encontre um lugar para ir","correct":"C"}

Languages:

The dataset is available in English and Portuguese (this dataset)

Source and Curators:

The dataset was curated by the Allen Institute for AI (AI2). Contributors include @bhavitvyamalik, @thomwolf, @patrickvonplaten, and @lewtun. You can find more information about the Social IQa dataset on the official AI2 page or explore it directly on Hugging Face Datasets