mm-eval/Mantis-Eval

1{2  "name": "Mantis-Eval",3  "release_date": "2024-05-25",4  "subsets": {5    "main": {6      "language": [7        "en"8      ],9      "modalities": [10        "multi_image_interleave"11      ],12      "task_type": "multiple_choice_qa",13      "score_pipeline": [14        "exact-match",15        "rule-match"16      ],17      "score_protocol": {18        "reference": "lmms-eval@lmms_eval/tasks/mantis/utils.py (get_prediction L65, parse_answer L44, get_option L56) — MCQ rows: parse 'Final Answer:'/'the answer is' then first alphabetic char as option letter, exact letter compare (with option-text fallback); short-answer rows: case-insensitive exact match (L97); no LLM. Identical logic in the authors' own repo at github.com/TIGER-AI-Lab/Mantis mantis/benchmark/eval.py (parse_answer L16, get_option L27, get_prediction L35).",19        "note": "Mixed format: 17/217 published rows are short-answer (no options; answers like '2', 'soccer', 'Yes', 'Different ones') — verified across all 217 rows. The subset-level prompt template renders an empty 'Options:' block plus \"Answer with the option's letter\" trailer for those 17 rows, diverging from the official short-answer prompt ('give your final answer as \"Final Answer:\"'). Official grading of short answers is strict lowercase exact match, which is brittle for phrase answers."20      },21      "prompt_template": "{{ question }}\nOptions:\n{% for k, v in options.items() %}{{ k }}. {{ v }}{% if not loop.last %}\n{% endif %}{% endfor %}\nAnswer with the option's letter from the given choices directly.",22      "mapping_from_source": {23        "media": {24          "from": "images",25          "type": "list",26          "min_items": 2,27          "max_items": 528        },29        "id": {30          "from": "id"31        },32        "question": {33          "from": "question"34        },35        "answer": {36          "from": "answer",37          "optional": true38        },39        "options": {40          "from": "options",41          "optional": true,42          "note": "list source values are normalized to {A,B,...} dict"43        },44        "extra": {45          "category": {46            "from": "category"47          }48        },49        "source": {50          "format": "json",51          "url": {52            "test": "https://huggingface.co/datasets/TIGER-Lab/Mantis-Eval"53          }54        }55      },56      "prompt_template_source": {57        "origin": "official",58        "reference": "https://github.com/TIGER-AI-Lab/Mantis/blob/main/data/eval/Mantis-Instruct (Mantis benchmark multi-choice template)",59        "notes": "Tier 1: Mantis-Eval canonical multi-image MCQ template (TIGER-AI-Lab Mantis authors)."60      }61    }62  }63}