mm-eval/HRBench8K

1{2  "name": "HRBench8K",3  "release_date": "2026-05-15",4  "subsets": {5    "main": {6      "language": [7        "en"8      ],9      "modalities": [10        "single_image_start"11      ],12      "task_type": "multiple_choice_qa",13      "score_pipeline": [14        "rule-match",15        "llm-match"16      ],17      "score_protocol": {18        "reference": "vlmevalkit@vlmeval/dataset/image_mcq.py:1302-1364 HRBenchDataset.evaluate -> mcq_vanilla_eval (can_infer + GPT extractor; DEFAULT_JUDGE ['chatgpt-0125','gpt-4-0125'] @vlmeval/dataset/image_mcq.py:1303) + report_acc_hrbench @vlmeval/dataset/utils/hrbench.py:7-29 (per-cycle x category accuracy, averaged over 4 cycles). HR-Bench authors (DreamMr/HR-Bench) vendor VLMEvalKit, so this is tier-1. Cross-check: lmms-eval hrbench also GPT-extracts (lmms_eval/tasks/hrbench/utils.py:24)",19        "note": "Official report averages accuracy per cycle_category (0-3) and per category (single/cross); the mm-eval copy ships all 4 rotations as rows (800 = 4 x 200, cycle_category field preserved in extra), so overall per-sample accuracy equals the official 'all' average (cycles partition the file); the per-category/per-cycle breakdown itself is not reproduced per-sample. Note this is NOT CircularEval — each rotation counts independently."20      },21      "prompt_template": "<image>Question: {{ question }}\nOptions:\nA. {{ options.A }}\nB. {{ options.B }}\nC. {{ options.C }}\nD. {{ options.D }}\nPlease select the correct answer from the options above. \n",22      "mapping_from_source": {23        "media": {24          "from": "image",25          "type": "list",26          "min_items": 1,27          "max_items": 128        },29        "id": {30          "from": "index"31        },32        "question": {33          "from": "question"34        },35        "answer": {36          "from": "answer",37          "optional": true38        },39        "extra": {40          "category": {41            "from": "category"42          },43          "A": {44            "from": "A"45          },46          "B": {47            "from": "B"48          },49          "C": {50            "from": "C"51          },52          "D": {53            "from": "D"54          },55          "cycle_category": {56            "from": "cycle_category"57          }58        },59        "source": {60          "format": "json",61          "url": {62            "hrbench_8k": "https://huggingface.co/datasets/DreamMr/HR-Bench"63          }64        }65      },66      "prompt_template_source": {67        "origin": "official",68        "reference": "https://github.com/open-compass/VLMEvalKit/blob/main/vlmeval/dataset/image_mcq.py (ImageMCQDataset.build_prompt — Question:/Options:/A.B.C.D./'Please select the correct answer from the options above. ' trailer; HR-Bench authors' eval at https://github.com/DreamMr/HR-Bench/blob/main/hrbench/vlmeval/dataset/image_mcq.py inherits this via HRBenchDataset)",69        "notes": "Tier 1/3: HR-Bench authors use VLMEvalKit's ImageMCQDataset.build_prompt; mm-eval prompt reproduces it byte-for-byte. lmms-eval's hrbench_doc_to_text uses a different trailer ('Answer the option letter directly.') and is not the canonical reference."70      }71    }72  }73}