visual-riddles/Visual-Riddles-Leaderboard

1Model	Open Ended VQA: % Human Rating	Multiple Choice VQA: % Accuracy	Hints-Multiple Choice VQA: % Accuracy 	Attributions-Multiple Choice VQA: % Accuracy 	Refernce Based-Automatic Evaluation: Accuracy of Judge Prediction Compared to Human Ratings	Refernce Free-Automatic Evaluation: Accuracy of Judge Prediction Compared to Human Ratings	Automatic Evaluation: % Auto-Rater Ratings	Hints-Automatic Evaluation: % Auto-Rater Ratings	Attributions-Automatic Evaluation: % Auto-Rater Ratings
2Humans	82	*	*	*	*	*	78	*	*
3Gemini Pro 1.5	40	38	66	72	87	52	53	62	29
4Gemini Pro Vision	30	41	62	*	75	38	34	47	
5GPT4	34	45	69	82	86	51	38	61	25
6LlaVA-1.6-34B	15	24	30	*	76	43	21	16	*
7LlaVA-1.5-7B	13	17	29	*	70	35	19	30	*
8InstructBlip	13	*	*	*	*	*	20	28	*
9Gemini Pro 1.5 Caption _ Gemini Pro 1.5	23	*	*	*	*	*	*	*	*
10Human (Oracle) Caption _ Gemini Pro 1.5	50	*	*	*	*	*	*	*	*
11Claude 3.5 Sonnet	*	46	45	*	*	*	39	*	*
12GPT4o	*	55	83	*	*	*	50	*	*
13Qwen-VL-Max	*	35	53	*	*	*	26	*	*
14Molmo-7B	*	34	42	*	*	*	36	*	*