Apppublic

steampunque/benchlm

sourceHugging Faceapache-2.0updated 20d agoView on Hugging Face

3likes

App README

To see results on github go to https://github.com/steampunque/benchlm

To see results on hf go to https://huggingface.co/spaces/steampunque/benchlm

Independent LLM benchmarks for a wide range of open weight models using custom prompts
including category and discipline summaries.  The model list is actively updated
with latest models releases.  Older obsoleted model results are not kept.

The primary model families being tracked as of 6/2/25 are:
Meta (Llama), Mistral, Google (Gemma), Qwen (Qwen2.5, Qwen2.5 coder, Qwen3, QwQ)

Secondary model being tracked as of 6/2/25 are:
Microsoft (Phi), Deepseek (Qwen R1 distills), Falcon family, internlm family, GLM family, ultravox family.

Models being tracked are selected based on general high popularity, high
performance or other innovation, with non-restrictive open source license terms.

Tests are run using a modified llama.cpp server (supporting logprob completion mode).

TEST CATEGORIES:
   CHAT    : wide range of synthetic evals for chat capable models
           : tests instruct tuned text in text out models
   THINK   : same set of evals as CHAT category
           : tests RL tuned reasoning models with <think> </think> block or equivalent
           : note think may be disabled on many evals as described in methodology section below
   CODE    : coding centric evals
           : tests code optimized models
   MATH    : models applied to Hendryks MATH500 set
           : tests math optimized and RL tuned models
   VISION  : image + text in, text out
           : tests vision enabled models
   AUDIO   : audio + text in, text out
           : tests audio enabled models
   MT      : machine language translation
           : tests MT optimized models

METHODOLOGY:
   -The majority of the benchmarks use zero shot instruct prompts, as opposed to other benchmarks suites
    such as lm-evaluation-harness which treats any model whether instruct tuned or not as a
    completion model and determines results based on completion logprobs and/or multi shot prompting
    to "teach" the model what it is supposed to do on a prompt by prompt basis.  As a result the
    majority of the benchmarks are run on instruct tuned models, with some exceptions such as
    coding benchmarks and a few native completion based benchmarks such as lambada and winogrande.
    This strategy is intentional and is explicitly designed to evaluate the capability of the
    instruction tuning of the models.  It also means results from the same benchmark can differ
    significantly with results from completion logprobs/multishot prompting evals such as
    lm-evaluation harness.  Since the majority use case of instruct models is zero shot by design
    this benchmark is more relevant to actual use case of the instruct tuned models.
   -All CoT, code, and math tests are zero shot.  A few BBH tests use fewshot examples.
   -Math CoT test such as GSM8K, APPLE, MATH etc. are self graded against correct answer using LLM under test
      If self grade does not work reliably (such as with very small model) the result is zeroed to mark invalid test.
   -All non-CoT MC tests do two queries, 1 with answers in test order and 2nd with answers circularly shifted 1.
      To score a correct answer in MC both queries must answer correctly.
   -All not-CoT tests (non-CoT MC, etc.) disable the think block prefix for thinking models.
    The think block is also disabled for all CODE tests with thinking models.
   -Cot MC tests (e.g. MMLUPRO, GPQA, etc.) do one query only.
   -Winogrande using logprob completion (evaluates the probability of a common completion for the two possible cases).
   -The new SQA test is not run on all models.  When it is run, the result is only added to SQA test.
      The SQA result is not added to knowledge and composite averages.  The test is not meant for
      small models and results are given for information only.
   -MMMU uses the validation split (30 questions from 30 categories) and CoT prompting
   -MMMUPRO uses the 10-question split and CoT prompting
   -BBA provides reference BBH results (BBHA) so dropoff from audio prompts can be seen
    BBHA results are not part of the final tablulated average for the audio test.
   -MT results are based on averaged sentence BLEU over the testsets using
      languages de, es, fr, ru, ja, and zh to/from en.
      unique tokenizers are used for ja, zh, and ru for bleu eval
   -All new tests are run with a maximum of 250 questions per test category on CoT tests.
      This is necessary to contain test time with new thinking models when can generate very lengthy responses.
      The result is printed in italics if there were more than 10 skipped questions in a test category.
      Note some very old runs had skips due to JSON errors in questions but these will not significantly impact averages.

TESTS:
   KNOWLEDGE:
      TQA - Truthful QA
      SQA - Simple QA 4333 question arcane knowledge quiz
      JEOPARDY - 100 Question JEOPARDY quiz
   LANGUAGE:
      LAMBADA - Language Modeling Broadened to Account for Discourse Aspects
   UNDERSTANDING:
      WG - Winogrande
      BOOLQ - Boolean questions
      STORYCLOZE - Story questions
      OBQA - Open Book Question / Answer
      SIQA - Social IQ
      RACE - Reading comprehension dataset from examinations
      MMLU - massive multitask language understanding
      MEDQA - medical QA
   REASONING
      CSQA - Common Sense Question Answer
      COPA - Choice of Plausible Alternatives
      HELLASWAG -Harder Endings, Longer contexts, and Low-shot Activities
                 for Situations With Adversarial Generations
      PIQA - Physical Interaction: Question Answering
      ARC - A12 Reasoning Challenge
      AGIEVAL - AGIEval logiqa, lsat, sat
      AGIEVALC  - Gaokao SAT, logiqa, jec (Chinese)
      MUSR - Multimodal Semantic Reasoning
   COT:
      GSM8K - Grade School Math CoT
      BBH  - Beyond the Imitation Game Bench Hard CoT
      GPQA - Google-Proof QA science CoT
      MMLUPRO - massive multitask language understanding pro CoT
      AGIEVAL - satmath, aquarat
      AGIEVALC  - mathcloze, mathqa (Chinese)
      MUSR - Multimodal Semantic Reasoning
      APPLE - 100 custom Apple Questions
   MATH:
      MATH1..MATH5 - MATH Datasets level 1 through 5 (Hendrycks et al.)
   CODE:
      HUMANEVAL - Python
      HUMANEVALP - Python, extended test
      HUMANEVALX - Python, Java, Javascript, C++
      MBPP - Python
      MBPPP - Python, extendend test
      CRUXEVAL - Python
      USE {TEST}FIM FOR FIM TEST, i.e. HUMANEVAL->HUMANEVALFIM
   VISION:
      CHARTQA - Chart Question/Answer
      DOCVQA  - Document Vision QA
      REALWORLDQA - Realworld QA
      MMMU - Massive Multi-discipline Multimodal Understanding (CoT)
      MMMUPRO - Massive Multi-discipline Multimodal Understanding Pro (CoT)
   AUDIO:
      BBA - Big Bench Audio
      BBHA - Big Bench Hard Audio subset with original text prompts
   MT:
      OPUS - Open Parallel Corpora
      FLORES200 - Facebook Low Resource

CHAT MODEL TESTS:

MODEL	Falcon3-1B-Instruct	Falcon3-7B-Instruct	Falcon3-10B-Instruct	gemma-2-9b-it	gemma-2-27b-it	gemma-3-1b-it	gemma-3-4b-it	gemma-3-12b-it	gemma-3-12b-it	gemma-3-27b-it	gemma-4-E4B-it	glm-4-9b-chat	glm-4-9b-chat	internlm3-8b-instruct	JoyAI-LLM-Flash	Ling-mini-2.0	Ling-mini-2.0	Llama-3.1-8B-Instruct	Llama-3.2-3B-Instruct	Llama-4-Scout-17B-16E-Instruct	Llama-4-Scout-17B-16E-Instruct	Llama-4-Scout-17B-16E-Instruct	Mistral-7B-Instruct-v0.3	Mistral-Small-3.1-24B-Instruct-2503	Mistral-Small-3.1-24B-Instruct-2503	Mistral-Small-3.1-24B-Instruct-2503	Mistral-Small-3.2-24B-Instruct-2506	Phi-3.5-mini-8k-instruct	Phi-3.5-mini-128k-instruct	Phi-4-mini-instruct	Phi-4	phi-4	Qwen2.5-3B-32k-Instruct	Qwen2.5-3B-32k-Instruct	Qwen2.5-7B-32k-Instruct	Qwen2.5-7B-32k-Instruct	Qwen2.5-14B-32k-Instruct	Qwen2.5-32B-Instruct	Qwen3-4B-Instruct-2507
params	1.67B	7.46B	10.31B	9.24B	27.23B	0.99989B	3.88B	11.77B	11.77B	27.01B	7.52B	9.40B	9.40B	8.80B	48.94B	16.26B	16.26B	8.03B	3.21B	107.77B	107.77B	107.77B	7.25B	23.57B	23.57B	23.57B	23.57B	3.82B	3.82B	3.84B	14.66B	14.66B	3.09B	3.09B	7.62B	7.62B	14.77B	32.76B	4.02B
quant	IQ4_XS	Q6_K	IQ4_XS	Q6_K	IQ4_XS	Q8_0	Q6_K	IQ4_XS	Q4KH	Q4KH	Q6KH	IQ4_XS	Q6_K	IQ4_XS	Q4KH	Q6_K	Q6KH	Q6_K	Q6_K	Q2KH	Q3KH	Q4KH	Q8_0	Q2KH	Q3KH	Q4KH	Q4KH	Q6_K	Q6_K	Q6_K	IQ4_XS	Q4KH	IQ4_XS	Q6_K	IQ4_XS	Q6_K	IQ4_XS	IQ4_XS	Q6KH
engine	llama.cpp version: 4341	llama.cpp version: 4341	llama.cpp version: 4341	llama.cpp version: 3266	llama.cpp version: 3389	llama.cpp version: 4877	llama.cpp version: 4888	llama.cpp version: 4938	llama.cpp version: 5572	llama.cpp version: 5586	llama.cpp version: 8777	llama.cpp version: 3496	llama.cpp version: 3334	llama.cpp version: 4488	llama.cpp version: 8086	llama.cpp version: 6827	llama.cpp version: 6827	llama.cpp version: 3428	llama.cpp version: 3825	llama.cpp version: 5236	llama.cpp version: 5279	llama.cpp version: 5335	llama.cpp version: 3262	llama.cpp version: 5509	llama.cpp version: 5509	llama.cpp version: 5509	llama.cpp version: 5742	llama.cpp version: 3609	llama.cpp version: 3600	llama.cpp version: 4792	llama.cpp version: 4295	llama.cpp version: 7562	llama.cpp version: 4038	llama.cpp version: 4038	llama.cpp version: 3943	llama.cpp version: 3870	llama.cpp version: 3821	llama.cpp version: 3821	llama.cpp version: 6628
TEST	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc
WG	0.600	0.670	0.700	0.762	0.772	0.576	0.692	0.743	0.741	0.748	0.662	0.759	0.753	0.708	0.680	0.634	0.637	0.741	0.685	-	-	-	0.751	0.775	0.772	0.784	0.780	0.744	0.734	0.707	0.708	0.699	0.687	0.695	0.709	0.709	0.754	0.746	0.677
LAMBADA	0.524	0.688	0.692	0.735	0.755	0.504	0.635	0.724	0.721	0.742	0.660	0.786	0.783	0.662	0.597	0.621	0.617	0.747	0.705	-	-	-	0.766	0.786	0.789	0.798	0.792	0.677	0.613	0.653	0.750	0.751	0.685	0.682	0.722	0.724	0.769	0.781	0.670
HELLASWAG	0.308	0.684	0.716	0.775	0.810	0.307	0.527	0.779	0.767	0.802	0.737	0.834	0.840	0.846	0.903	0.673	0.700	0.696	0.559	-	-	-	0.591	0.866	0.877	0.899	0.872	0.716	0.669	0.542	0.801	0.819	0.670	0.713	0.820	0.822	0.863	0.894	0.775
BOOLQ	0.364	0.591	0.621	0.687	0.739	0.521	0.603	0.669	-	0.701	0.603	0.633	0.625	0.562	0.723	0.544	0.585	0.610	0.478	-	-	-	0.658	-	-	0.646	0.684	0.562	0.573	0.453	0.653	0.649	0.517	0.533	0.617	0.623	0.647	0.701	0.606
STORYCLOZE	0.774	0.949	0.947	0.958	0.973	0.685	0.900	0.948	-	0.964	0.951	0.967	0.976	0.982	0.957	0.893	0.924	0.895	0.870	-	-	-	0.917	-	-	0.968	0.969	0.531	0.921	0.889	0.754	0.964	0.913	0.896	0.920	0.915	0.938	0.981	0.928
CSQA	0.488	0.725	0.746	0.751	0.763	0.339	0.614	0.716	-	0.741	0.717	0.727	0.733	0.730	0.805	0.717	0.746	0.686	0.642	-	-	-	0.627	-	-	0.756	0.751	0.669	0.660	0.633	0.740	0.769	0.701	0.717	0.768	0.781	0.795	0.823	0.737
OBQA	0.380	0.761	0.745	0.846	0.860	0.334	0.648	0.807	-	0.855	0.823	0.821	0.802	0.801	0.870	0.787	0.806	0.765	0.709	-	-	-	0.676	-	-	0.866	0.880	0.751	0.720	0.719	0.857	0.859	0.700	0.731	0.802	0.804	0.863	0.904	0.804
COPA	0.612	0.870	0.903	0.925	0.949	0.415	0.785	0.932	-	0.944	0.922	0.955	0.944	0.927	0.952	0.863	0.884	0.889	0.749	-	-	-	0.812	-	-	0.924	0.932	0.884	0.870	0.834	0.934	0.944	0.841	0.858	0.925	0.919	0.935	0.958	0.887
PIQA	0.233	0.696	0.732	0.801	0.841	0.386	0.653	0.784	-	0.818	0.793	0.773	0.779	0.777	0.866	0.749	0.776	0.725	0.637	-	-	-	0.708	-	-	0.826	0.831	0.733	0.677	0.674	0.832	0.849	0.695	0.713	0.794	0.807	0.848	0.870	0.761
SIQA	0.425	0.658	0.688	0.693	0.731	0.385	0.588	0.699	-	0.716	0.697	0.664	0.665	0.706	0.729	0.638	0.653	0.648	0.622	-	-	-	0.620	-	-	0.737	0.710	0.667	0.661	0.645	0.639	0.721	0.656	0.663	0.721	0.712	0.746	0.742	0.692
MEDQA	0.141	0.420	0.430	0.501	0.549	0.073	0.292	0.503	-	0.553	0.516	0.436	0.445	0.457	0.663	0.443	0.472	0.500	0.413	-	-	-	0.334	-	-	0.593	0.597	0.423	0.395	0.361	0.560	0.610	0.344	0.363	0.453	0.458	0.542	0.610	0.494
SQA	-	0.033	-	-	0.117	-	0.052	0.092	-	0.092	0.058	-	-	0.039	0.058	0.059	0.079	0.073	-	-	-	-	-	-	-	0.066	0.073	-	-	0.039	-	0.072	-	-	-	-	-	-	0.058
JEOPARDY	0.010	0.400	0.310	0.580	0.760	-	0.350	0.550	0.560	0.830	0.460	0.370	0.420	0.210	0.490	0.680	0.490	0.510	0.350	0.680	0.580	0.540	0.490	0.680	-	0.740	0.640	0.320	0.250	0.280	0.390	0.520	0.120	0.120	0.300	0.290	0.540	0.600	0.310
GSM8K	0.485	0.890	0.918	0.890	0.899	-	0.843	0.928	0.928	0.964	0.904	0.855	0.839	0.890	0.964	0.956	0.964	0.872	0.822	-	-	-	0.611	-	-	0.940	0.968	0.855	0.714	0.868	0.946	0.944	0.829	0.856	0.909	0.880	0.938	0.950	0.960
APPLE	0.150	0.810	0.740	0.750	0.730	-	0.630	0.740	0.770	0.850	0.850	0.630	0.610	0.670	0.940	0.860	0.890	0.690	0.610	0.840	0.860	0.860	0.390	0.830	0.780	0.820	0.890	0.560	0.560	0.640	0.910	0.850	0.640	0.560	0.740	0.750	0.830	0.860	0.850
HUMANEVAL	0.115	0.737	0.774	0.658	0.743	0.408	0.701	0.859	0.829	0.890	0.792	0.737	0.731	0.804	0.884	0.829	0.841	0.652	0.585	-	-	-	0.390	0.841	0.823	0.853	0.871	0.682	0.621	0.646	0.847	0.829	0.695	0.780	0.798	0.817	0.804	0.884	0.841
HUMANEVALP	0.073	0.628	0.664	0.548	0.615	0.317	0.597	0.713	-	0.719	0.713	0.615	0.634	0.713	0.780	0.731	0.713	0.536	0.475	-	-	-	0.329	-	-	0.731	0.750	0.591	0.524	0.554	0.725	0.713	0.615	0.682	0.670	0.658	0.676	0.768	0.713
HUMANEVALFIM	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
MBPP	0.334	0.677	0.653	0.595	0.642	0.536	0.614	0.692	-	0.677	0.533	0.579	0.591	0.552	0.439	0.696	0.665	0.564	0.498	-	-	-	0.451	-	-	0.618	0.642	0.610	0.498	0.501	0.673	0.673	0.595	0.599	0.669	0.661	0.669	0.684	0.665
MBPPP	0.312	0.629	0.611	0.584	0.638	0.531	0.598	0.642	-	0.625	0.504	0.562	0.575	0.477	0.464	-	0.660	0.540	0.482	-	-	-	0.397	-	-	0.593	0.647	0.575	0.477	0.504	0.651	0.638	0.540	0.584	0.633	0.651	0.633	0.700	0.629
HUMANEVALX_cpp	0.054	0.506	0.603	0.512	0.579	0.158	0.585	0.756	-	0.780	0.603	0.439	0.432	0.402	0.695	-	0.719	0.457	0.323	-	-	-	0.225	-	-	0.292	0.713	0.280	0.219	0.445	0.676	0.670	0.420	0.237	0.475	0.554	0.323	0.701	0.652
HUMANEVALX_java	0.042	0.640	0.719	0.640	0.768	0.317	0.658	0.804	-	0.810	0.689	0.207	0.628	0.597	0.908	-	0.798	0.487	0.439	-	-	-	0.256	-	-	0.804	0.829	0.079	0.060	0.536	0.634	0.524	0.640	0.615	0.695	0.737	0.780	0.865	0.823
HUMANEVALX_js	0.115	0.676	0.652	0.579	0.743	0.359	0.664	0.835	-	0.841	0.481	0.628	0.628	0.670	0.878	-	0.829	0.560	0.067	-	-	-	0.402	-	-	0.786	0.786	0.560	0.451	0.548	0.786	0.804	0.646	0.689	0.719	0.750	0.798	0.847	0.841
HUMANEVALX	0.071	0.607	0.658	0.577	0.697	0.278	0.636	0.798	-	0.810	0.591	0.424	0.563	0.556	0.827	-	0.782	0.502	0.276	-	-	-	0.294	-	-	0.628	0.776	0.306	0.243	0.510	0.699	0.666	0.569	0.514	0.630	0.680	0.634	0.804	0.772
CRUXEVAL_input	0.210	0.411	0.448	0.462	0.485	0.038	0.388	0.440	-	0.528	0.733	0.416	0.406	0.477	0.475	-	0.472	0.435	0.353	-	-	-	0.276	-	-	0.547	0.550	0.398	0.388	0.336	0.447	0.461	0.350	0.331	0.387	0.412	0.541	0.517	0.518
CRUXEVAL_output	0.152	0.355	0.410	0.375	0.482	0.196	0.348	0.457	-	0.491	0.805	0.356	0.338	0.372	0.522	-	0.458	0.360	0.291	-	-	-	0.303	-	-	0.516	0.498	0.342	0.296	0.317	0.463	0.475	0.275	0.311	0.382	0.386	0.471	0.455	0.463
CRUXEVAL	0.181	0.383	0.429	0.418	0.483	0.117	0.368	0.448	-	0.510	0.769	0.386	0.372	0.425	0.498	-	0.465	0.397	0.322	-	-	-	0.290	-	-	0.531	0.524	0.370	0.342	0.326	0.455	0.468	0.312	0.321	0.385	0.399	0.506	0.486	0.491
CRUXEVALFIM_input	-	0.418	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
CRUXEVALFIM_output	-	0.356	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
CRUXEVALFIM	-	0.387	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
TQA_mc	0.146	0.523	0.510	0.701	0.767	0.115	0.468	0.663	-	0.696	0.643	0.636	0.640	0.637	0.755	-	0.621	0.564	0.555	-	-	-	0.549	-	-	0.767	0.713	0.621	0.581	0.477	0.725	0.740	0.516	0.548	0.654	0.657	0.747	0.804	0.707
TQA_tf	0.381	0.410	0.431	0.692	0.725	0.390	0.491	0.677	-	0.634	0.654	0.484	0.457	0.593	0.654	-	0.592	0.512	0.566	-	-	-	0.548	-	-	0.735	0.670	0.483	0.487	0.588	0.686	0.700	0.414	0.300	0.574	0.568	0.706	0.731	0.640
TQA	0.354	0.423	0.440	0.693	0.730	0.358	0.488	0.676	-	0.641	0.653	0.502	0.478	0.598	0.666	-	0.596	0.518	0.565	-	-	-	0.548	-	-	0.739	0.675	0.499	0.498	0.575	0.691	0.705	0.426	0.329	0.583	0.578	0.711	0.740	0.648
ARC_challenge	0.374	0.809	0.819	0.882	0.897	0.319	0.699	0.869	-	0.899	0.874	0.835	0.853	0.871	0.918	-	0.877	0.776	0.706	-	-	-	0.688	-	-	0.912	0.907	0.813	0.802	0.759	0.911	0.912	0.750	0.777	0.843	0.851	0.911	0.934	0.875
ARC_easy	0.598	0.925	0.933	0.952	0.963	0.563	0.875	0.955	-	0.971	0.953	0.933	0.940	0.945	0.970	-	0.956	0.906	0.843	-	-	-	0.843	-	-	0.970	0.968	0.934	0.932	0.908	0.970	0.969	0.895	0.904	0.945	0.946	0.969	0.978	0.952
ARC	0.524	0.886	0.895	0.929	0.941	0.482	0.817	0.927	-	0.947	0.927	0.901	0.911	0.920	0.953	-	0.930	0.863	0.798	-	-	-	0.792	-	-	0.951	0.948	0.894	0.889	0.859	0.950	0.950	0.847	0.862	0.911	0.915	0.950	0.963	0.927
RACE_high	0.431	0.698	0.730	0.802	0.833	0.338	0.633	0.795	-	0.829	0.794	0.788	0.787	0.830	0.872	-	0.773	0.679	0.589	-	-	-	0.607	-	-	0.853	0.853	0.613	0.625	0.690	0.819	0.830	0.698	0.712	0.779	0.788	0.852	0.882	0.802
RACE_middle	0.463	0.777	0.793	0.849	0.883	0.398	0.713	0.858	-	0.883	0.839	0.816	0.825	0.866	0.905	-	0.839	0.734	0.680	-	-	-	0.696	-	-	0.894	0.885	0.706	0.692	0.745	0.861	0.868	0.775	0.776	0.841	0.853	0.887	0.923	0.850
RACE	0.440	0.721	0.748	0.816	0.847	0.355	0.656	0.813	-	0.844	0.807	0.796	0.798	0.840	0.882	-	0.793	0.695	0.615	-	-	-	0.633	-	-	0.865	0.862	0.640	0.645	0.706	0.831	0.841	0.720	0.730	0.797	0.807	0.862	0.894	0.816

MMLU abstractalgebra | 0.180 | 0.410 | 0.450 | 0.330 | 0.310 | 0.110 | 0.160 | 0.350 | - | 0.410 | 0.410 | 0.220 | 0.210 | 0.300 | 0.560 | - | 0.340 | 0.200 | 0.270 | - | - | - | 0.190 | - | - | 0.370 | 0.430 | 0.300 | 0.210 | 0.210 | 0.410 | 0.510 | 0.240 | 0.250 | 0.440 | 0.430 | 0.570 | 0.600 | 0.510 | anatomy | 0.318 | 0.577 | 0.592 | 0.626 | 0.607 | 0.296 | 0.459 | 0.577 | - | 0.629 | 0.607 | 0.503 | 0.511 | 0.637 | 0.725 | - | 0.592 | 0.555 | 0.540 | - | - | - | 0.447 | - | - | 0.718 | 0.725 | 0.570 | 0.585 | 0.503 | 0.703 | 0.711 | 0.525 | 0.562 | 0.622 | 0.622 | 0.644 | 0.733 | 0.585 | astronomy | 0.263 | 0.736 | 0.756 | 0.760 | 0.828 | 0.296 | 0.559 | 0.769 | - | 0.848 | 0.703 | 0.644 | 0.651 | 0.802 | 0.855 | - | 0.743 | 0.677 | 0.565 | - | - | - | 0.573 | - | - | 0.888 | 0.888 | 0.703 | 0.703 | 0.611 | 0.776 | 0.835 | 0.618 | 0.657 | 0.763 | 0.769 | 0.868 | 0.875 | 0.789 | businessethics | 0.260 | 0.570 | 0.560 | 0.620 | 0.670 | 0.270 | 0.480 | 0.630 | - | 0.710 | 0.660 | 0.570 | 0.610 | 0.670 | 0.760 | - | 0.700 | 0.550 | 0.480 | - | - | - | 0.520 | - | - | 0.730 | 0.720 | 0.620 | 0.620 | 0.560 | 0.740 | 0.750 | 0.630 | 0.590 | 0.680 | 0.710 | 0.750 | 0.800 | 0.680 | clinicalknowledge | 0.373 | 0.652 | 0.683 | 0.743 | 0.788 | 0.316 | 0.554 | 0.716 | - | 0.784 | 0.743 | 0.618 | 0.622 | 0.716 | 0.781 | - | 0.716 | 0.675 | 0.592 | - | - | - | 0.581 | - | - | 0.811 | 0.777 | 0.713 | 0.698 | 0.637 | 0.781 | 0.781 | 0.633 | 0.645 | 0.709 | 0.713 | 0.803 | 0.815 | 0.732 | collegebiology | 0.340 | 0.763 | 0.777 | 0.854 | 0.895 | 0.256 | 0.618 | 0.826 | - | 0.847 | 0.868 | 0.687 | 0.715 | 0.777 | 0.923 | - | 0.777 | 0.722 | 0.625 | - | - | - | 0.625 | - | - | 0.888 | 0.902 | 0.805 | 0.763 | 0.694 | 0.868 | 0.895 | 0.694 | 0.694 | 0.784 | 0.784 | 0.854 | 0.923 | 0.840 | collegechemistry | 0.180 | 0.470 | 0.430 | 0.470 | 0.430 | 0.180 | 0.260 | 0.400 | - | 0.470 | 0.460 | 0.380 | 0.380 | 0.440 | 0.460 | - | 0.440 | 0.400 | 0.310 | - | - | - | 0.350 | - | - | 0.480 | 0.500 | 0.460 | 0.430 | 0.420 | 0.520 | 0.540 | 0.310 | 0.370 | 0.480 | 0.490 | 0.460 | 0.530 | 0.490 | collegecomputerscience | 0.110 | 0.540 | 0.590 | 0.460 | 0.580 | 0.200 | 0.360 | 0.500 | - | 0.560 | 0.610 | 0.470 | 0.480 | 0.640 | 0.740 | - | 0.600 | 0.400 | 0.350 | - | - | - | 0.320 | - | - | 0.650 | 0.580 | 0.480 | 0.410 | 0.420 | 0.600 | 0.640 | 0.390 | 0.460 | 0.620 | 0.590 | 0.630 | 0.720 | 0.620 | collegemathematics | 0.090 | 0.320 | 0.320 | 0.260 | 0.300 | 0.080 | 0.170 | 0.300 | - | 0.400 | 0.380 | 0.240 | 0.280 | 0.280 | 0.530 | - | 0.460 | 0.260 | 0.210 | - | - | - | 0.180 | - | - | 0.340 | 0.380 | 0.270 | 0.170 | 0.160 | 0.340 | 0.410 | 0.200 | 0.180 | 0.380 | 0.350 | 0.490 | 0.540 | 0.380 | collegemedicine | 0.283 | 0.566 | 0.612 | 0.658 | 0.716 | 0.260 | 0.462 | 0.624 | - | 0.682 | 0.653 | 0.572 | 0.589 | 0.676 | 0.728 | - | 0.658 | 0.589 | 0.491 | - | - | - | 0.456 | - | - | 0.722 | 0.728 | 0.612 | 0.566 | 0.531 | 0.728 | 0.757 | 0.560 | 0.606 | 0.606 | 0.624 | 0.710 | 0.739 | 0.647 | collegephysics | 0.186 | 0.372 | 0.411 | 0.352 | 0.421 | 0.098 | 0.196 | 0.411 | - | 0.470 | 0.411 | 0.313 | 0.323 | 0.362 | 0.480 | - | 0.490 | 0.313 | 0.303 | - | - | - | 0.254 | - | - | 0.529 | 0.539 | 0.333 | 0.294 | 0.284 | 0.529 | 0.509 | 0.382 | 0.392 | 0.401 | 0.372 | 0.519 | 0.656 | 0.529 | computersecurity | 0.370 | 0.710 | 0.690 | 0.730 | 0.710 | 0.350 | 0.590 | 0.740 | - | 0.760 | 0.730 | 0.710 | 0.730 | 0.720 | 0.750 | - | 0.690 | 0.690 | 0.620 | - | - | - | 0.600 | - | - | 0.710 | 0.720 | 0.700 | 0.650 | 0.700 | 0.730 | 0.760 | 0.650 | 0.690 | 0.720 | 0.710 | 0.730 | 0.800 | 0.710 | conceptualphysics | 0.234 | 0.680 | 0.680 | 0.638 | 0.727 | 0.174 | 0.404 | 0.634 | - | 0.748 | 0.672 | 0.561 | 0.587 | 0.646 | 0.800 | - | 0.697 | 0.463 | 0.361 | - | - | - | 0.365 | - | - | 0.744 | 0.731 | 0.565 | 0.553 | 0.544 | 0.748 | 0.765 | 0.485 | 0.519 | 0.642 | 0.642 | 0.800 | 0.834 | 0.753 | econometrics | 0.122 | 0.649 | 0.587 | 0.557 | 0.587 | 0.140 | 0.315 | 0.535 | - | 0.570 | 0.561 | 0.456 | 0.464 | 0.578 | 0.631 | - | 0.578 | 0.482 | 0.359 | - | - | - | 0.318 | - | - | 0.587 | 0.614 | 0.456 | 0.421 | 0.368 | 0.596 | 0.561 | 0.421 | 0.438 | 0.605 | 0.596 | 0.649 | 0.675 | 0.622 | electricalengineering | 0.220 | 0.641 | 0.648 | 0.558 | 0.593 | 0.296 | 0.393 | 0.558 | - | 0.627 | 0.586 | 0.544 | 0.572 | 0.655 | 0.689 | - | 0.662 | 0.524 | 0.462 | - | - | - | 0.393 | - | - | 0.703 | 0.662 | 0.496 | 0.475 | 0.572 | 0.634 | 0.675 | 0.441 | 0.434 | 0.606 | 0.606 | 0.648 | 0.703 | 0.634 | elementarymathematics | 0.113 | 0.505 | 0.497 | 0.476 | 0.476 | 0.058 | 0.288 | 0.502 | - | 0.719 | 0.497 | 0.367 | 0.373 | 0.481 | 0.574 | - | 0.626 | 0.357 | 0.280 | - | - | - | 0.222 | - | - | 0.653 | 0.621 | 0.423 | 0.388 | 0.335 | 0.544 | 0.579 | 0.407 | 0.417 | 0.560 | 0.568 | 0.791 | 0.838 | 0.616 | formallogic | 0.182 | 0.444 | 0.484 | 0.293 | 0.468 | 0.142 | 0.269 | 0.452 | - | 0.507 | 0.531 | 0.325 | 0.357 | 0.420 | 0.571 | - | 0.380 | 0.420 | 0.253 | - | - | - | 0.277 | - | - | 0.563 | 0.484 | 0.452 | 0.380 | 0.380 | 0.531 | 0.587 | 0.325 | 0.341 | 0.452 | 0.428 | 0.539 | 0.626 | 0.579 | globalfacts | 0.120 | 0.190 | 0.290 | 0.330 | 0.370 | 0.070 | 0.110 | 0.300 | - | 0.420 | 0.230 | 0.200 | 0.240 | 0.330 | 0.540 | - | 0.220 | 0.150 | 0.110 | - | - | - | 0.160 | - | - | 0.520 | 0.450 | 0.240 | 0.130 | 0.120 | 0.320 | 0.340 | 0.140 | 0.200 | 0.260 | 0.260 | 0.470 | 0.430 | 0.270 | highschoolbiology | 0.348 | 0.764 | 0.774 | 0.851 | 0.890 | 0.358 | 0.645 | 0.816 | - | 0.854 | 0.822 | 0.800 | 0.809 | 0.825 | 0.900 | - | 0.838 | 0.729 | 0.677 | - | - | - | 0.654 | - | - | 0.874 | 0.870 | 0.793 | 0.774 | 0.729 | 0.887 | 0.883 | 0.722 | 0.754 | 0.803 | 0.806 | 0.845 | 0.896 | 0.858 | highschoolchemistry | 0.216 | 0.522 | 0.507 | 0.586 | 0.600 | 0.167 | 0.359 | 0.517 | - | 0.610 | 0.596 | 0.546 | 0.517 | 0.527 | 0.689 | - | 0.566 | 0.467 | 0.433 | - | - | - | 0.310 | - | - | 0.650 | 0.640 | 0.512 | 0.492 | 0.482 | 0.655 | 0.699 | 0.413 | 0.463 | 0.532 | 0.536 | 0.596 | 0.724 | 0.684 | highschoolcomputerscience | 0.250 | 0.740 | 0.740 | 0.710 | 0.770 | 0.270 | 0.570 | 0.750 | - | 0.830 | 0.760 | 0.660 | 0.660 | 0.760 | 0.830 | - | 0.830 | 0.610 | 0.540 | - | - | - | 0.490 | - | - | 0.810 | 0.790 | 0.610 | 0.580 | 0.590 | 0.870 | 0.820 | 0.600 | 0.660 | 0.770 | 0.770 | 0.830 | 0.870 | 0.780 | highschooleuropeanhistory | 0.490 | 0.757 | 0.745 | 0.806 | 0.830 | 0.363 | 0.678 | 0.818 | - | 0.824 | 0.812 | 0.812 | 0.830 | 0.787 | 0.830 | - | 0.775 | 0.709 | 0.672 | - | - | - | 0.678 | - | - | 0.830 | 0.830 | 0.727 | 0.672 | 0.678 | 0.812 | 0.848 | 0.733 | 0.733 | 0.787 | 0.800 | 0.824 | 0.818 | 0.787 | highschoolgeography | 0.393 | 0.717 | 0.747 | 0.878 | 0.888 | 0.424 | 0.646 | 0.818 | - | 0.858 | 0.828 | 0.792 | 0.818 | 0.792 | 0.893 | - | 0.818 | 0.757 | 0.671 | - | - | - | 0.671 | - | - | 0.853 | 0.853 | 0.792 | 0.737 | 0.722 | 0.888 | 0.888 | 0.712 | 0.732 | 0.833 | 0.833 | 0.868 | 0.883 | 0.797 | highschoolgovernmentandpolitics | 0.487 | 0.875 | 0.875 | 0.926 | 0.963 | 0.450 | 0.772 | 0.911 | - | 0.937 | 0.860 | 0.875 | 0.870 | 0.880 | 0.968 | - | 0.891 | 0.818 | 0.725 | - | - | - | 0.805 | - | - | 0.937 | 0.948 | 0.849 | 0.834 | 0.839 | 0.937 | 0.937 | 0.772 | 0.797 | 0.917 | 0.917 | 0.958 | 0.968 | 0.860 | highschoolmacroeconomics | 0.235 | 0.653 | 0.687 | 0.717 | 0.758 | 0.238 | 0.474 | 0.682 | - | 0.771 | 0.687 | 0.651 | 0.653 | 0.733 | 0.820 | - | 0.682 | 0.556 | 0.497 | - | - | - | 0.478 | - | - | 0.766 | 0.743 | 0.646 | 0.635 | 0.617 | 0.807 | 0.805 | 0.564 | 0.592 | 0.684 | 0.684 | 0.802 | 0.825 | 0.715 | highschoolmathematics | 0.088 | 0.344 | 0.337 | 0.277 | 0.325 | 0.033 | 0.211 | 0.337 | - | 0.422 | 0.348 | 0.237 | 0.240 | 0.285 | 0.392 | - | 0.429 | 0.255 | 0.233 | - | - | - | 0.162 | - | - | 0.362 | 0.348 | 0.214 | 0.203 | 0.185 | 0.274 | 0.411 | 0.270 | 0.244 | 0.440 | 0.422 | 0.500 | 0.537 | 0.396 | highschoolmicroeconomics | 0.268 | 0.823 | 0.827 | 0.801 | 0.852 | 0.315 | 0.533 | 0.798 | - | 0.831 | 0.819 | 0.760 | 0.773 | 0.798 | 0.911 | - | 0.810 | 0.684 | 0.575 | - | - | - | 0.540 | - | - | 0.873 | 0.878 | 0.794 | 0.743 | 0.760 | 0.861 | 0.886 | 0.672 | 0.697 | 0.827 | 0.827 | 0.857 | 0.907 | 0.840 | highschoolphysics | 0.099 | 0.509 | 0.496 | 0.423 | 0.496 | 0.112 | 0.198 | 0.403 | - | 0.562 | 0.503 | 0.344 | 0.364 | 0.443 | 0.602 | - | 0.589 | 0.317 | 0.211 | - | - | - | 0.165 | - | - | 0.596 | 0.582 | 0.377 | 0.384 | 0.324 | 0.569 | 0.589 | 0.317 | 0.311 | 0.470 | 0.456 | 0.635 | 0.695 | 0.602 | highschoolpsychology | 0.445 | 0.827 | 0.853 | 0.896 | 0.910 | 0.445 | 0.750 | 0.882 | - | 0.900 | 0.880 | 0.840 | 0.858 | 0.862 | 0.917 | - | 0.867 | 0.834 | 0.761 | - | - | - | 0.764 | - | - | 0.913 | 0.900 | 0.855 | 0.844 | 0.823 | 0.904 | 0.913 | 0.803 | 0.796 | 0.858 | 0.856 | 0.882 | 0.902 | 0.860 | highschoolstatistics | 0.185 | 0.564 | 0.625 | 0.574 | 0.615 | 0.129 | 0.337 | 0.574 | - | 0.555 | 0.643 | 0.509 | 0.500 | 0.638 | 0.703 | - | 0.625 | 0.462 | 0.342 | - | - | - | 0.361 | - | - | 0.648 | 0.648 | 0.569 | 0.523 | 0.458 | 0.643 | 0.722 | 0.481 | 0.518 | 0.615 | 0.648 | 0.717 | 0.782 | 0.671 | highschoolushistory | 0.436 | 0.764 | 0.794 | 0.829 | 0.867 | 0.348 | 0.705 | 0.843 | - | 0.872 | 0.799 | 0.833 | 0.867 | 0.774 | 0.892 | - | 0.764 | 0.784 | 0.696 | - | - | - | 0.699 | - | - | 0.897 | 0.921 | 0.759 | 0.735 | 0.754 | 0.877 | 0.872 | 0.715 | 0.759 | 0.843 | 0.852 | 0.882 | 0.906 | 0.803 | highschoolworldhistory | 0.535 | 0.759 | 0.818 | 0.872 | 0.881 | 0.392 | 0.696 | 0.864 | - | 0.907 | 0.831 | 0.810 | 0.827 | 0.801 | 0.898 | - | 0.805 | 0.789 | 0.725 | - | - | - | 0.720 | - | - | 0.869 | 0.873 | 0.746 | 0.742 | 0.772 | 0.869 | 0.898 | 0.776 | 0.793 | 0.818 | 0.827 | 0.869 | 0.877 | 0.822 | humanaging | 0.309 | 0.596 | 0.627 | 0.690 | 0.739 | 0.345 | 0.524 | 0.645 | - | 0.699 | 0.636 | 0.582 | 0.591 | 0.641 | 0.713 | - | 0.663 | 0.618 | 0.569 | - | - | - | 0.542 | - | - | 0.713 | 0.704 | 0.582 | 0.547 | 0.587 | 0.726 | 0.730 | 0.569 | 0.587 | 0.681 | 0.690 | 0.717 | 0.771 | 0.641 | humansexuality | 0.351 | 0.648 | 0.694 | 0.746 | 0.755 | 0.358 | 0.519 | 0.740 | - | 0.763 | 0.709 | 0.648 | 0.633 | 0.648 | 0.793 | - | 0.694 | 0.671 | 0.587 | - | - | - | 0.569 | - | - | 0.839 | 0.816 | 0.664 | 0.587 | 0.603 | 0.740 | 0.816 | 0.625 | 0.625 | 0.740 | 0.717 | 0.786 | 0.839 | 0.664 | internationallaw | 0.404 | 0.727 | 0.776 | 0.801 | 0.760 | 0.495 | 0.677 | 0.801 | - | 0.809 | 0.752 | 0.735 | 0.752 | 0.743 | 0.859 | - | 0.760 | 0.776 | 0.710 | - | - | - | 0.710 | - | - | 0.834 | 0.818 | 0.735 | 0.727 | 0.727 | 0.892 | 0.867 | 0.710 | 0.685 | 0.768 | 0.785 | 0.834 | 0.867 | 0.752 | jurisprudence | 0.444 | 0.740 | 0.768 | 0.785 | 0.833 | 0.379 | 0.648 | 0.740 | - | 0.796 | 0.787 | 0.675 | 0.722 | 0.777 | 0.833 | - | 0.777 | 0.731 | 0.574 | - | - | - | 0.626 | - | - | 0.833 | 0.805 | 0.722 | 0.750 | 0.722 | 0.787 | 0.879 | 0.694 | 0.712 | 0.759 | 0.750 | 0.824 | 0.824 | 0.759 | logicalfallacies | 0.380 | 0.711 | 0.730 | 0.811 | 0.797 | 0.300 | 0.644 | 0.779 | - | 0.871 | 0.803 | 0.730 | 0.754 | 0.717 | 0.834 | - | 0.797 | 0.736 | 0.687 | - | - | - | 0.660 | - | - | 0.797 | 0.785 | 0.785 | 0.754 | 0.766 | 0.779 | 0.846 | 0.705 | 0.723 | 0.773 | 0.766 | 0.834 | 0.877 | 0.803 | machinelearning | 0.196 | 0.508 | 0.491 | 0.437 | 0.571 | 0.169 | 0.285 | 0.464 | - | 0.482 | 0.580 | 0.419 | 0.401 | 0.500 | 0.642 | - | 0.553 | 0.366 | 0.285 | - | - | - | 0.321 | - | - | 0.571 | 0.571 | 0.437 | 0.375 | 0.375 | 0.544 | 0.598 | 0.339 | 0.321 | 0.437 | 0.410 | 0.526 | 0.642 | 0.526 | management | 0.417 | 0.825 | 0.786 | 0.825 | 0.844 | 0.475 | 0.708 | 0.864 | - | 0.825 | 0.796 | 0.737 | 0.766 | 0.834 | 0.825 | - | 0.834 | 0.737 | 0.669 | - | - | - | 0.708 | - | - | 0.844 | 0.864 | 0.786 | 0.776 | 0.747 | 0.854 | 0.834 | 0.689 | 0.718 | 0.805 | 0.825 | 0.825 | 0.864 | 0.786 | marketing | 0.517 | 0.820 | 0.854 | 0.863 | 0.893 | 0.508 | 0.782 | 0.858 | - | 0.897 | 0.871 | 0.850 | 0.858 | 0.888 | 0.905 | - | 0.829 | 0.837 | 0.799 | - | - | - | 0.756 | - | - | 0.893 | 0.888 | 0.820 | 0.803 | 0.837 | 0.914 | 0.910 | 0.811 | 0.816 | 0.888 | 0.893 | 0.897 | 0.901 | 0.833 | medicalgenetics | 0.340 | 0.720 | 0.750 | 0.780 | 0.810 | 0.240 | 0.510 | 0.720 | - | 0.790 | 0.740 | 0.630 | 0.640 | 0.710 | 0.910 | - | 0.750 | 0.720 | 0.660 | - | - | - | 0.600 | - | - | 0.850 | 0.880 | 0.710 | 0.700 | 0.700 | 0.860 | 0.830 | 0.660 | 0.690 | 0.770 | 0.770 | 0.820 | 0.900 | 0.740 | miscellaneous | 0.420 | 0.749 | 0.768 | 0.830 | 0.854 | 0.401 | 0.687 | 0.825 | - | 0.879 | 0.782 | 0.775 | 0.796 | 0.768 | 0.860 | - | 0.790 | 0.773 | 0.736 | - | - | - | 0.727 | - | - | 0.872 | 0.872 | 0.777 | 0.759 | 0.734 | 0.864 | 0.869 | 0.724 | 0.726 | 0.807 | 0.814 | 0.871 | 0.885 | 0.782 | moraldisputes | 0.323 | 0.609 | 0.618 | 0.680 | 0.736 | 0.332 | 0.511 | 0.664 | - | 0.719 | 0.644 | 0.604 | 0.612 | 0.635 | 0.722 | - | 0.604 | 0.621 | 0.560 | - | - | - | 0.524 | - | - | 0.748 | 0.754 | 0.615 | 0.621 | 0.653 | 0.748 | 0.777 | 0.537 | 0.566 | 0.664 | 0.676 | 0.725 | 0.760 | 0.606 | moralscenarios | 0.115 | 0.165 | 0.411 | 0.325 | 0.366 | 0.117 | 0.143 | 0.207 | - | 0.489 | 0.293 | 0.307 | 0.360 | 0.188 | 0.460 | - | 0.241 | 0.205 | 0.410 | - | - | - | 0.122 | - | - | 0.482 | 0.377 | 0.366 | 0.404 | 0.270 | 0.582 | 0.655 | 0.130 | 0.058 | 0.318 | 0.368 | 0.546 | 0.565 | 0.268 | nutrition | 0.313 | 0.650 | 0.666 | 0.683 | 0.758 | 0.333 | 0.565 | 0.676 | - | 0.764 | 0.683 | 0.643 | 0.653 | 0.751 | 0.790 | - | 0.722 | 0.689 | 0.620 | - | - | - | 0.555 | - | - | 0.843 | 0.826 | 0.669 | 0.620 | 0.630 | 0.771 | 0.764 | 0.647 | 0.630 | 0.745 | 0.745 | 0.790 | 0.797 | 0.686 | philosophy | 0.327 | 0.681 | 0.675 | 0.658 | 0.713 | 0.363 | 0.536 | 0.726 | - | 0.742 | 0.662 | 0.652 | 0.659 | 0.688 | 0.800 | - | 0.630 | 0.617 | 0.578 | - | - | - | 0.587 | - | - | 0.736 | 0.781 | 0.630 | 0.588 | 0.630 | 0.784 | 0.778 | 0.562 | 0.565 | 0.675 | 0.688 | 0.774 | 0.778 | 0.649 | prehistory | 0.308 | 0.660 | 0.697 | 0.728 | 0.783 | 0.342 | 0.577 | 0.759 | - | 0.827 | 0.728 | 0.635 | 0.663 | 0.641 | 0.854 | - | 0.722 | 0.700 | 0.604 | - | - | - | 0.580 | - | - | 0.805 | 0.805 | 0.697 | 0.663 | 0.617 | 0.805 | 0.824 | 0.641 | 0.666 | 0.762 | 0.756 | 0.836 | 0.861 | 0.703 | professionalaccounting | 0.184 | 0.418 | 0.432 | 0.496 | 0.514 | 0.152 | 0.280 | 0.436 | - | 0.531 | 0.514 | 0.404 | 0.425 | 0.429 | 0.585 | - | 0.457 | 0.393 | 0.336 | - | - | - | 0.336 | - | - | 0.531 | 0.517 | 0.418 | 0.386 | 0.421 | 0.510 | 0.592 | 0.386 | 0.414 | 0.457 | 0.460 | 0.560 | 0.631 | 0.471 | professionallaw | 0.202 | 0.397 | 0.417 | 0.478 | 0.528 | 0.177 | 0.323 | 0.441 | - | 0.489 | 0.479 | 0.404 | 0.408 | 0.417 | 0.577 | - | 0.391 | 0.397 | 0.369 | - | - | - | 0.333 | - | - | 0.518 | 0.505 | 0.410 | 0.401 | 0.394 | 0.492 | 0.554 | 0.340 | 0.337 | 0.401 | 0.402 | 0.477 | 0.541 | 0.404 | professionalmedicine | 0.235 | 0.639 | 0.636 | 0.756 | 0.794 | 0.113 | 0.481 | 0.761 | - | 0.783 | 0.750 | 0.654 | 0.680 | 0.680 | 0.878 | - | 0.724 | 0.724 | 0.713 | - | - | - | 0.564 | - | - | 0.827 | 0.827 | 0.687 | 0.658 | 0.588 | 0.823 | 0.812 | 0.573 | 0.580 | 0.680 | 0.683 | 0.812 | 0.845 | 0.709 | professionalpsychology | 0.300 | 0.647 | 0.665 | 0.728 | 0.805 | 0.272 | 0.495 | 0.718 | - | 0.753 | 0.720 | 0.598 | 0.609 | 0.684 | 0.828 | - | 0.689 | 0.642 | 0.509 | - | - | - | 0.521 | - | - | 0.790 | 0.777 | 0.655 | 0.617 | 0.619 | 0.799 | 0.805 | 0.586 | 0.591 | 0.707 | 0.702 | 0.776 | 0.810 | 0.668 | publicrelations | 0.409 | 0.563 | 0.600 | 0.700 | 0.672 | 0.354 | 0.509 | 0.690 | - | 0.681 | 0.727 | 0.572 | 0.627 | 0.618 | 0.718 | - | 0.590 | 0.518 | 0.545 | - | - | - | 0.554 | - | - | 0.736 | 0.736 | 0.554 | 0.572 | 0.600 | 0.727 | 0.754 | 0.563 | 0.572 | 0.627 | 0.645 | 0.736 | 0.663 | 0.636 | securitystudies | 0.240 | 0.608 | 0.644 | 0.746 | 0.763 | 0.285 | 0.632 | 0.661 | - | 0.755 | 0.693 | 0.624 | 0.632 | 0.718 | 0.767 | - | 0.673 | 0.665 | 0.616 | - | - | - | 0.600 | - | - | 0.787 | 0.767 | 0.669 | 0.673 | 0.661 | 0.730 | 0.759 | 0.620 | 0.653 | 0.718 | 0.718 | 0.767 | 0.775 | 0.714 | sociology | 0.412 | 0.781 | 0.791 | 0.815 | 0.860 | 0.517 | 0.666 | 0.820 | - | 0.850 | 0.820 | 0.736 | 0.741 | 0.810 | 0.850 | - | 0.791 | 0.786 | 0.741 | - | - | - | 0.716 | - | - | 0.860 | 0.875 | 0.820 | 0.781 | 0.766 | 0.870 | 0.865 | 0.716 | 0.736 | 0.815 | 0.825 | 0.855 | 0.860 | 0.791 | usforeignpolicy | 0.510 | 0.780 | 0.790 | 0.868 | 0.840 | 0.460 | 0.740 | 0.890 | - | 0.860 | 0.750 | 0.780 | 0.800 | 0.840 | 0.880 | - | 0.800 | 0.800 | 0.800 | - | - | - | 0.757 | - | - | 0.920 | 0.910 | 0.760 | 0.770 | 0.770 | 0.890 | 0.880 | 0.750 | 0.780 | 0.820 | 0.820 | 0.890 | 0.880 | 0.770 | virology | 0.246 | 0.433 | 0.445 | 0.472 | 0.506 | 0.283 | 0.415 | 0.469 | - | 0.481 | 0.403 | 0.415 | 0.439 | 0.469 | 0.493 | - | 0.427 | 0.439 | 0.415 | - | - | - | 0.387 | - | - | 0.512 | 0.512 | 0.403 | 0.367 | 0.367 | 0.500 | 0.487 | 0.373 | 0.427 | 0.463 | 0.457 | 0.487 | 0.518 | 0.475 | worldreligions | 0.403 | 0.748 | 0.801 | 0.800 | 0.847 | 0.350 | 0.684 | 0.795 | - | 0.836 | 0.766 | 0.766 | 0.766 | 0.748 | 0.865 | - | 0.789 | 0.789 | 0.742 | - | - | - | 0.747 | - | - | 0.871 | 0.853 | 0.742 | 0.725 | 0.707 | 0.836 | 0.818 | 0.783 | 0.760 | 0.818 | 0.818 | 0.859 | 0.871 | 0.771 | MMLU | 0.285 | 0.591 | 0.623 | 0.647 | 0.687 | 0.269 | 0.477 | 0.631 | - | 0.701 | 0.643 | 0.580 | 0.595 | 0.617 | 0.734 | - | 0.629 | 0.570 | 0.525 | - | - | - | 0.486 | - | - | 0.717 | 0.704 | 0.599 | 0.578 | 0.560 | 0.710 | 0.737 | 0.532 | 0.540 | 0.639 | 0.643 | 0.721 | 0.757 | 0.639 | AGIEVAL aquarat | 0.374 | 0.602 | 0.562 | 0.665 | 0.602 | 0.409 | 0.763 | 0.846 | - | 0.844 | 0.884 | 0.653 | 0.637 | 0.783 | 0.856 | - | 0.884 | 0.598 | 0.633 | - | - | - | 0.279 | - | - | 0.516 | 0.764 | 0.409 | 0.574 | 0.724 | 0.834 | 0.580 | 0.732 | 0.728 | 0.799 | 0.830 | 0.822 | 0.870 | 0.896 | logiqa | 0.208 | 0.356 | 0.337 | 0.447 | 0.477 | 0.145 | 0.342 | 0.479 | - | 0.509 | 0.420 | 0.399 | 0.416 | 0.433 | 0.511 | - | 0.387 | 0.328 | 0.265 | - | - | - | 0.264 | - | - | 0.468 | 0.447 | 0.281 | 0.267 | 0.267 | 0.445 | 0.466 | 0.316 | 0.342 | 0.427 | 0.436 | 0.493 | 0.554 | 0.465 | lsatar | 0.213 | 0.213 | 0.282 | 0.208 | 0.260 | 0.217 | 0.213 | 0.365 | - | 0.317 | 0.747 | 0.073 | 0.217 | 0.308 | 0.800 | - | 0.673 | 0.295 | 0.239 | - | - | - | 0.186 | - | - | 0.269 | 0.639 | 0.256 | 0.247 | 0.234 | 0.369 | 0.269 | 0.230 | 0.226 | 0.260 | 0.300 | 0.321 | 0.400 | 0.791 | lsatlr | 0.203 | 0.486 | 0.537 | 0.635 | 0.654 | 0.115 | 0.374 | 0.596 | - | 0.686 | 0.607 | 0.505 | 0.515 | 0.592 | 0.776 | - | 0.662 | 0.441 | 0.327 | - | - | - | 0.366 | - | - | 0.709 | 0.686 | 0.415 | 0.386 | 0.401 | 0.621 | 0.660 | 0.452 | 0.449 | 0.598 | 0.603 | 0.729 | 0.811 | 0.686 | lsatrc | 0.312 | 0.594 | 0.646 | 0.750 | 0.754 | 0.208 | 0.475 | 0.702 | - | 0.717 | 0.728 | 0.635 | 0.643 | 0.706 | 0.825 | - | 0.669 | 0.624 | 0.486 | - | - | - | 0.520 | - | - | 0.814 | 0.806 | 0.531 | 0.524 | 0.557 | 0.762 | 0.758 | 0.553 | 0.617 | 0.661 | 0.687 | 0.810 | 0.836 | 0.706 | saten | 0.470 | 0.791 | 0.810 | 0.834 | 0.868 | 0.305 | 0.728 | 0.854 | - | 0.893 | 0.830 | 0.815 | 0.820 | 0.844 | 0.893 | - | 0.781 | 0.781 | 0.689 | - | - | - | 0.679 | - | - | 0.893 | 0.873 | 0.713 | 0.708 | 0.723 | 0.830 | 0.854 | 0.733 | 0.776 | 0.810 | 0.844 | 0.888 | 0.922 | 0.839 | satmath | 0.559 | 0.790 | 0.822 | 0.886 | 0.768 | 0.468 | 0.945 | 0.981 | - | 0.936 | 0.981 | 0.863 | 0.868 | 0.968 | 0.950 | - | 0.990 | 0.618 | 0.845 | - | - | - | 0.400 | - | - | 0.804 | 0.813 | 0.713 | 0.754 | 0.890 | 0.977 | 0.613 | 0.900 | 0.922 | 0.963 | 0.963 | 0.990 | 0.981 | 0.995 | AGIEVAL | 0.294 | 0.503 | 0.523 | 0.598 | 0.602 | 0.226 | 0.488 | 0.639 | - | 0.663 | 0.667 | 0.525 | 0.546 | 0.611 | 0.745 | - | 0.652 | 0.480 | 0.433 | - | - | - | 0.359 | - | - | 0.615 | 0.665 | 0.429 | 0.438 | 0.475 | 0.638 | 0.583 | 0.501 | 0.520 | 0.599 | 0.616 | 0.681 | 0.734 | 0.702 | AGIEVALCbiology | - | 0.365 | - | - | - | 0.104 | 0.334 | 0.595 | - | 0.665 | 0.634 | 0.756 | 0.778 | 0.869 | 0.847 | - | 0.856 | - | - | - | - | - | - | - | - | 0.721 | 0.739 | - | - | - | - | - | 0.660 | 0.700 | 0.804 | 0.813 | 0.834 | 0.582 | 0.826 | AGIEVALCchemistry | - | 0.269 | - | - | - | 0.078 | 0.289 | 0.446 | - | 0.480 | 0.426 | 0.642 | 0.691 | 0.715 | 0.691 | - | 0.710 | - | - | - | - | - | - | - | - | 0.509 | 0.509 | - | - | - | - | - | 0.441 | 0.470 | 0.583 | 0.627 | 0.696 | 0.789 | 0.691 | AGIEVALCchinese | - | 0.247 | - | - | - | 0.048 | 0.231 | 0.373 | - | 0.439 | 0.378 | 0.642 | 0.650 | 0.723 | 0.739 | - | 0.654 | - | - | - | - | - | - | - | - | 0.569 | 0.577 | - | - | - | - | - | 0.508 | 0.504 | 0.585 | 0.593 | 0.760 | 0.735 | 0.573 | AGIEVALCenglish | - | 0.774 | - | - | - | 0.444 | 0.728 | 0.862 | - | 0.866 | 0.862 | 0.823 | 0.833 | 0.905 | 0.908 | - | 0.843 | - | - | - | - | - | - | - | - | 0.892 | 0.892 | - | - | - | - | - | 0.794 | 0.839 | 0.856 | 0.849 | 0.915 | 0.924 | 0.849 | AGIEVALCgeography | - | 0.407 | - | - | - | 0.246 | 0.396 | 0.608 | - | 0.678 | 0.648 | 0.728 | 0.728 | 0.814 | 0.829 | - | 0.693 | - | - | - | - | - | - | - | - | 0.718 | 0.718 | - | - | - | - | - | 0.643 | 0.633 | 0.753 | 0.778 | 0.804 | 0.839 | 0.693 | AGIEVALChistory | - | 0.374 | - | - | - | 0.225 | 0.421 | 0.642 | - | 0.689 | 0.697 | 0.829 | 0.834 | 0.872 | 0.863 | - | 0.838 | - | - | - | - | - | - | - | - | 0.736 | 0.736 | - | - | - | - | - | 0.740 | 0.744 | 0.774 | 0.800 | 0.842 | 0.923 | 0.770 | AGIEVALCjecqaca | - | 0.221 | - | - | - | 0.142 | 0.258 | 0.292 | - | 0.348 | 0.354 | 0.414 | 0.440 | 0.660 | 0.544 | - | 0.454 | - | - | - | - | - | - | - | - | 0.416 | 0.410 | - | - | - | - | - | 0.425 | 0.424 | 0.482 | 0.487 | 0.564 | 0.622 | 0.409 | AGIEVALCjecqakd | - | 0.223 | - | - | - | 0.118 | 0.229 | 0.356 | - | 0.400 | 0.413 | 0.549 | 0.559 | 0.759 | 0.678 | - | 0.561 | - | - | - | - | - | - | - | - | 0.465 | 0.461 | - | - | - | - | - | 0.498 | 0.526 | 0.592 | 0.605 | 0.732 | 0.747 | 0.521 | AGIEVALClogiqa | - | 0.310 | - | - | - | 0.193 | 0.328 | 0.488 | - | 0.523 | 0.443 | 0.479 | 0.490 | 0.556 | 0.549 | - | 0.454 | - | - | - | - | - | - | - | - | 0.525 | 0.525 | - | - | - | - | - | 0.399 | 0.405 | 0.497 | 0.500 | 0.565 | 0.588 | 0.499 | AGIEVALCmathcloze | - | 0.508 | - | - | - | - | 0.567 | 0.779 | - | 0.855 | 0.855 | 0.491 | 0.542 | 0.508 | 0.949 | - | 0.957 | - | - | - | - | - | - | - | - | 0.754 | 0.915 | - | - | - | - | - | 0.508 | 0.440 | 0.694 | 0.686 | 0.737 | 0.805 | 0.949 | AGIEVALCmathqa | - | 0.569 | - | - | - | 0.322 | 0.616 | 0.779 | - | 0.744 | 0.872 | 0.621 | 0.648 | 0.845 | 0.888 | - | 0.892 | - | - | - | - | - | - | - | - | 0.664 | 0.844 | - | - | - | - | - | 0.595 | 0.683 | 0.779 | 0.755 | 0.808 | 0.834 | 0.932 | AGIEVALCphysics | - | 0.327 | - | - | - | 0.091 | 0.206 | 0.304 | - | 0.471 | 0.413 | 0.396 | 0.425 | 0.563 | 0.500 | - | 0.609 | - | - | - | - | - | - | - | - | 0.431 | 0.477 | - | - | - | - | - | 0.390 | 0.413 | 0.431 | 0.500 | 0.683 | 0.770 | 0.465 | AGIEVALC | - | 0.361 | - | - | - | 0.187 | 0.368 | 0.514 | - | 0.554 | 0.537 | 0.589 | 0.607 | 0.724 | 0.705 | - | 0.646 | - | - | - | - | - | - | - | - | 0.583 | 0.603 | - | - | - | - | - | 0.529 | 0.548 | 0.627 | 0.636 | 0.716 | 0.734 | 0.626 | BBH booleanexpressions | 0.544 | 0.860 | 0.876 | 0.768 | 0.460 | 0.632 | 0.880 | 0.880 | - | 0.732 | 0.824 | 0.848 | 0.868 | 0.800 | 0.896 | - | 0.760 | 0.844 | 0.480 | - | - | - | 0.764 | - | - | 0.872 | 0.860 | 0.852 | 0.832 | 0.776 | 0.936 | 0.932 | 0.756 | 0.796 | 0.864 | 0.880 | 0.888 | 0.808 | 0.700 | causaljudgement | 0.550 | 0.577 | 0.582 | 0.598 | 0.604 | 0.550 | 0.582 | 0.652 | - | 0.620 | 0.561 | 0.550 | 0.550 | 0.641 | 0.636 | - | 1.000 | 0.540 | 0.518 | - | - | - | 0.588 | - | - | 0.631 | 0.582 | 0.588 | 0.593 | 0.577 | 0.647 | 0.614 | 0.497 | 0.529 | 0.508 | 0.513 | 0.647 | 0.700 | 0.625 | dateunderstanding | 0.324 | 0.668 | 0.748 | 0.748 | 0.788 | 0.408 | 0.868 | 0.920 | - | 0.760 | 0.916 | 0.580 | 0.572 | 0.832 | 0.912 | - | 0.888 | 0.716 | 0.664 | - | - | - | 0.548 | - | - | 0.728 | 0.920 | 0.696 | 0.576 | 0.648 | 0.932 | 0.876 | 0.616 | 0.648 | 0.764 | 0.740 | 0.856 | 0.872 | 0.936 | disambiguationqa | 0.400 | 0.712 | 0.668 | 0.660 | 0.720 | 0.284 | 0.432 | 0.448 | - | 0.612 | 0.360 | 0.584 | 0.636 | 0.716 | 0.392 | - | 0.616 | 0.516 | 0.472 | - | - | - | 0.600 | - | - | 0.388 | 0.516 | 0.720 | 0.752 | 0.608 | 0.768 | 0.828 | 0.544 | 0.556 | 0.656 | 0.636 | 0.764 | 0.780 | 0.720 | dycklanguages | 0.424 | 0.704 | 0.712 | 0.728 | 0.600 | 0.344 | 0.636 | 0.824 | - | 0.892 | 0.496 | 0.516 | 0.544 | 0.592 | 0.628 | - | 0.756 | 0.796 | 0.680 | - | - | - | 0.744 | - | - | 0.792 | 0.684 | 0.580 | 0.468 | 0.656 | 0.776 | 0.736 | 0.596 | 0.628 | 0.868 | 0.836 | 0.648 | 0.820 | 0.540 | formalfallacies | 0.624 | 0.740 | 0.660 | 0.832 | 0.760 | 0.612 | 0.876 | 0.832 | - | 0.820 | 0.940 | 0.568 | 0.660 | 0.984 | 0.948 | - | 0.984 | 0.984 | 0.816 | - | - | - | 0.852 | - | - | 0.964 | 0.692 | 0.808 | 0.808 | 0.592 | 0.804 | 0.796 | 0.928 | 0.852 | 0.628 | 0.628 | 0.784 | 0.812 | 0.980 | geometricshapes | 0.056 | 0.544 | 0.456 | 0.436 | 0.420 | 0.128 | 0.376 | 0.456 | - | 0.544 | 0.724 | 0.392 | 0.400 | 0.812 | 0.664 | - | 0.804 | 0.440 | 0.416 | - | - | - | 0.288 | - | - | 0.280 | 0.716 | 0.416 | 0.292 | 0.316 | 0.648 | 0.676 | 0.204 | 0.212 | 0.544 | 0.604 | 0.584 | 0.640 | 0.812 | hyperbaton | 0.512 | 0.572 | 0.680 | 0.884 | 0.836 | 0.108 | 0.940 | 0.976 | - | 0.932 | 0.980 | 0.740 | 0.824 | 0.884 | 0.984 | - | 0.940 | 0.880 | 0.624 | - | - | - | 0.656 | - | - | 0.884 | 0.892 | 0.936 | 0.936 | 0.860 | 0.996 | 0.988 | 0.636 | 0.676 | 0.832 | 0.792 | 0.868 | 0.956 | 0.968 | logicaldeductionfiveobjects | 0.176 | 0.700 | 0.532 | 0.568 | 0.608 | 0.284 | 0.604 | 0.840 | - | 0.784 | 0.972 | 0.528 | 0.516 | 0.784 | 0.968 | - | 0.980 | 0.568 | 0.484 | - | - | - | 0.352 | - | - | 0.600 | 0.968 | 0.632 | 0.532 | 0.536 | 0.940 | 0.928 | 0.468 | 0.528 | 0.752 | 0.728 | 0.876 | 0.924 | 0.960 | logicaldeductionsevenobjects | 0.152 | 0.556 | 0.492 | 0.560 | 0.552 | 0.212 | 0.640 | 0.740 | - | 0.776 | 0.960 | 0.444 | 0.500 | 0.756 | 0.960 | - | 0.972 | 0.488 | 0.408 | - | - | - | 0.296 | - | - | 0.616 | 0.944 | 0.568 | 0.500 | 0.472 | 0.920 | 0.880 | 0.420 | 0.436 | 0.668 | 0.656 | 0.792 | 0.864 | 0.928 | logicaldeductionthreeobjects | 0.376 | 0.868 | 0.820 | 0.844 | 0.892 | 0.428 | 0.860 | 0.992 | - | 0.912 | 1.000 | 0.836 | 0.840 | 0.960 | 0.996 | - | 0.996 | 0.804 | 0.652 | - | - | - | 0.608 | - | - | 0.840 | 0.988 | 0.844 | 0.804 | 0.796 | 0.992 | 0.988 | 0.696 | 0.720 | 0.940 | 0.956 | 0.980 | 0.992 | 0.988 | movierecommendation | 0.424 | 0.652 | 0.676 | 0.552 | 0.508 | 0.372 | 0.536 | 0.664 | - | 0.632 | 0.680 | 0.604 | 0.648 | 0.740 | 0.768 | - | 0.600 | 0.536 | 0.456 | - | - | - | 0.508 | - | - | 0.684 | 0.672 | 0.520 | 0.508 | 0.528 | 0.992 | 0.632 | 0.604 | 0.568 | 0.556 | 0.536 | 0.672 | 0.648 | 0.496 | multisteparithmetictwo | 0.136 | 0.944 | 0.968 | 0.488 | 0.472 | - | 0.868 | 0.888 | - | 0.972 | 0.988 | 0.580 | 0.524 | 0.508 | 1.000 | - | 0.992 | 0.700 | 0.532 | - | - | - | 0.108 | - | - | 0.832 | 0.956 | 0.836 | 0.420 | 0.408 | 0.984 | 0.976 | 0.852 | 0.876 | 0.896 | 0.948 | 0.964 | 0.976 | 0.992 | navigate | 0.540 | 0.580 | 0.588 | 0.596 | 0.648 | 0.592 | 0.648 | 0.724 | - | 0.744 | 0.852 | 0.420 | 0.420 | 0.580 | 0.976 | - | 1.000 | 0.580 | 0.580 | - | - | - | 0.600 | - | - | 0.680 | 0.464 | 0.588 | 0.584 | 0.612 | 0.640 | 0.956 | 0.576 | 0.572 | 0.596 | 0.596 | 0.624 | 0.684 | 0.992 | objectcounting | 0.464 | 0.764 | 0.820 | 0.848 | 0.856 | - | 0.908 | 0.908 | - | 0.976 | 0.896 | 0.616 | 0.660 | 0.892 | 0.988 | - | 0.868 | 0.864 | 0.808 | - | - | - | 0.608 | - | - | 0.832 | 0.984 | 0.836 | 0.344 | 0.596 | 0.996 | 0.992 | 0.740 | 0.764 | 0.848 | 0.804 | 0.892 | 0.896 | 0.960 | penguinsinatable | 0.369 | 0.842 | 0.746 | 0.890 | 0.842 | 0.267 | 0.876 | 0.986 | - | 0.739 | 0.972 | 0.917 | 0.917 | 0.958 | 0.986 | - | 0.993 | 0.856 | 0.801 | - | - | - | 0.623 | - | - | 0.616 | 0.993 | 0.883 | 0.712 | 0.801 | 1.000 | 0.876 | 0.821 | 0.849 | 0.945 | 0.924 | 0.958 | 0.986 | 1.000 | reasoningaboutcoloredobjects | 0.276 | 0.860 | 0.800 | 0.744 | 0.900 | 0.180 | 0.752 | 0.888 | - | 0.844 | 0.988 | 0.876 | 0.796 | 0.940 | 0.992 | - | 0.976 | 0.824 | 0.568 | - | - | - | 0.608 | - | - | 0.804 | 0.992 | 0.808 | 0.656 | 0.776 | 0.968 | 0.956 | 0.700 | 0.764 | 0.904 | 0.868 | 0.944 | 0.984 | 0.996 | ruinnames | 0.176 | 0.484 | 0.636 | 0.716 | 0.760 | 0.172 | 0.468 | 0.696 | - | 0.816 | 0.804 | 0.696 | 0.652 | 0.716 | 0.764 | - | 0.760 | 0.744 | 0.532 | - | - | - | 0.400 | - | - | 0.764 | 0.748 | 0.612 | 0.600 | 0.588 | 0.816 | 0.784 | 0.396 | 0.324 | 0.440 | 0.544 | 0.692 | 0.760 | 0.720 | salienttranslationerrordetection | 0.212 | 0.448 | 0.508 | 0.548 | 0.568 | 0.172 | 0.560 | 0.640 | - | 0.564 | 0.692 | 0.476 | 0.488 | 0.580 | 0.672 | - | 0.652 | 0.512 | 0.464 | - | - | - | 0.444 | - | - | 0.600 | 0.656 | 0.520 | 0.532 | 0.540 | 0.636 | 0.372 | 0.452 | 0.432 | 0.560 | 0.572 | 0.612 | 0.700 | 0.716 | snarks | 0.483 | 0.685 | 0.707 | 0.691 | 0.719 | 0.033 | 0.724 | 0.803 | - | 0.634 | 0.814 | 0.702 | 0.707 | 0.769 | 0.876 | - | 0.747 | 0.651 | 0.657 | - | - | - | 0.606 | - | - | 0.634 | 0.786 | 0.747 | 0.786 | 0.814 | 0.882 | 0.876 | 0.662 | 0.623 | 0.747 | 0.780 | 0.831 | 0.865 | 0.842 | sportsunderstanding | 0.584 | 0.672 | 0.692 | 0.788 | 0.816 | 0.488 | 0.696 | 0.804 | - | 0.844 | 0.568 | 0.472 | 0.468 | 0.668 | 0.732 | - | 0.544 | 0.720 | 0.644 | - | - | - | 0.716 | - | - | 0.796 | 0.708 | 0.596 | 0.600 | 0.544 | 0.740 | 0.736 | 0.620 | 0.616 | 0.676 | 0.684 | 0.680 | 0.748 | 0.680 | temporalsequences | 0.164 | 0.528 | 0.540 | 0.708 | 0.748 | 0.436 | 0.988 | 0.996 | - | 0.940 | 0.996 | 0.756 | 0.840 | 0.956 | 0.996 | - | 1.000 | 0.856 | 0.712 | - | - | - | 0.404 | - | - | 0.844 | 0.992 | 0.784 | 0.508 | 0.768 | 1.000 | 0.996 | 0.324 | 0.388 | 0.800 | 0.820 | 0.988 | 0.992 | 0.992 | trackingshuffledobjectsfiveobjects | 0.208 | 0.560 | 0.616 | 0.600 | 0.692 | 0.508 | 0.924 | 1.000 | - | 0.648 | 0.996 | 0.544 | 0.536 | 0.864 | 0.988 | - | 0.988 | 0.656 | 0.500 | - | - | - | 0.344 | - | - | 0.716 | 0.992 | 0.940 | 0.712 | 0.852 | 1.000 | 0.988 | 0.420 | 0.452 | 0.840 | 0.908 | 0.924 | 0.972 | 0.988 | trackingshuffledobjectssevenobjects | 0.140 | 0.324 | 0.524 | 0.572 | 0.640 | 0.228 | 0.884 | 0.988 | - | 0.660 | 0.980 | 0.512 | 0.436 | 0.764 | 0.940 | - | 0.980 | 0.592 | 0.420 | - | - | - | 0.296 | - | - | 0.744 | 0.984 | 0.896 | 0.612 | 0.848 | 0.984 | 0.880 | 0.292 | 0.312 | 0.800 | 0.868 | 0.848 | 0.980 | 0.948 | trackingshuffledobjectsthreeobjects | 0.288 | 0.696 | 0.732 | 0.732 | 0.848 | 0.808 | 0.972 | 0.992 | - | 0.548 | 1.000 | 0.620 | 0.696 | 0.956 | 0.976 | - | 0.996 | 0.728 | 0.608 | - | - | - | 0.436 | - | - | 0.880 | 0.996 | 0.960 | 0.788 | 0.884 | 1.000 | 0.996 | 0.604 | 0.664 | 0.832 | 0.872 | 0.856 | 0.996 | 0.992 | weboflies | 0.476 | 0.576 | 0.520 | 0.520 | 0.488 | 0.488 | 0.516 | 0.540 | - | 0.532 | 0.988 | 0.476 | 0.488 | 0.512 | 0.996 | - | 1.000 | 0.512 | 0.544 | - | - | - | 0.488 | - | - | 0.560 | 0.504 | 0.488 | 0.492 | 0.512 | 0.512 | 0.964 | 0.512 | 0.512 | 0.528 | 0.532 | 0.544 | 0.624 | 1.000 | wordsorting | 0.056 | 0.204 | 0.292 | 0.404 | 0.540 | 0.080 | 0.236 | 0.424 | - | 0.536 | 0.280 | 0.404 | 0.392 | 0.144 | 0.300 | - | 0.316 | 0.512 | 0.360 | - | - | - | 0.280 | - | - | 0.632 | 0.592 | 0.204 | 0.152 | 0.140 | 0.360 | 0.392 | 0.156 | 0.156 | 0.212 | 0.220 | 0.292 | 0.400 | 0.272 | BBH | 0.334 | 0.638 | 0.650 | 0.664 | 0.674 | 0.355 | 0.711 | 0.794 | - | 0.743 | 0.823 | 0.596 | 0.608 | 0.749 | 0.849 | - | 0.853 | 0.681 | 0.566 | - | - | - | 0.506 | - | - | 0.714 | 0.806 | 0.696 | 0.592 | 0.627 | 0.846 | 0.838 | 0.554 | 0.567 | 0.709 | 0.718 | 0.775 | 0.827 | 0.841 | MUSR murdermystery | 0.552 | 0.640 | 0.592 | 0.668 | 0.576 | 0.528 | 0.592 | 0.608 | - | 0.552 | 0.672 | 0.616 | 0.584 | 0.620 | 0.712 | - | 0.640 | 0.584 | 0.576 | - | - | - | 0.516 | - | - | 0.712 | 0.680 | 0.636 | 0.620 | 0.588 | 0.708 | 0.680 | 0.544 | 0.612 | 0.604 | 0.584 | 0.652 | 0.640 | 0.636 | objectplacements | 0.429 | 0.535 | 0.578 | 0.519 | 0.542 | 0.296 | 0.480 | 0.542 | - | 0.448 | 0.472 | 0.492 | 0.531 | 0.460 | 0.520 | - | 0.580 | 0.546 | 0.523 | - | - | - | 0.453 | - | - | 0.516 | 0.532 | 0.503 | 0.457 | 0.453 | 0.464 | 0.404 | 0.472 | 0.476 | 0.531 | 0.554 | 0.519 | 0.265 | 0.532 | teamallocation | 0.436 | 0.512 | 0.496 | 0.460 | 0.476 | 0.328 | 0.400 | 0.560 | - | 0.572 | 0.692 | 0.572 | 0.588 | 0.448 | 0.736 | - | 0.640 | 0.460 | 0.396 | - | - | - | 0.356 | - | - | 0.612 | 0.576 | 0.536 | 0.480 | 0.508 | 0.628 | 0.556 | 0.444 | 0.384 | 0.512 | 0.476 | 0.556 | 0.592 | 0.708 | MUSR | 0.472 | 0.562 | 0.555 | 0.548 | 0.531 | 0.383 | 0.490 | 0.570 | - | 0.524 | 0.612 | 0.559 | 0.567 | 0.509 | 0.656 | - | 0.620 | 0.530 | 0.498 | - | - | - | 0.441 | - | - | 0.613 | 0.596 | 0.558 | 0.518 | 0.515 | 0.599 | 0.546 | 0.486 | 0.490 | 0.548 | 0.538 | 0.575 | 0.497 | 0.625 | GPQAdiamond | - | - | - | - | - | - | - | - | - | 0.388 | 0.545 | - | - | - | 0.666 | - | 0.520 | - | - | 0.479 | 0.469 | - | - | - | - | 0.358 | 0.540 | - | - | - | - | 0.545 | - | - | - | - | - | - | 0.570 | GPQA | - | - | - | - | - | - | - | - | - | 0.388 | 0.545 | - | - | - | 0.666 | - | 0.520 | - | - | 0.479 | 0.469 | - | - | - | - | 0.358 | 0.540 | - | - | - | - | 0.545 | - | - | - | - | - | - | 0.570 | MMLUPRO biology | 0.324 | 0.708 | 0.702 | 0.747 | 0.772 | 0.361 | 0.640 | 0.794 | - | 0.752 | 0.828 | 0.676 | 0.695 | 0.750 | 0.888 | - | 0.804 | 0.686 | 0.623 | - | - | - | 0.582 | - | - | 0.776 | 0.584 | 0.702 | 0.662 | 0.682 | 0.835 | 0.740| 0.610 | 0.638 | 0.709 | 0.729 | 0.797 | 0.764 | 0.852 | business | 0.190 | 0.624 | 0.525 | 0.583 | 0.626 | 0.173 | 0.518 | 0.659 | - | 0.616 | 0.744 | 0.522 | 0.562 | 0.628 | 0.836 | - | 0.760 | 0.558 | 0.458 | - | - | - | 0.335 | - | - | 0.612 | 0.756 | 0.571 | 0.509 | 0.588 | 0.785 | 0.756| 0.504 | 0.558 | 0.647 | 0.661 | 0.718 | 0.755 | 0.768 | chemistry | 0.166 | 0.639 | 0.500 | 0.503 | 0.546 | 0.115 | 0.380 | 0.574 | - | 0.536 | 0.788 | 0.465 | 0.467 | 0.589 | 0.816 | - | 0.844 | 0.467 | 0.390 | - | - | - | - | - | - | 0.488 | 0.728 | 0.463 | 0.296 | 0.513 | 0.765 | 0.700| 0.387 | 0.451 | 0.559 | 0.580 | 0.684 | 0.701 | 0.824 | computerscience | 0.197 | 0.602 | 0.590 | 0.482 | 0.560 | 0.170 | 0.421 | 0.643 | - | 0.560 | 0.772 | 0.497 | 0.502 | 0.585 | 0.824 | - | 0.756 | 0.485 | 0.414 | - | - | - | - | - | - | 0.556 | 0.680 | 0.475 | 0.448 | 0.456 | 0.734 | 0.708| 0.434 | 0.402 | 0.590 | 0.604 | 0.663 | 0.734 | 0.736 | economics | 0.236 | 0.663 | 0.662 | 0.668 | 0.678 | 0.206 | 0.534 | 0.699 | - | 0.660 | 0.804 | 0.617 | 0.610 | 0.662 | 0.844 | - | 0.744 | 0.568 | 0.492 | - | - | - | - | - | - | 0.648 | 0.612 | 0.609 | 0.587 | 0.629 | 0.792 | 0.716| 0.521 | 0.550 | 0.674 | 0.687 | 0.721 | 0.787 | 0.756 | engineering | 0.157 | 0.437 | 0.424 | 0.406 | 0.414 | 0.138 | 0.253 | 0.373 | - | 0.420 | 0.564 | 0.303 | 0.298 | 0.454 | 0.716 | - | 0.612 | 0.378 | 0.302 | - | - | - | - | - | - | 0.488 | 0.544 | 0.297 | 0.283 | 0.361 | 0.589 | 0.604| 0.296 | 0.309 | 0.418 | 0.420 | 0.512 | 0.573 | 0.668 | health | 0.158 | 0.503 | 0.517 | 0.545 | 0.621 | 0.156 | 0.399 | 0.596 | - | 0.548 | 0.636 | 0.492 | 0.496 | 0.544 | 0.736 | - | 0.660 | 0.558 | 0.437 | - | - | - | - | - | - | 0.544 | 0.616 | 0.515 | 0.466 | 0.506 | 0.700 | 0.632| 0.388 | 0.416 | 0.556 | 0.569 | 0.643 | 0.690 | 0.644 | history | 0.149 | 0.406 | 0.467 | 0.493 | 0.490 | 0.152 | 0.354 | 0.540 | - | 0.588 | 0.568 | 0.425 | 0.438 | 0.459 | 0.712 | - | 0.532 | 0.451 | 0.380 | - | - | - | - | - | - | 0.592 | 0.568 | 0.380 | 0.380 | 0.409 | 0.627 | 0.560| 0.333 | 0.367 | 0.459 | 0.464 | 0.566 | 0.624 | 0.568 | law | 0.123 | 0.268 | 0.295 | 0.343 | 0.405 | 0.158 | 0.263 | 0.372 | - | 0.400 | 0.452 | 0.299 | 0.284 | 0.307 | 0.620 | - | 0.404 | 0.303 | 0.243 | - | - | - | - | - | - | 0.384 | 0.328 | 0.276 | 0 | 0.294 | 0.500 | 0.408| 0.220 | 0.237 | 0.300 | 0.292 | 0.366 | 0.455 | 0.412 | math | 0.203 | 0.694 | 0.564 | 0.538 | 0.570 | 0.180 | 0.586 | 0.739 | - | 0.664 | 0.884 | 0.490 | 0.523 | 0.617 | 0.884 | - | 0.900 | 0.555 | 0.511 | - | - | - | - | - | - | 0.536 | 0.812 | 0.522 | 0.458 | 0.578 | 0.816 | 0.672| 0.581 | 0.603 | 0.712 | 0.723 | 0.775 | 0.814 | 0.884 | other | 0.164 | 0.450 | 0.496 | 0.551 | 0.574 | 0.173 | 0.428 | 0.580 | - | 0.552 | 0.600 | 0.464 | 0.458 | 0.536 | 0.724 | - | 0.568 | 0.487 | 0.389 | - | - | - | - | - | - | 0.484 | 0.592 | 0.500 | 0.433 | 0.493 | 0.706 | 0.648| 0.410 | 0.405 | 0.529 | 0.551 | 0.611 | 0.664 | 0.604 | philosophy | 0.148 | 0.442 | 0.462 | 0.448 | 0.488 | 0.176 | 0.356 | 0.555 | - | 0.560 | 0.572 | 0.408 | 0.412 | 0.424 | 0.692 | - | 0.576 | 0.382 | 0.326 | - | - | - | - | - | - | 0.476 | 0.580 | 0.406 | 0.390 | 0.394 | 0.633 | 0.604| 0.376 | 0.364 | 0.480 | 0.464 | 0.557 | 0.599 | 0.544 | physics | 0.159 | 0.583 | 0.493 | 0.501 | 0.559 | 0.125 | 0.397 | 0.595 | - | 0.512 | 0.788 | 0.441 | 0.461 | 0.587 | 0.852 | - | 0.808 | 0.488 | 0.397 | - | - | - | - | - | - | 0.488 | 0.724 | 0.455 | 0.425 | 0.500 | 0.765 | 0.748| 0.419 | 0.456 | 0.589 | 0.602 | 0.702 | 0.543 | 0.872 | psychology | 0.258 | 0.621 | 0.645 | 0.647 | 0.692 | 0.273 | 0.567 | 0.685 | - | 0.632 | 0.660 | 0.586 | 0.602 | 0.665 | 0.764 | - | 0.664 | 0.637 | 0.518 | - | - | - | - | - | - | 0.680 | 0.544 | 0.621 | 0.572 | 0.583 | 0.759 | 0.652| 0.526 | 0.563 | 0.636 | 0.644 | 0.721 | 0.749 | 0.684 | MMLUPRO | 0.186 | 0.552 | 0.517 | 0.528 | 0.568 | 0.177 | 0.436 | 0.597 | - | 0.571 | 0.690 | 0.471 | 0.480 | 0.559 | 0.779 | - | 0.688 | 0.499 | 0.419 | - | - | - | 0.453 | - | - | 0.553 | 0.619 | 0.482 | 0.408 | 0.502 | 0.719 | 0.653| 0.430 | 0.457 | 0.564 | 0.575 | 0.649 | 0.671 | 0.701 | CATEGORIES REASONING | 0.367 | 0.713 | 0.738 | 0.788 | 0.814 | 0.344 | 0.598 | 0.787 | 0.767 | 0.809 | 0.766 | 0.804 | 0.811 | 0.815 | 0.879 | - | 0.746 | 0.713 | 0.606 | - | - | - | 0.628 | 0.866 | 0.877 | 0.863 | 0.848 | 0.724 | 0.691 | 0.619 | 0.809 | 0.822 | 0.689 | 0.719 | 0.805 | 0.809 | 0.850 | 0.874 | 0.785 | UNDERSTANDING | 0.366 | 0.644 | 0.670 | 0.707 | 0.742 | 0.327 | 0.552 | 0.695 | 0.741 | 0.746 | 0.698 | 0.661 | 0.670 | 0.691 | 0.774 | - | 0.679 | 0.631 | 0.579 | - | - | - | 0.563 | 0.775 | 0.772 | 0.764 | 0.756 | 0.614 | 0.622 | 0.617 | 0.728 | 0.766 | 0.605 | 0.613 | 0.692 | 0.696 | 0.761 | 0.793 | 0.695 | LANGUAGE | 0.524 | 0.688 | 0.692 | 0.735 | 0.755 | 0.504 | 0.635 | 0.724 | 0.721 | 0.742 | 0.660 | 0.786 | 0.783 | 0.662 | 0.597 | - | 0.617 | 0.747 | 0.705 | - | - | - | 0.766 | 0.786 | 0.789 | 0.798 | 0.792 | 0.677 | 0.613 | 0.653 | 0.750 | 0.751 | 0.685 | 0.682 | 0.722 | 0.724 | 0.769 | 0.781 | 0.670 | KNOWLEDGE | 0.354 | 0.442 | 0.496 | 0.690 | 0.733 | 0.353 | 0.478 | 0.626 | 0.560 | 0.630 | 0.603 | 0.553 | 0.543 | 0.615 | 0.682 | - | 0.597 | 0.547 | 0.536 | 0.680 | 0.580 | 0.540 | 0.582 | 0.680 | - | 0.676 | 0.653 | 0.517 | 0.519 | 0.534 | 0.676 | 0.686 | 0.469 | 0.426 | 0.595 | 0.597 | 0.693 | 0.725 | 0.622 | COT | 0.220 | 0.552 | 0.530 | 0.550 | 0.582 | 0.201 | 0.470 | 0.616 | - | 0.630 | 0.685 | 0.485 | 0.500 | 0.586 | 0.749 | - | 0.706 | 0.530 | 0.446 | 0.479 | 0.469 | - | 0.498 | - | - | 0.600 | 0.651 | 0.506 | 0.440 | 0.513 | 0.725 | 0.669| 0.443 | 0.462 | 0.570 | 0.581 | 0.653 | 0.684 | 0.708 | MATHCOT | 0.369 | 0.730 | 0.752 | 0.735 | 0.740 | 0.417 | 0.793 | 0.879 | 0.882 | 0.790 | 0.934 | 0.682 | 0.679 | 0.813 | 0.953 | - | 0.945 | 0.728 | 0.647 | 0.840 | 0.860 | 0.860 | 0.493 | 0.830 | 0.780 | 0.743 | 0.908 | 0.767 | 0.638 | 0.745 | 0.919 | 0.899| 0.667 | 0.694 | 0.823 | 0.821 | 0.869 | 0.903 | 0.946 | CODE | 0.176 | 0.460 | 0.534 | 0.495 | 0.568 | 0.241 | 0.485 | 0.582 | 0.829 | 0.618 | 0.695 | 0.456 | 0.475 | 0.500 | 0.584 | - | 0.587 | 0.463 | 0.366 | - | - | - | 0.321 | 0.841 | 0.823 | 0.409 | 0.619 | 0.427 | 0.376 | 0.418 | 0.568 | 0.567 | 0.437 | 0.445 | 0.510 | 0.528 | 0.578 | 0.612 | 0.597 | DISCIPLINES NLP | 0.408 | 0.647 | 0.670 | 0.755 | 0.786 | 0.392 | 0.595 | 0.748 | 0.751 | 0.761 | 0.722 | 0.729 | 0.728 | 0.737 | 0.784 | - | 0.692 | 0.677 | 0.609 | - | - | - | 0.642 | 0.834 | 0.841 | 0.808 | 0.792 | 0.647 | 0.637 | 0.614 | 0.755 | 0.776 | 0.632 | 0.630 | 0.731 | 0.734 | 0.791 | 0.818 | 0.733 | MATH | 0.294 | 0.669 | 0.659 | 0.637 | 0.653 | 0.298 | 0.659 | 0.775 | 0.882 | 0.727 | 0.809 | 0.590 | 0.597 | 0.720 | 0.842 | - | 0.844 | 0.629 | 0.556 | 0.840 | 0.860 | 0.860 | 0.451 | 0.830 | 0.780 | 0.678 | 0.789 | 0.646 | 0.543 | 0.625 | 0.817 | 0.795| 0.576 | 0.599 | 0.741 | 0.742 | 0.799 | 0.843 | 0.831 | SCIENCE | 0.350 | 0.706 | 0.713 | 0.739 | 0.769 | 0.304 | 0.580 | 0.737 | - | 0.797 | 0.797 | 0.686 | 0.698 | 0.756 | 0.863 | - | 0.823 | 0.676 | 0.605 | 0.479 | 0.469 | - | 0.673 | - | - | 0.806 | 0.821 | 0.696 | 0.660 | 0.681 | 0.845 | 0.858| 0.629 | 0.657 | 0.738 | 0.748 | 0.815 | 0.806 | 0.830 | ENGINEERING | 0.166 | 0.464 | 0.453 | 0.426 | 0.438 | 0.158 | 0.271 | 0.397 | - | 0.496 | 0.572 | 0.334 | 0.333 | 0.480 | 0.706 | - | 0.630 | 0.397 | 0.323 | - | - | - | 0.393 | - | - | 0.567 | 0.587 | 0.323 | 0.308 | 0.388 | 0.595 | 0.630| 0.315 | 0.325 | 0.443 | 0.444 | 0.530 | 0.590 | 0.655 | MEDICINE | 0.216 | 0.524 | 0.540 | 0.595 | 0.648 | 0.182 | 0.411 | 0.598 | - | 0.642 | 0.606 | 0.521 | 0.530 | 0.570 | 0.721 | - | 0.593 | 0.577 | 0.496 | - | - | - | 0.447 | - | - | 0.681 | 0.684 | 0.537 | 0.501 | 0.490 | 0.672 | 0.684| 0.459 | 0.478 | 0.574 | 0.580 | 0.655 | 0.702 | 0.595 | HUMANITIES | 0.291 | 0.550 | 0.615 | 0.645 | 0.679 | 0.272 | 0.495 | 0.641 | 0.560 | 0.705 | 0.638 | 0.593 | 0.610 | 0.622 | 0.743 | - | 0.628 | 0.578 | 0.529 | 0.680 | 0.580 | 0.540 | 0.536 | 0.680 | - | 0.710 | 0.698 | 0.588 | 0.567 | 0.562 | 0.739 | 0.745| 0.527 | 0.533 | 0.629 | 0.638 | 0.716 | 0.742 | 0.634 | BUSINESS | 0.252 | 0.679 | 0.655 | 0.678 | 0.709 | 0.245 | 0.537 | 0.704 | - | 0.743 | 0.754 | 0.623 | 0.637 | 0.696 | 0.835 | - | 0.745 | 0.598 | 0.517 | - | - | - | 0.466 | - | - | 0.749 | 0.762 | 0.637 | 0.604 | 0.635 | 0.801 | 0.792| 0.565 | 0.596 | 0.701 | 0.710 | 0.759 | 0.802 | 0.759 | LAW | 0.200 | 0.362 | 0.427 | 0.483 | 0.524 | 0.172 | 0.316 | 0.443 | - | 0.504 | 0.517 | 0.417 | 0.429 | 0.494 | 0.652 | - | 0.507 | 0.406 | 0.344 | - | - | - | 0.370 | - | - | 0.537 | 0.543 | 0.392 | 0.310 | 0.390 | 0.541 | 0.582| 0.374 | 0.383 | 0.451 | 0.456 | 0.541 | 0.604 | 0.514 | COMPOSITE AVERAGE AVG | 0.342 | 0.612 | 0.641 | 0.692 | 0.724 | 0.324 | 0.555 | 0.701 | 0.753 | 0.729 | 0.709 | 0.648 | 0.654 | 0.689 | 0.775 | - | 0.692 | 0.629 | 0.561 | 0.620 | 0.595 | 0.700 | 0.578 | 0.833 | 0.841 | 0.740 | 0.757 | 0.616 | 0.585 | 0.591 | 0.748 | 0.760| 0.578 | 0.586 | 0.686 | 0.691 | 0.754 | 0.783 | 0.716_ |

THINKING MODEL TESTS:

MODEL	Qwen3-0.6B	Qwen3-1.7B	Qwen3-4B	Qwen3-4B	Qwen3-4B-Thinking-2507	Qwen3-8B	Qwen3-8B	Qwen3-8B	Qwen3-14B	Qwen3-14B	Qwen3-30B-A3B	Qwen3-30B-A3B	Qwen3-32B	Qwen3-32B	QwQ-32B-Preview	QwQ-32B	Ring-mini-2.0
params	0.75163B	2.03B	4.02B	4.02B	4.02B	8.19B	8.19B	8.19B	14.77B	14.77B	30.53B	30.53B	32.8B	32.8B	32.76B	32.76B	16.26B
quant	Q8_0	Q8_0	Q8_0	Q80H	Q6KH	Q4KH	Q6KH	Q6_K	IQ4_XS	Q4KH	IQ4_XS	Q4KH	IQ4_XS	Q4KH	IQ4_XS	Q4KH	Q6KH
engine	llama.cpp version: 5679	llama.cpp version: 5415	llama.cpp version: 5242	llama.cpp version: 5509	llama.cpp version: 6653	llama.cpp version: 5279	llama.cpp version: 5223	llama.cpp version: 5153	llama.cpp version: 5223	llama.cpp version: 5379	llama.cpp version: 5279	llama.cpp version: 5353	llama.cpp version: 5466	llama.cpp version: 5466	llama.cpp version: 4273	llama.cpp version: 6118	llama.cpp version: 6815
TEST	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc
WG	0.564	0.610	0.662	0.642	0.662	0.651	0.689	0.678	0.722	0.726	0.699	0.700	0.712	0.731	0.750	0.689	0.606
LAMBADA	0.471	0.590	0.644	0.638	0.662	0.660	0.700	0.700	0.729	0.714	0.698	0.725	0.692	0.701	0.780	0.651	0.522
HELLASWAG	0.277	0.553	0.721	0.726	0.722	0.718	0.787	0.787	0.827	0.792	0.815	0.832	0.838	0.812	0.875	0.883	0.689
BOOLQ	0.449	0.531	0.626	0.626	0.485	0.641	0.608	0.611	0.662	0.632	-	0.502	0.603	0.574	0.629	0.658	0.556
STORYCLOZE	0.764	0.833	0.849	0.843	0.809	0.809	0.852	0.873	-	0.905	-	0.917	0.960	0.959	0.964	0.935	0.937
CSQA	0.377	0.567	0.705	0.705	0.704	0.685	0.740	0.748	-	0.749	-	0.742	-	0.778	0.796	0.680	0.724
OBQA	0.394	0.584	0.756	0.754	0.729	0.719	0.767	0.774	-	0.787	-	0.836	-	0.869	0.882	0.815	0.764
COPA	0.569	0.765	0.872	0.865	0.801	0.829	0.828	0.864	-	0.919	-	0.919	-	0.946	0.936	0.962	0.877
PIQA	0.393	0.574	0.710	0.710	0.685	0.744	0.769	0.781	-	0.798	-	0.845	-	0.815	0.829	0.871	0.755
SIQA	0.363	0.569	0.664	0.664	0.640	0.637	0.671	0.679	-	0.689	-	0.693	-	0.714	0.714	0.747	0.664
MEDQA	0.135	0.278	0.435	0.428	0.450	0.448	0.499	0.509	-	0.531	-	0.597	-	0.553	0.598	0.518	0.450
SQA	0.249	0.032	0.039	0.040	0.147	0.036	0.039	0.042	-	0.045	-	0.055	-	0.047	-	0.060	0.022
JEOPARDY	0.640	0.270	0.280	0.220	0.400	0.410	0.280	0.240	0.480	0.490	0.520	0.470	0.470	0.470	0.600	0.440	0.030
GSM8K	0.748	0.920	0.946	0.960	0.960	0.946	0.953	0.956	-	0.948	-	0.962	-	0.972	0.962	0.964	0.952
APPLE	0.460	0.790	0.850	0.840	0.880	0.790	0.880	0.890	0.910	0.920	0.820	0.850	0.910	0.910	0.870	0.880	0.850
HUMANEVAL	0.445	0.682	0.817	0.804	0.786	0.835	0.865	0.859	-	0.859	-	0.884	-	0.890	0.414	0.512	0.853
HUMANEVALP	0.335	0.591	0.682	0.676	0.670	0.725	0.713	0.731	-	0.737	-	0.750	-	0.780	0.359	0.432	0.750
HUMANEVALFIM	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
MBPP	0.408	0.544	0.645	0.642	0.536	0.571	0.618	0.630	-	0.700	-	0.677	-	0.684	0.404	0.568	0.665
MBPPP	0.388	0.482	0.598	0.602	0.517	0.580	0.611	0.566	-	0.651	-	-	-	0.678	0.392	0.584	0.642
HUMANEVALX_cpp	0.231	0.359	0.463	0.353	0.646	0.524	0.615	0.554	-	0.652	-	-	-	0.737	0.378	0.603	0.750
HUMANEVALX_java	0.274	0.548	0.731	0.737	0.774	0.737	0.780	0.798	-	0.841	-	-	-	0.847	0.097	0.280	0.835
HUMANEVALX_js	0.256	0.518	0.719	0.695	0.439	0.762	0.774	0.774	-	0.786	-	-	-	0.817	0.493	0.493	0.786
HUMANEVALX	0.254	0.475	0.638	0.595	0.619	0.674	0.723	0.709	-	0.760	-	-	-	0.800	0.323	0.459	0.790
CRUXEVAL_input	0.353	0.406	0.457	0.453	0.500	0.445	0.528	0.510	-	0.537	-	-	-	0.450	0.200	0.498	0.398
CRUXEVAL_output	0.241	0.338	0.420	0.403	0.440	0.405	0.446	0.447	-	0.501	-	-	-	0.431	0.368	0.513	0.391
CRUXEVAL	0.297	0.372	0.438	0.428	0.470	0.425	0.487	0.478	-	0.519	-	-	-	0.440	0.284	0.506	0.395
CRUXEVALFIM_input	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
CRUXEVALFIM_output	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
CRUXEVALFIM	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
TQA_mc	0.261	0.406	0.600	0.598	0.656	0.592	0.641	0.635	-	0.676	-	-	-	0.742	0.795	0.701	0.608
TQA_tf	0.429	0.445	0.502	0.500	0.519	0.513	0.430	0.458	-	0.614	-	-	-	0.456	0.523	0.628	0.513
TQA	0.409	0.441	0.514	0.511	0.535	0.523	0.455	0.479	-	0.621	-	-	-	0.490	0.554	0.637	0.524
ARC_challenge	0.275	0.686	0.854	0.852	0.850	0.833	0.882	0.882	-	0.896	-	-	-	0.910	0.917	0.843	0.824
ARC_easy	0.502	0.850	0.937	0.933	0.948	0.934	0.952	0.955	-	0.964	-	-	-	0.974	0.975	0.906	0.937
ARC	0.427	0.796	0.910	0.906	0.916	0.901	0.929	0.931	-	0.942	-	-	-	0.953	0.956	0.886	0.900
RACE_high	0.359	0.594	0.759	0.756	0.784	0.747	0.794	0.798	-	0.826	-	-	-	0.822	0.871	0.862	0.748
RACE_middle	0.397	0.652	0.808	0.808	0.830	0.818	0.842	0.844	-	0.873	-	-	-	0.881	-	0.889	0.802
RACE	0.370	0.611	0.774	0.771	0.797	0.768	0.808	0.811	-	0.839	-	-	-	0.839	0.871	0.870	0.764

MMLU abstractalgebra | 0.100 | 0.240 | 0.410 | 0.420 | 0.460 | 0.360 | 0.430 | 0.470 | - | 0.500 | - | - | - | 0.470 | - | 0.470 | 0.380 | anatomy | 0.274 | 0.437 | 0.540 | 0.555 | 0.614 | 0.592 | 0.622 | 0.607 | - | 0.651 | - | - | - | 0.644 | - | 0.681 | 0.622 | astronomy | 0.328 | 0.611 | 0.723 | 0.730 | 0.750 | 0.796 | 0.822 | 0.828 | - | 0.861 | - | - | - | 0.868 | - | 0.802 | 0.750 | businessethics | 0.290 | 0.460 | 0.670 | 0.670 | 0.630 | 0.610 | 0.650 | 0.650 | - | 0.720 | - | - | - | 0.730 | - | 0.720 | 0.630 | clinicalknowledge | 0.350 | 0.528 | 0.690 | 0.709 | 0.686 | 0.728 | 0.758 | 0.743 | - | 0.762 | - | - | - | 0.781 | - | 0.766 | 0.694 | collegebiology | 0.388 | 0.604 | 0.770 | 0.784 | 0.805 | 0.784 | 0.805 | 0.812 | - | 0.847 | - | - | - | 0.881 | - | 0.819 | 0.694 | collegechemistry | 0.230 | 0.340 | 0.420 | 0.420 | 0.460 | 0.420 | 0.480 | 0.490 | - | 0.550 | - | - | - | 0.500 | - | 0.460 | 0.490 | collegecomputerscience | 0.230 | 0.380 | 0.580 | 0.580 | 0.620 | 0.610 | 0.650 | 0.700 | - | 0.630 | - | - | - | 0.700 | - | 0.580 | 0.470 | collegemathematics | 0.160 | 0.300 | 0.370 | 0.390 | 0.360 | 0.390 | 0.450 | 0.500 | - | 0.450 | - | - | - | 0.400 | - | 0.350 | 0.390 | collegemedicine | 0.312 | 0.537 | 0.676 | 0.664 | 0.664 | 0.676 | 0.716 | 0.722 | - | 0.739 | - | - | - | 0.722 | - | 0.722 | 0.595 | collegephysics | 0.137 | 0.254 | 0.519 | 0.509 | 0.460 | 0.500 | 0.529 | 0.578 | - | 0.578 | - | - | - | 0.558 | - | 0.421 | 0.441 | computersecurity | 0.430 | 0.650 | 0.690 | 0.700 | 0.710 | 0.710 | 0.750 | 0.740 | - | 0.770 | - | - | - | 0.770 | - | 0.630 | 0.640 | conceptualphysics | 0.255 | 0.514 | 0.685 | 0.685 | 0.702 | 0.697 | 0.736 | 0.761 | - | 0.821 | - | - | - | 0.821 | - | 0.668 | 0.668 | econometrics | 0.140 | 0.394 | 0.587 | 0.614 | 0.622 | 0.552 | 0.614 | 0.631 | - | 0.622 | - | - | - | 0.605 | - | 0.543 | 0.447 | electricalengineering | 0.324 | 0.475 | 0.586 | 0.579 | 0.579 | 0.586 | 0.648 | 0.648 | - | 0.717 | - | - | - | 0.724 | - | 0.517 | 0.558 | elementarymathematics | 0.148 | 0.391 | 0.582 | 0.574 | 0.589 | 0.584 | 0.640 | 0.650 | - | 0.701 | - | - | - | 0.679 | - | 0.547 | 0.476 | formallogic | 0.238 | 0.357 | 0.547 | 0.531 | 0.484 | 0.460 | 0.476 | 0.476 | - | 0.515 | - | - | - | 0.579 | - | 0.611 | 0.436 | globalfacts | 0.140 | 0.110 | 0.180 | 0.220 | 0.180 | 0.240 | 0.260 | 0.280 | - | 0.300 | - | - | - | 0.310 | - | 0.430 | 0.200 | highschoolbiology | 0.432 | 0.600 | 0.822 | 0.822 | 0.832 | 0.822 | 0.848 | 0.858 | - | 0.883 | - | - | - | 0.906 | - | 0.783 | 0.790 | highschoolchemistry | 0.147 | 0.423 | 0.600 | 0.591 | 0.625 | 0.551 | 0.635 | 0.650 | - | 0.709 | - | - | - | 0.665 | - | 0.596 | 0.581 | highschoolcomputerscience | 0.340 | 0.580 | 0.750 | 0.750 | 0.760 | 0.730 | 0.820 | 0.830 | - | 0.830 | - | - | - | 0.810 | - | 0.800 | 0.700 | highschooleuropeanhistory | 0.387 | 0.600 | 0.703 | 0.690 | 0.781 | 0.727 | 0.818 | 0.812 | - | 0.787 | - | - | - | 0.806 | - | 0.824 | 0.696 | highschoolgeography | 0.393 | 0.631 | 0.803 | 0.808 | 0.792 | 0.752 | 0.787 | 0.808 | - | 0.853 | - | - | - | 0.878 | - | 0.858 | 0.797 | highschoolgovernmentandpolitics | 0.284 | 0.621 | 0.849 | 0.854 | 0.834 | 0.834 | 0.891 | 0.906 | - | 0.901 | - | - | - | 0.958 | - | 0.886 | 0.823 | highschoolmacroeconomics | 0.292 | 0.474 | 0.661 | 0.656 | 0.653 | 0.669 | 0.712 | 0.712 | - | 0.787 | - | - | - | 0.810 | - | 0.692 | 0.623 | highschoolmathematics | 0.166 | 0.292 | 0.348 | 0.355 | 0.403 | 0.351 | 0.418 | 0.396 | - | 0.474 | - | - | - | 0.381 | - | 0.296 | 0.355 | highschoolmicroeconomics | 0.390 | 0.609 | 0.773 | 0.768 | 0.798 | 0.802 | 0.882 | 0.886 | - | 0.911 | - | - | - | 0.894 | - | 0.676 | 0.735 | highschoolphysics | 0.119 | 0.350 | 0.556 | 0.549 | 0.529 | 0.549 | 0.602 | 0.602 | - | 0.649 | - | - | - | 0.662 | - | 0.456 | 0.437 | highschoolpsychology | 0.526 | 0.746 | 0.842 | 0.844 | 0.849 | 0.849 | 0.877 | 0.877 | - | 0.891 | - | - | - | 0.921 | - | 0.790 | 0.836 | highschoolstatistics | 0.333 | 0.462 | 0.648 | 0.657 | 0.657 | 0.625 | 0.694 | 0.689 | - | 0.703 | - | - | - | 0.736 | - | 0.537 | 0.550 | highschoolushistory | 0.338 | 0.553 | 0.784 | 0.764 | 0.759 | 0.710 | 0.823 | 0.848 | - | 0.862 | - | - | - | 0.897 | - | 0.872 | 0.754 | highschoolworldhistory | 0.459 | 0.645 | 0.793 | 0.776 | 0.789 | 0.797 | 0.839 | 0.831 | - | 0.827 | - | - | - | 0.864 | - | 0.877 | 0.725 | humanaging | 0.331 | 0.439 | 0.587 | 0.578 | 0.596 | 0.600 | 0.609 | 0.623 | - | 0.668 | - | - | - | 0.771 | - | 0.721 | 0.551 | humansexuality | 0.374 | 0.549 | 0.641 | 0.664 | 0.679 | 0.664 | 0.740 | 0.755 | - | 0.770 | - | - | - | 0.809 | - | 0.770 | 0.656 | internationallaw | 0.429 | 0.537 | 0.669 | 0.652 | 0.694 | 0.628 | 0.694 | 0.710 | - | 0.826 | - | - | - | 0.801 | - | 0.809 | 0.743 | jurisprudence | 0.398 | 0.527 | 0.675 | 0.694 | 0.759 | 0.675 | 0.731 | 0.731 | - | 0.805 | - | - | - | 0.777 | - | 0.787 | 0.675 | logicalfallacies | 0.319 | 0.613 | 0.791 | 0.785 | 0.791 | 0.717 | 0.779 | 0.803 | - | 0.822 | - | - | - | 0.803 | - | 0.644 | 0.723 | machinelearning | 0.276 | 0.339 | 0.526 | 0.508 | 0.500 | 0.392 | 0.491 | 0.455 | - | 0.562 | - | - | - | 0.455 | - | 0.616 | 0.553 | management | 0.514 | 0.640 | 0.786 | 0.805 | 0.805 | 0.834 | 0.844 | 0.873 | - | 0.825 | - | - | - | 0.796 | - | 0.718 | 0.834 | marketing | 0.602 | 0.739 | 0.816 | 0.807 | 0.837 | 0.820 | 0.884 | 0.876 | - | 0.876 | - | - | - | 0.876 | - | 0.773 | 0.794 | medicalgenetics | 0.370 | 0.580 | 0.750 | 0.710 | 0.810 | 0.750 | 0.780 | 0.750 | - | 0.790 | - | - | - | 0.840 | - | 0.750 | 0.750 | miscellaneous | 0.390 | 0.597 | 0.752 | 0.744 | 0.761 | 0.757 | 0.789 | 0.795 | - | 0.831 | - | - | - | 0.854 | - | 0.786 | 0.740 | moraldisputes | 0.289 | 0.473 | 0.580 | 0.583 | 0.578 | 0.540 | 0.609 | 0.615 | - | 0.638 | - | - | - | 0.699 | - | 0.760 | 0.589 | moralscenarios | 0.109 | 0 | 0.145 | 0.140 | 0.230 | 0.234 | 0.322 | 0.269 | - | 0.330 | - | - | - | 0.292 | - | 0.518 | 0.082 | nutrition | 0.316 | 0.509 | 0.660 | 0.669 | 0.699 | 0.660 | 0.702 | 0.702 | - | 0.771 | - | - | - | 0.771 | - | 0.771 | 0.656 | philosophy | 0.225 | 0.501 | 0.623 | 0.617 | 0.636 | 0.575 | 0.636 | 0.643 | - | 0.675 | - | - | - | 0.710 | - | 0.755 | 0.639 | prehistory | 0.345 | 0.530 | 0.688 | 0.688 | 0.703 | 0.688 | 0.753 | 0.743 | - | 0.774 | - | - | - | 0.796 | - | 0.833 | 0.663 | professionalaccounting | 0.237 | 0.308 | 0.439 | 0.443 | 0.443 | 0.404 | 0.482 | 0.482 | - | 0.510 | - | - | - | 0.546 | - | 0.588 | 0.393 | professionallaw | 0.201 | 0.288 | 0.378 | 0.382 | 0.370 | 0.369 | 0.414 | 0.424 | - | 0.431 | - | - | - | 0.475 | - | 0.505 | 0.393 | professionalmedicine | 0.209 | 0.463 | 0.698 | 0.705 | 0.720 | 0.676 | 0.757 | 0.775 | - | 0.786 | - | - | - | 0.830 | - | 0.790 | 0.632 | professionalpsychology | 0.303 | 0.459 | 0.651 | 0.655 | 0.620 | 0.601 | 0.676 | 0.679 | - | 0.733 | - | - | - | 0.766 | - | 0.712 | 0.616 | publicrelations | 0.345 | 0.500 | 0.572 | 0.563 | 0.654 | 0.581 | 0.600 | 0.636 | - | 0.645 | - | - | - | 0.709 | - | 0.672 | 0.600 | securitystudies | 0.412 | 0.604 | 0.636 | 0.636 | 0.685 | 0.677 | 0.730 | 0.730 | - | 0.746 | - | - | - | 0.755 | - | 0.804 | 0.640 | sociology | 0.427 | 0.656 | 0.731 | 0.746 | 0.800 | 0.766 | 0.800 | 0.815 | - | 0.781 | - | - | - | 0.855 | - | 0.840 | 0.736 | usforeignpolicy | 0.470 | 0.610 | 0.720 | 0.710 | 0.740 | 0.780 | 0.830 | 0.830 | - | 0.830 | - | - | - | 0.840 | - | 0.860 | 0.800 | virology | 0.319 | 0.379 | 0.433 | 0.433 | 0.457 | 0.409 | 0.463 | 0.475 | - | 0.487 | - | - | - | 0.487 | - | 0.542 | 0.415 | worldreligions | 0.362 | 0.637 | 0.719 | 0.719 | 0.760 | 0.783 | 0.783 | 0.777 | - | 0.818 | - | - | - | 0.807 | - | 0.865 | 0.748 | MMLU | 0.298 | 0.457 | 0.598 | 0.598 | 0.613 | 0.598 | 0.651 | 0.654 | - | 0.684 | - | - | - | 0.698 | - | 0.674 | 0.577 | AGIEVAL aquarat | 0.572 | 0.760 | 0.866 | 0.840 | 0.860 | 0.834 | 0.866 | 0.897 | - | 0.885 | - | - | - | 0.860 | - | 0.848 | 0.864 | logiqa | 0.062 | 0.230 | 0.451 | 0.453 | 0.462 | 0.393 | 0.420 | 0.431 | - | 0.465 | - | - | - | 0.520 | - | 0.586 | 0.414 | lsatar | 0.208 | 0.313 | 0.486 | 0.500 | 0.904 | 0.430 | 0.486 | 0.517 | - | 0.469 | - | - | - | 0.495 | - | 0.678 | 0.808 | lsatlr | 0.164 | 0.372 | 0.601 | 0.594 | 0.672 | 0.574 | 0.641 | 0.658 | - | 0.725 | - | - | - | 0.768 | - | 0.813 | 0.703 | lsatrc | 0.327 | 0.464 | 0.669 | 0.665 | 0.710 | 0.657 | 0.687 | 0.713 | - | 0.713 | - | - | - | 0.806 | - | 0.828 | 0.695 | saten | 0.412 | 0.655 | 0.830 | 0.825 | 0.830 | 0.825 | 0.820 | 0.820 | - | 0.834 | - | - | - | 0.873 | - | 0.898 | 0.737 | satmath | 0.772 | 0.950 | 0.990 | 0.981 | 0.977 | 0.986 | 0.990 | 0.990 | - | 0.990 | - | - | - | 0.995 | - | 0.972 | 0.954 | AGIEVAL | 0.282 | 0.458 | 0.641 | 0.636 | 0.703 | 0.608 | 0.643 | 0.659 | - | 0.678 | - | - | - | 0.717 | - | 0.764 | 0.676 | AGIEVALCbiology | 0.152 | 0.539 | 0.765 | 0.760 | 0.773 | 0.769 | 0.834 | 0.847 | - | 0.856 | - | - | - | 0.878 | - | 0.708 | 0.695 | AGIEVALCchemistry | 0.117 | 0.397 | 0.622 | 0.602 | 0.661 | 0.568 | 0.647 | 0.656 | - | 0.720 | - | - | - | 0.803 | - | 0.754 | 0.558 | AGIEVALCchinese | 0.081 | 0.365 | 0.516 | 0.504 | 0.573 | 0.581 | 0.609 | 0.634 | - | 0.678 | - | - | - | 0.739 | - | 0.707 | 0.536 | AGIEVALCenglish | 0.477 | 0.728 | 0.856 | 0.849 | 0.866 | 0.820 | 0.856 | 0.856 | - | 0.866 | - | - | - | 0.872 | - | 0.888 | 0.813 | AGIEVALCgeography | 0.281 | 0.547 | 0.708 | 0.683 | 0.628 | 0.648 | 0.758 | 0.743 | - | 0.768 | - | - | - | 0.829 | - | 0.819 | 0.628 | AGIEVALChistory | 0.319 | 0.612 | 0.736 | 0.727 | 0.774 | 0.702 | 0.761 | 0.770 | - | 0.821 | - | - | - | 0.872 | - | 0.889 | 0.702 | AGIEVALCjecqaca | 0.183 | 0.303 | 0.397 | 0.392 | 0.369 | 0.378 | 0.425 | 0.439 | - | 0.482 | - | - | - | 0.566 | - | 0.652 | 0.367 | AGIEVALCjecqakd | 0.123 | 0.359 | 0.480 | 0.484 | 0.492 | 0.513 | 0.561 | 0.574 | - | 0.613 | - | - | - | 0.676 | - | 0.701 | 0.473 | AGIEVALClogiqa | 0.122 | 0.317 | 0.496 | 0.483 | 0.508 | 0.471 | 0.499 | 0.497 | - | 0.562 | - | - | - | 0.599 | - | 0.642 | 0.465 | AGIEVALCmathcloze | 0.728 | 0.669 | 0.923 | 0.957 | 0.957 | 0.838 | 0.830 | 0.923 | - | 0.881 | - | - | - | 0.932 | 0.864 | 0.889 | 0.983 | AGIEVALCmathqa | 0.500 | 0.704 | 0.828 | 0.812 | 0.928 | 0.764 | 0.863 | 0.851 | - | 0.813 | - | - | - | 0.852 | 0.828 | 0.904 | 0.892 | AGIEVALCphysics | 0.080 | 0.333 | 0.436 | 0.431 | 0.471 | 0.454 | 0.563 | 0.545 | - | 0.626 | - | - | - | 0.701 | 0.741 | 0.528 | 0.419 | AGIEVALC | 0.225 | 0.448 | 0.602 | 0.589 | 0.611 | 0.580 | 0.639 | 0.646 | - | 0.680 | - | - | - | 0.729 | 0.811 | 0.733 | 0.574 | BBH booleanexpressions | 0.724 | 0.728 | 0.820 | 0.812 | 0.648 | 0.612 | 0.560 | 0.620 | - | 0.900 | - | - | - | 0.832 | - | 0.768 | 0.840 | causaljudgement | 0.491 | 0.561 | 0.593 | 0.582 | 0.604 | 0.540 | 0.588 | 0.572 | - | 0.604 | - | - | - | 0.631 | - | 0.636 | 0.636 | dateunderstanding | 0.504 | 0.752 | 0.880 | 0.888 | 0.928 | 0.852 | 0.936 | 0.912 | - | 0.916 | - | - | - | 0.940 | - | 0.884 | 0.904 | disambiguationqa | 0.448 | 0.464 | 0.648 | 0.588 | 0.680 | 0.464 | 0.544 | 0.520 | - | 0.636 | - | - | - | 0.448 | - | 0.436 | 0.568 | dycklanguages | 0.412 | 0.524 | 0.580 | 0.572 | 0.500 | 0.672 | 0.696 | 0.688 | - | 0.772 | - | - | - | 0.816 | - | 0.848 | 0.448 | formalfallacies | 0.800 | 0.800 | 0.748 | 0.776 | 1.000 | 0.568 | 0.992 | 0.604 | - | 0.768 | - | - | - | 0.728 | - | 0.976 | 0.520 | geometricshapes | 0.228 | 0.572 | 0.536 | 0.556 | 0.804 | 0.692 | 0.716 | 0.676 | - | 0.688 | - | - | - | 0.728 | - | 0.780 | 0.764 | hyperbaton | 0.576 | 0.692 | 0.872 | 0.856 | 0.960 | 0.912 | 0.952 | 0.960 | - | 0.976 | - | - | - | 0.948 | - | 0.940 | 0.984 | logicaldeductionfiveobjects | 0.416 | 0.772 | 0.884 | 0.868 | 1.000 | 0.856 | 0.872 | 0.928 | - | 0.936 | - | - | - | 0.972 | - | 0.988 | 0.988 | logicaldeductionsevenobjects | 0.360 | 0.664 | 0.856 | 0.840 | 1.000 | 0.816 | 0.860 | 0.880 | - | 0.888 | - | - | - | 0.924 | - | 0.968 | 0.996 | logicaldeductionthreeobjects | 0.612 | 0.932 | 0.988 | 0.988 | 0.996 | 0.988 | 0.984 | 0.980 | - | 0.996 | - | - | - | 1.000 | - | 0.988 | 0.980 | movierecommendation | 0.360 | 0.416 | 0.528 | 0.504 | 0.604 | 0.492 | 0.520 | 0.544 | - | 0.572 | - | - | - | 0.616 | - | 0.668 | 0.684 | multisteparithmetictwo | 0.896 | 0.988 | 0.996 | 0.988 | 1.000 | 0.984 | 1.000 | 1.000 | - | 0.572 | - | - | - | 0.996 | - | 1.000 | 0.984 | navigate | 0.516 | 0.576 | 0.580 | 0.580 | 1.000 | 0.508 | 0.992 | 0.608 | - | 0.680 | - | - | - | 0.728 | - | 0.996 | 0.996 | objectcounting | 0.664 | 0.872 | 0.992 | 0.996 | 1.000 | 0.996 | 0.992 | 0.996 | - | 0.996 | - | - | - | 1.000 | - | 0.996 | 0.952 | penguinsinatable | 0.602 | 0.897 | 0.945 | 0.958 | 0.986 | 0.993 | 1.000 | 0.993 | - | 1.000 | - | - | - | 1.000 | - | 1.000 | 0.986 | reasoningaboutcoloredobjects | 0.520 | 0.792 | 0.952 | 0.960 | 0.996 | 0.928 | 0.940 | 0.960 | - | 0.948 | - | - | - | 0.984 | - | 0.964 | 0.968 | ruinnames | 0.164 | 0.512 | 0.508 | 0.516 | 0.632 | 0.604 | 0.656 | 0.652 | - | 0.772 | - | - | - | 0.776 | - | 0.768 | 0.776 | salienttranslationerrordetection | 0.316 | 0.488 | 0.612 | 0.632 | 0.696 | 0.604 | 0.628 | 0.572 | - | 0.660 | - | - | - | 0.680 | - | 0.576 | 0.652 | snarks | 0.471 | 0.573 | 0.730 | 0.685 | 0.780 | 0.735 | 0.792 | 0.735 | - | 0.780 | - | - | - | 0.837 | - | 0.831 | 0.764 | sportsunderstanding | 0.472 | 0.524 | 0.624 | 0.596 | 0.708 | 0.540 | 0.644 | 0.636 | - | 0.560 | - | - | - | 0.776 | - | 0.464 | 0.600 | temporalsequences | 0.136 | 0.400 | 0.912 | 0.892 | 1.000 | 0.940 | 0.992 | 0.992 | - | 0.980 | - | - | - | 0.992 | - | 0.948 | 0.996 | trackingshuffledobjectsfiveobjects | 0.280 | 0.648 | 0.940 | 0.964 | 1.000 | 0.968 | 0.956 | 0.936 | - | 0.996 | - | - | - | 0.996 | - | 0.988 | 0.988 | trackingshuffledobjectssevenobjects | 0.232 | 0.564 | 0.852 | 0.884 | 0.996 | 0.924 | 0.952 | 0.972 | - | 0.944 | - | - | - | 0.948 | - | 0.980 | 0.952 | trackingshuffledobjectsthreeobjects | 0.408 | 0.736 | 0.896 | 0.884 | 1.000 | 0.920 | 0.920 | 0.952 | - | 0.996 | - | - | - | 0.996 | - | 0.992 | 0.992 | weboflies | 0.456 | 0.460 | 0.552 | 0.544 | 1.000 | 0.488 | - | 0.488 | - | 0.540 | - | - | - | 0.492 | - | 1.000 | 1.000 | wordsorting | 0.080 | 0.136 | 0.220 | 0.228 | 0.280 | 0.292 | 0.292 | 0.288 | - | 0.324 | - | - | - | 0.324 | - | 0.400 | 0.220 | BBH | 0.446 | 0.628 | 0.748 | 0.744 | 0.845 | 0.734 | 0.763 | 0.763 | - | 0.791 | - | - | - | 0.817 | - | 0.825 | 0.819 | MUSR murdermystery | 0.524 | 0.560 | 0.640 | 0.668 | 0.588 | 0.584 | 0.636 | 0.652 | - | 0.672 | - | - | - | 0.636 | - | 0.560 | 0.620 | objectplacements | 0.480 | 0.512 | 0.566 | 0.556 | 0.476 | 0.536 | 0.582 | 0.578 | - | 0.528 | - | - | - | 0.516 | - | 0.436 | 0.476 | teamallocation | 0.280 | 0.468 | 0.628 | 0.612 | 0.708 | 0.648 | 0.656 | 0.668 | - | 0.564 | - | - | - | 0.632 | - | 0.728 | 0.464 | MUSR | 0.428 | 0.513 | 0.611 | 0.612 | 0.590 | 0.589 | 0.624 | 0.632 | - | 0.588 | - | - | - | 0.594 | - | 0.574 | 0.520 | GPQAdiamond | 0.262 | 0.282 | 0.434 | 0.489 | 0.616 | 0.398 | 0.530 | - | - | 0.439 | - | - | - | 0.555 | - | 0.555 | 0.540 | GPQA | 0.262 | 0.282 | 0.434 | 0.489 | 0.616 | 0.398 | 0.530 | - | - | 0.439 | - | - | - | 0.555 | - | 0.555 | 0.540 | MMLUPRO biology | 0.268 | 0.596 | 0.799 | 0.784 | 0.852 | 0.804 | 0.822 | 0.831 | - | 0.824 | - | - | - | 0.852 | - | 0.812 | 0.716 | business | 0.348 | 0.548 | 0.717 | 0.692 | 0.784 | 0.724 | 0.740 | 0.738 | - | 0.784 | - | - | - | 0.812 | - | 0.788 | 0.676 | chemistry | 0.300 | 0.564 | 0.720 | 0.700 | 0.848 | 0.700 | 0.746 | 0.747 | - | 0.796 | - | - | - | 0.780 | - | 0.740 | 0.704 | computerscience | 0.232 | 0.532 | 0.680 | 0.664 | 0.744 | 0.676 | 0.704 | 0.707 | - | 0.704 | - | - | - | 0.804 | - | 0.780 | 0.676 | economics | 0.276 | 0.544 | 0.716 | 0.732 | 0.784 | 0.732 | 0.741 | 0.759 | - | 0.808 | - | - | - | 0.832 | - | 0.796 | 0.700 | engineering | 0.232 | 0.456 | 0.557 | 0.596 | 0.624 | 0.536 | 0.587 | 0.600 | - | 0.620 | - | - | - | 0.676 | - | 0.476 | 0.496 | health | 0.192 | 0.312 | 0.559 | 0.564 | 0.668 | 0.632 | 0.630 | 0.639 | - | 0.652 | - | - | - | 0.680 | - | 0.652 | 0.576 | history | 0.200 | 0.300 | 0.488 | 0.524 | 0.604 | 0.536 | 0.561 | 0.556 | - | 0.600 | - | - | - | 0.680 | - | 0.588 | 0.492 | law | 0.076 | 0.208 | 0.295 | 0.304 | 0.336 | 0.348 | 0.353 | 0.379 | - | 0.384 | - | - | - | 0.468 | - | 0.392 | 0.320 | math | 0.444 | 0.676 | 0.832 | 0.800 | 0.884 | 0.780 | 0.824 | 0.827 | - | 0.844 | - | - | - | 0.872 | - | 0.860 | 0.796 | other | 0.188 | 0.356 | 0.484 | 0.528 | 0.600 | 0.560 | 0.590 | 0.593 | - | 0.656 | - | - | - | 0.704 | - | 0.636 | 0.544 | philosophy | 0.168 | 0.336 | 0.504 | 0.520 | 0.600 | 0.492 | 0.559 | 0.567 | - | 0.580 | - | - | - | 0.664 | - | 0.612 | 0.484 | physics | 0.232 | 0.580 | 0.708 | 0.740 | 0.852 | 0.720 | 0.753 | 0.752 | - | 0.756 | - | - | - | 0.824 | - | 0.736 | 0.724 | psychology | 0.212 | 0.512 | 0.672 | 0.632 | 0.700 | 0.668 | 0.725 | 0.692 | - | 0.728 | - | - | - | 0.748 | - | 0.672 | 0.624 | MMLUPRO | 0.240 | 0.465 | 0.622 | 0.627 | 0.705 | 0.636 | 0.674 | 0.679 | - | 0.695 | - | - | - | 0.742 | - | 0.681 | 0.609 | CATEGORIES REASONING | 0.332 | 0.593 | 0.744 | 0.746 | 0.742 | 0.738 | 0.787 | 0.792 | 0.827 | 0.802 | 0.815 | 0.832 | 0.838 | 0.824 | 0.885 | 0.850 | 0.730 | UNDERSTANDING | 0.353 | 0.521 | 0.653 | 0.651 | 0.661 | 0.646 | 0.693 | 0.697 | 0.722 | 0.725 | 0.699 | 0.747 | 0.849 | 0.743 | 0.809 | 0.731 | 0.643 | LANGUAGE | 0.471 | 0.590 | 0.644 | 0.638 | 0.662 | 0.660 | 0.700 | 0.700 | 0.729 | 0.714 | 0.698 | 0.725 | 0.692 | 0.701 | 0.780 | 0.651 | 0.522 | KNOWLEDGE | 0.367 | 0.456 | 0.552 | 0.548 | 0.531 | 0.558 | 0.527 | 0.541 | 0.657 | 0.632 | 0.520 | 0.501 | 0.599 | 0.563 | 0.581 | 0.659 | 0.531 | COT | 0.307 | 0.478 | 0.616 | 0.611 | 0.712 | 0.611 | 0.667 | 0.670 | - | 0.682 | - | - | - | 0.717 | - | 0.674 | 0.636 | MATHCOT | 0.545 | 0.766 | 0.900 | 0.890 | 0.954 | 0.882 | 0.884 | 0.895 | 0.910 | 0.904 | 0.820 | 0.954 | 0.910 | 0.924 | 0.927 | 0.948 | 0.949 | CODE | 0.317 | 0.443 | 0.538 | 0.524 | 0.534 | 0.532 | 0.582 | 0.573 | - | 0.618 | - | 0.755 | - | 0.586 | 0.321 | 0.506 | 0.551 | DISCIPLINES NLP | 0.402 | 0.562 | 0.674 | 0.672 | 0.673 | 0.671 | 0.694 | 0.700 | 0.767 | 0.739 | 0.769 | 0.764 | 0.768 | 0.723 | 0.772 | 0.765 | 0.654 | MATH | 0.433 | 0.638 | 0.789 | 0.767 | 0.829 | 0.779 | 0.808 | 0.815 | 0.910 | 0.824 | 0.820 | 0.954 | 0.910 | 0.828 | 0.927 | 0.829 | 0.803 | SCIENCE | 0.340 | 0.648 | 0.783 | 0.789 | 0.813 | 0.782 | 0.808 | 0.818 | - | 0.847 | - | - | - | 0.866 | 0.946 | 0.793 | 0.774 | ENGINEERING | 0.265 | 0.463 | 0.561 | 0.589 | 0.607 | 0.554 | 0.595 | 0.606 | - | 0.655 | - | - | - | 0.693 | - | 0.491 | 0.518 | MEDICINE | 0.232 | 0.392 | 0.554 | 0.552 | 0.578 | 0.566 | 0.613 | 0.620 | - | 0.642 | - | 0.597 | - | 0.668 | 0.598 | 0.642 | 0.547 | HUMANITIES | 0.283 | 0.449 | 0.595 | 0.593 | 0.627 | 0.598 | 0.642 | 0.639 | 0.480 | 0.677 | 0.520 | 0.470 | 0.470 | 0.707 | 0.600 | 0.717 | 0.577 | BUSINESS | 0.358 | 0.555 | 0.717 | 0.717 | 0.744 | 0.725 | 0.756 | 0.762 | - | 0.807 | - | - | - | 0.815 | - | 0.724 | 0.683 | LAW | 0.200 | 0.329 | 0.438 | 0.458 | 0.490 | 0.448 | 0.473 | 0.488 | - | 0.535 | - | - | - | 0.586 | - | 0.625 | 0.489 | COMPOSITE AVERAGE AVG | 0.363 | 0.539 | 0.664 | 0.661 | 0.676 | 0.663 | 0.693 | 0.699 | 0.767 | 0.732 | 0.768 | 0.764 | 0.767 | 0.731 | 0.759 | 0.743 | 0.652_ |

CODE MODEL TESTS:

MODEL	Codestral-22B-v0.1	Codestral-22B-Instruct-v0.1	Deepseek-Coder-V2-Lite-Instruct	Qwen2.5-Coder-0.5B-32k-Instruct	Qwen2.5-Coder-1.5B-Instruct	Qwen2.5-Coder-3B-Instruct	Qwen2.5-Coder-3B-Instruct	Qwen2.5-3B-32k-Instruct	Qwen2.5-Coder-7B	Qwen2.5-Coder-7B	Qwen2.5-Coder-7B-Instruct	Qwen2.5-Coder-7B-Instruct	Qwen2.5-Coder-7B-Instruct	Qwen2.5-Coder-14B-Instruct	Qwen2.5-Coder-14B-Instruct	Qwen2.5-Coder-14B	Qwen2.5-Coder-32B-Instruct	Qwen2.5-Coder-32B-Instruct	Qwen3-Coder-30B-A3B-Instruct	Qwen3.8-27B
params	22B	22B	14.77B	0.49403B	1.54B	3.09B	3.09B	3.09B	7.62B	7.62B	7.62B	7.62B	7.62B	14.77B	14.77B	14.77B	32.76B	32.76B	30.53B	26.9B
quant	IQ4_XS	IQ4_XS	IQ4_XS	Q6_K	Q6_K	Q6_K	Q6KH	Q6_K	IQ4_XS	Q6_K	IQ4_XS	Q4KH	Q6KH	IQ4_XS	Q4KH	IQ4_XS	IQ4_XS	Q4KH	Q4KH	Q4EH
engine	llama.cpp version: 4132	llama.cpp version: 4191	llama.cpp version: 4488	llama.cpp version: 4150	llama.cpp version: 4150	llama.cpp version: 4150	llama.cpp version: 7021	llama.cpp version: 4150	llama.cpp version: 4295	llama.cpp version: 4132	llama.cpp version: 4094	llama.cpp version: 7003	llama.cpp version: 7003	llama.cpp version: 4120	llama.cpp version: 7048	llama.cpp version: 4150	llama.cpp version: 4150	llama.cpp version: 7192	llama.cpp version: 8929	llama.cpp version: 10444
TEST	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc
HUMANEVAL	0.664	0.810	0.847	0.518	0.676	0.835	0.847	0.780	0.640	0.713	0.829	0.841	0.841	0.878	0.871	0.676	0.884	0.896	0.951	0.945
HUMANEVALP	0.554	0.682	-	0.432	0.567	0.719	0.731	0.682	0.530	0.579	0.707	0.713	0.713	0.756	0.731	0.536	0.756	0.762	0.817	0.841
HUMANEVALFIM	0.719	0.719	0.621	0.518	0.524	0.634	0.408	-	0.713	0.756	0.493	0.786	0.737	0.829	0.829	0.518	0.890	0.829	0.731	-
MBPP	0.630	0.653	-	0.408	0.560	0.618	0.638	0.599	0.614	0.571	0.735	0.731	0.708	0.727	0.723	0.661	0.715	0.708	0.677	0.727
MBPPP	0.558	0.593	-	0.352	0.504	0.589	0.598	0.584	0.540	0.513	0.687	0.687	0.660	0.665	0.656	0.558	0.669	0.678	0.656	0.714
HUMANEVALX_cpp	0.640	0.621	-	0.286	0.426	0.567	0.554	0.237	0.548	0.475	0.676	0.701	0.719	0.506	0.463	0.573	0.689	0.682	0.847	0.823
HUMANEVALX_java	0.756	0.670	-	0.512	0.609	0.743	0.737	0.615	0.725	0.652	0.798	0.792	0.810	0.201	0.262	0.762	0.841	0.859	0.902	0.939
HUMANEVALX_js	0.658	0.621	-	0.493	0.615	0.670	0.756	0.682	0.628	0.658	0.798	0.768	0.774	0.817	0.810	0.695	0.835	0.823	0.878	0.890
HUMANEVALX	0.684	0.638	-	0.430	0.550	0.660	0.682	0.512	0.634	0.595	0.758	0.754	0.768	0.508	0.512	0.676	0.788	0.788	0.876	0.884
CRUXEVAL_input	0.438	0.351	-	0.435	0.416	0.481	0.477	0.347	0.255	0.267	0.578	0.580	0.578	0.677	0.666	0.281	0.676	0.673	0.548	0.657
CRUXEVAL_output	0.465	0.447	-	0.278	0.332	0.413	0.415	0.311	0.381	0.435	0.507	0.511	0.505	0.577	0.580	0.422	0.610	0.616	0.557	0.643
CRUXEVAL	0.451	0.399	-	0.356	0.374	0.447	0.446	0.329	0.318	0.351	0.543	0.545	0.541	0.627	0.623	0.351	0.643	0.645	0.553	0.650
CRUXEVALFIM_input	0.295	0.351	-	0.017	0.155	0.208	0.271	-	0.296	0.313	0.322	0.432	0.447	0.421	0.542	0.346	0.592	0.585	0.412	-
CRUXEVALFIM_output	0.441	0.355	-	0.098	0.222	0.323	0.335	-	0.352	0.365	0.481	0.436	0.446	0.546	0.395	0.481	0.483	0.506	0.442	-
CRUXEVALFIM	0.368	0.353	-	0.058	0.188	0.266	0.303	-	0.324	0.339	0.401	0.434	0.446	0.483	0.468	0.413	0.536	0.545	0.427	-
CODE	0.483	0.467	0.734	0.278	0.368	0.453	0.462	0.449	0.413	0.427	0.548	0.571	0.571	0.593	0.585	0.458	0.648	0.650	0.585	0.729

MATH TESTS:

MODEL	Deepseek-R1-Distill-Llama-8B	Deepseek-R1-Distill-Llama-8B	Deepseek-R1-Distill-Qwen-1.5B	Deepseek-R1-Distill-Qwen-7B	Deepseek-R1-Distill-Qwen-7B	Deepseek-R1-Distill-Qwen-14B	Deepseek-R1-Distill-Qwen-14B	Deepseek-R1-Distill-Qwen-32B	Deepseek-R1-Distill-Qwen-32B	gemma-4-12B-it	gemma-4-12B-it-qat-q4_0-unquantized	gemma-4-26B-A4B-it	gemma-4-26B-A4B-it-qat-q4_0-unquantized	GLM-4.7-Flash	GLM-4.7-Flash	GLM-4.7-Flash	GLM-Z1-9B-0414	GLM-Z1-9B-0414	GLM-Z1-9B-0414	GLM-Z1-32B-0414	Qwen3-32B	Qwen3.5-27B	Qwen3.8-27B	QwQ-32B	QwQ-32B
params	8.03B	8.03B	1.78B	7.62B	7.62B	14.77B	14.77B	32.76B	32.76B	11.91B	11.91B	25.23B	25.23B	29.94B	29.94B	29.94B	9.40B	9.40B	9.40B	32.57B	32.8B	26.9B	26.9B	32.76B	32.76B
quant	Q6_K	Q6KH	Q8_0	IQ4_XS	Q6KH	IQ4_XS	Q4KH	IQ4_XS	Q4KH	Q4EH	Q4EH	Q6KH	Q4PH	Q4EH	Q4KH	Q6KH	Q4KH	Q4PH	Q6KH	Q4KH	Q4KH	Q4KH	Q4EH	IQ4_XS	Q4KH
engine	llama.cpp version: 4707	llama.cpp version: 5898	llama.cpp version: 4763	llama.cpp version: 4644	llama.cpp version: 7699	llama.cpp version: 4657	llama.cpp version: 7710	llama.cpp version: 4559	llama.cpp version: 7719	llama.cpp version: 9530	llama.cpp version: 9617	llama.cpp version: 8672	llama.cpp version: 9670	llama.cpp version: 9548	llama.cpp version: 7885	llama.cpp version: 7845	llama.cpp version: 7230	llama.cpp version: 7268	llama.cpp version: 5935	llama.cpp version: 7607	llama.cpp version: 8880	llama.cpp version: 8485	llama.cpp version: 10444	llama.cpp version: 4820	llama.cpp version: 6026
TEST	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc
GSM8K	-	0.888	-	-	0.888	-	0.944	-	0.956	0.964	0.964	0.944	0.964	0.996	0.992	0.984	0.968	0.968	0.964	0.988	0.968	0.984	0.980	-	0.964
APPLE	-	0.870	-	-	0.810	-	0.790	-	0.810	0.900	0.910	0.900	0.920	0.950	0.960	1.000	0.880	0.920	0.880	0.910	0.900	0.930	0.940	-	0.880
GPQA_diamond	-	0.308	-	-	0.323	-	0.489	-	0.494	0.580	0.575	0.636	0.691	0.444	0.444	0.444	0.555	0.540	0.434	0.585	0.494	0.691	0.717	-	0.555
GPQA	-	0.308	-	-	0.323	-	0.489	-	0.494	0.580	0.575	0.636	0.691	0.444	0.444	0.444	0.555	0.540	0.434	0.585	0.494	0.691	0.717	-	0.555
MATH1_algebra	0.933	0.977	0.918	0.962	1.000	0.925	0.970	0.962	0.985	1.000	1.000	1.000	1.000	1.000	1.000	1.000	0.985	0.992	0.992	0.992	0.992	1.000	0.992	0.992	1.000
MATH1countingand_probability	0.820	0.948	0.794	0.948	1.000	0.923	0.948	0.948	0.948	0.974	0.974	0.948	0.948	1.000	1.000	1.000	-	0.974	1.000	1.000	1.000	1.000	1.000	0.974	1.000
MATH1_geometry	0.842	0.868	0.710	0.736	0.921	0.868	0.921	0.921	0.894	0.868	0.894	0.894	0.921	1.000	1.000	0.973	-	-	0.947	0.947	0.921	0.921	0.894	0.921	0.921
MATH1intermediatealgebra	0.923	0.980	0.730	0.903	0.961	0.865	0.980	0.961	0.942	1.000	0.980	0.961	0.980	1.000	1.000	1.000	-	-	0.961	0.980	0.980	1.000	0.980	1.000	0.980
MATH1numbertheory	0.700	0.900	0.866	0.800	1.000	0.700	0.966	0.933	0.966	1.000	1.000	1.000	1.000	1.000	1.000	1.000	-	-	0.900	1.000	0.666	1.000	1.000	0.800	0.833
MATH1_prealgebra	0.813	0.941	0.883	0.965	0.976	0.883	0.988	0.953	0.988	0.988	1.000	0.976	0.976	1.000	1.000	1.000	-	-	0.988	1.000	0.941	0.976	0.988	0.953	0.976
MATH1_precalculus	0.684	1.000	0.596	0.859	1.000	0.842	0.947	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000	-	-	0.947	0.929	1.000	1.000	0.964	0.982	0.929
MATH1	0.842	0.956	0.814	0.910	0.983	0.878	0.965	0.958	0.970	0.983	0.986	0.977	0.981	1.000	1.000	0.997	-	-	0.972	0.981	0.954	0.988	0.979	0.963	0.965
MATH2_algebra	0.845	0.975	0.825	0.930	0.985	0.900	0.965	0.995	0.985	0.985	0.995	0.985	0.990	1.000	1.000	1.000	-	-	0.980	0.995	0.995	1.000	0.990	0.975	0.990
MATH2countingand_probability	0.831	0.930	0.782	0.851	0.891	0.841	0.881	0.950	0.900	0.990	0.980	0.970	0.990	0.990	1.000	1.000	-	-	0.980	0.980	0.990	0.990	0.970	0.990	0.970
MATH2_geometry	0.841	0.963	0.743	0.914	0.951	0.792	0.939	0.914	0.987	0.963	0.963	0.926	0.975	0.987	1.000	1.000	-	-	0.987	0.939	0.926	1.000	1.000	0.963	0.963
MATH2intermediatealgebra	0.859	0.953	0.664	0.875	0.992	0.835	0.960	0.968	0.976	0.960	0.953	0.960	0.953	1.000	1.000	1.000	-	-	0.968	0.984	0.976	0.960	0.968	0.960	0.984
MATH2numbertheory	0.826	0.913	0.782	0.826	0.967	0.891	0.934	0.934	0.913	0.956	0.956	0.945	0.967	1.000	0.989	0.989	-	-	0.956	0.989	0.891	0.967	0.945	0.945	0.967
MATH2_prealgebra	0.898	0.966	0.887	0.909	0.977	0.875	0.949	0.971	0.971	0.977	0.971	0.988	0.983	0.994	1.000	1.000	-	-	0.988	0.988	0.971	0.988	0.988	0.971	0.960
MATH2_precalculus	0.787	0.964	0.663	0.902	1.000	0.805	0.964	0.955	0.991	0.982	0.991	0.938	0.973	1.000	1.000	1.000	-	-	0.955	0.823	0.964	0.982	0.946	0.964	0.964
MATH2	0.846	0.956	0.777	0.893	0.970	0.856	0.946	0.963	0.965	0.975	0.975	0.965	0.977	0.996	0.998	0.998	-	-	0.975	0.963	0.966	0.985	0.975	0.968	0.973
MATH3_algebra	0.873	0.938	0.854	0.934	0.984	0.911	0.961	0.992	0.980	0.992	0.996	0.980	1.000	1.000	0.988	1.000	-	-	0.980	0.988	0.980	0.988	0.992	0.996	0.984
MATH3countingand_probability	0.800	0.890	0.730	0.770	0.920	0.830	0.850	0.930	0.940	0.980	0.970	0.960	0.980	1.000	1.000	1.000	-	-	0.970	0.980	0.940	0.990	0.980	1.000	1.000
MATH3_geometry	0.794	0.901	0.627	0.911	0.960	0.794	0.911	0.901	0.941	0.941	0.901	0.970	0.931	1.000	0.970	0.990	-	-	0.970	0.950	0.921	0.970	0.990	0.970	0.931
MATH3intermediatealgebra	0.825	0.969	0.635	0.902	0.964	0.882	0.953	0.964	0.984	0.933	0.933	0.969	0.964	0.994	0.989	1.000	-	-	0.969	0.938	0.974	0.979	0.948	0.969	0.938
MATH3numbertheory	0.819	0.934	0.696	0.754	0.942	0.770	0.844	0.926	0.909	0.950	0.950	0.893	0.926	0.967	0.991	1.000	-	-	0.926	0.942	0.844	0.942	0.877	0.942	0.918
MATH3_prealgebra	0.875	0.950	0.763	0.883	0.959	0.892	0.941	0.946	0.950	0.982	0.991	0.950	0.986	0.995	0.995	1.000	-	-	0.986	0.986	0.955	0.991	0.982	0.982	0.977
MATH3_precalculus	0.661	0.929	0.582	0.874	0.952	0.818	0.960	0.968	0.929	0.929	0.937	0.944	0.944	0.992	1.000	1.000	-	-	0.968	0.850	0.937	0.976	0.968	0.897	0.905
MATH3	0.822	0.937	0.719	0.876	0.960	0.859	0.929	0.954	0.954	0.962	0.961	0.956	0.969	0.993	0.991	0.999	-	-	0.970	0.954	0.946	0.979	0.966	0.969	0.955
MATH4_algebra	0.848	0.950	0.805	0.897	0.982	0.922	0.964	0.957	0.985	0.992	0.996	0.989	0.992	0.989	0.992	1.000	-	-	0.989	0.978	0.992	0.989	1.000	0.992	0.985
MATH4countingand_probability	0.729	0.882	0.639	0.738	0.900	0.711	0.864	0.945	0.945	0.945	0.954	0.927	0.972	1.000	0.990	0.990	-	-	0.963	0.945	0.945	0.981	0.963	0.981	0.981
MATH4_geometry	0.792	0.896	0.576	0.776	0.896	0.768	0.832	0.832	0.856	0.888	0.904	0.896	0.896	0.960	0.976	0.976	-	-	0.920	0.888	0.840	0.952	0.912	0.872	0.840
MATH4intermediatealgebra	0.778	0.947	0.588	0.858	0.987	0.850	0.931	0.935	0.915	0.923	0.935	0.955	0.955	0.979	0.987	0.995	-	-	0.939	0.895	0.939	0.987	0.959	0.947	0.907
MATH4numbertheory	0.795	0.950	0.697	0.809	0.915	0.725	0.859	0.894	0.929	0.936	0.964	0.922	0.929	0.985	0.992	0.992	-	-	0.929	0.964	0.852	0.964	0.929	0.943	0.936
MATH4_prealgebra	0.806	0.931	0.785	0.874	0.958	0.827	0.942	0.921	0.931	0.973	0.947	0.952	0.958	0.994	0.989	0.989	-	-	0.958	0.963	0.937	0.963	0.968	0.926	0.942
MATH4_precalculus	0.719	0.956	0.570	0.868	0.956	0.728	0.921	0.947	0.912	0.885	0.885	0.938	0.947	0.991	0.991	0.991	-	-	0.947	0.807	0.956	0.973	0.938	0.973	0.868
MATH4	0.792	0.935	0.684	0.845	0.953	0.816	0.915	0.925	0.932	0.943	0.948	0.948	0.956	0.985	0.989	0.992	-	-	0.953	0.929	0.933	0.976	0.960	0.952	0.930
MATH5_algebra	0.768	0.947	0.752	0.899	0.970	0.853	0.934	0.970	0.967	0.973	0.973	0.980	0.967	0.990	0.986	0.996	-	-	0.960	0.967	0.980	0.986	0.973	0.964	0.977
MATH5countingand_probability	0.699	0.910	0.569	0.756	0.829	0.699	0.788	0.910	0.861	0.943	0.967	0.967	0.959	0.983	0.959	0.983	-	-	0.934	0.967	0.967	0.959	0.967	0.934	0.902
MATH5_geometry	0.712	0.886	0.545	0.810	0.840	0.727	0.818	0.840	0.810	0.818	0.780	0.856	0.878	0.969	0.946	0.984	-	-	0.878	0.810	0.787	0.901	0.901	0.833	0.742
MATH5intermediatealgebra	0.682	0.900	0.453	0.821	0.875	0.778	0.810	0.810	0.846	0.832	0.810	0.882	0.889	0.953	0.932	0.964	-	-	0.889	0.832	0.875	0.914	0.907	0.860	0.800
MATH5numbertheory	0.811	0.909	0.707	0.727	0.935	0.792	0.915	0.935	0.935	0.941	0.967	0.935	0.967	0.993	0.967	0.993	-	-	0.961	0.954	0.883	0.967	0.941	0.941	0.935
MATH5_prealgebra	0.777	0.849	0.720	0.808	0.896	0.782	0.823	0.875	0.891	0.901	0.922	0.927	0.943	0.979	0.958	0.984	-	-	0.953	0.937	0.953	0.979	0.968	0.927	0.948
MATH5_precalculus	0.562	0.903	0.437	0.851	0.851	0.792	0.800	0.814	0.851	0.800	0.800	0.918	0.918	0.948	0.918	0.955	-	-	0.888	0.666	0.874	0.925	0.903	0.851	0.770
MATH5	0.723	0.904	0.609	0.822	0.897	0.787	0.851	0.884	0.889	0.893	0.893	0.926	0.932	0.974	0.955	0.981	-	-	0.926	0.886	0.911	0.950	0.940	0.907	0.879
MATHCOT	0.795	0.930	0.700	0.860	0.940	0.831	0.910	0.930	0.934	0.944	0.945	0.948	0.958	0.987	0.983	0.992	-	-	0.954	0.936	0.938	0.972	0.961	0.948	0.933

COMPOSITE AVERAGE AVG | 0.795 | 0.907 | 0.700 | 0.860 | 0.918 | 0.831 | 0.895 | 0.930 | 0.918 | 0.931 | 0.932 | 0.937 | 0.948 | 0.968 | 0.964 | 0.972 | - | - | 0.936 | 0.923 | 0.923 | 0.962 | 0.952 | 0.948 | 0.920 |

VISION MODEL TESTS:

MODEL	gemma-3-4b-it	gemma-3-4b-it	gemma-3-12b-it	gemma-3-12b-it	gemma-3-27b-it	gemma-4-12B-it	gemma-4-12B-it-qat-q4_0-unquantized	gemma-4-26B-A4B-it-qat-q4_0-unquantized	GLM-4.6V-Flash	LFM2-VL-1.6B	Llama-4-Scout-17B-16E-Instruct	MiniCPM-V-4_5	MiniCPM-V-4_5	Ministral-3-3B-Instruct-2512	Ministral-3-3B-Instruct-2512	Ministral-3-8B-Instruct-2512	Ministral-3-14B-Instruct-2512	Mistral-Small-3.1-24B-Instruct-2503	Mistral-Small-3.2-24B-Instruct-2506	Qwen2.5-Omni-3B	Qwen2.5-Omni-3B	Qwen2.5-Omni-3B	Qwen2.5-Omni-7B	Qwen2.5-Omni-7B	Qwen2.5-VL-3B-Instruct	Qwen2.5-VL-3B-Instruct	Qwen2.5-VL-3B-Instruct	Qwen2.5-VL-7B-Instruct	Qwen2.5-VL-7B-Instruct	Qwen2.5-VL-32B-Instruct	Qwen3-Omni-30B-A3B-Instruct	Qwen3-VL-2B-Instruct	Qwen3-VL-4B-Instruct	Qwen3-VL-8B-Instruct	Qwen3-VL-8B-Instruct	Qwen3-VL-30B-A3B-Instruct	Qwen3-VL-32B-Instruct	Qwen3-VL-8B-Thinking	Qwen3.5-9B	Qwen3.5-27B	Qwen3.5-35B-A3B	Qwen3.8-27B
params	3.88B	3.88B	11.77B	11.77B	27.01B	11.91B	11.91B	25.23B	9.46B	1.17B	107.77B	8.19B	8.19B	3.43B	3.43B	8.49B	13.51B	23.57B	23.57B	3.09B	3.09B	3.09B	7.62B	7.62B	3.09B	3.09B	3.09B	7.62B	7.62B	32.76B	30.53B	1.72B	4.02B	8.19B	8.19B	30.53B	30.53B	8.19B	8.95B	26.9B	34.66B	26.9B
quant	Q6_K	Q6KH	Q4KH	Q4EH	Q4KH	Q4EH	Q4EH	Q4PH	Q4PH	Q80H	Q2KH	Q4KH	Q6KH	Q4KH	Q6KH	Q4KH	Q4KH	Q4KH	Q4KH	Q4KH	Q6KH	Q80H	Q4KH	Q6KH	Q4KH	Q6KH	Q80H	Q4KH	Q6KH	Q4KH	Q4KH	Q80H	Q6KH	Q4KH	Q6KH	Q4KH	Q4KH	Q6KH	Q4KH	Q4KH	Q4KH	Q4EH
engine	llama.cpp version: 5706	llama.cpp version: 5819	llama.cpp version: 5819	llama.cpp version: 9895	llama.cpp version: 5780	llama.cpp version: 9518	llama.cpp version: 9632	llama.cpp version: 9670	llama.cpp version: 7445	llama.cpp version: 7211	llama.cpp version: 5935	llama.cpp version: 7188	llama.cpp version: 7188	llama.cpp version: 7310	llama.cpp version: 7310	llama.cpp version: 7278	llama.cpp version: 7278	llama.cpp version: 5662	llama.cpp version: 5780	llama.cpp version: 7154	llama.cpp version: 7154	llama.cpp version: 7003	llama.cpp version: 7130	llama.cpp version: 6937	llama.cpp version: 7154	llama.cpp version: 7154	llama.cpp version: 6937	llama.cpp version: 7130	llama.cpp version: 6915	llama.cpp version: 7091	llama.cpp version: 8786	llama.cpp version: 6937	llama.cpp version: 6924	llama.cpp version: 7215	llama.cpp version: 6915	llama.cpp version: 6915	llama.cpp version: 7002	llama.cpp version: 6937	llama.cpp version: 8204	llama.cpp version: 8156	llama.cpp version: 8181	llama.cpp version: 10444
TEST	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc
CHARTQA	0.464	0.456	0.558	0.622	0.662	0.746	0.745	0.748	0.801	0.715	0.719	0.782	0.784	0.702	0.737	0.787	0.734	0.743	0.716	0.825	0.831	0.816	0.794	0.794	0.843	0.884	0.882	0.802	0.800	0.789	0.754	0.720	0.755	0.754	0.762	0.787	0.799	0.734	0.783	0.840	0.807	0.798
DOCVQA	0.567	0.563	0.711	0.798	0.795	0.717	0.757	0.919	0.933	0.728	0.862	0.918	0.911	0.833	0.874	0.926	0.899	0.892	0.866	0.908	0.908	0.908	0.936	0.938	0.874	0.894	0.895	0.939	0.935	0.914	0.862	0.863	0.923	0.918	0.921	0.923	0.909	0.894	0.908	0.936	0.914	0.882
REALWORLDQA	-	-	-	0.587	-	0.647	0.651	0.661	0.700	0.464	-	0.693	0.686	0.579	0.574	0.599	0.599	-	-	0.811	0.812	0.781	0.680	0.681	-	-	-	0.682	0.676	0.682	0.723	0.646	0.717	0.686	0.676	0.713	0.750	0.673	0.739	0.778	0.769	0.754
MMMU_Accounting	0.366	0.400	0.566	0.500	0.700	0.700	0.800	0.800	0.600	0.333	0.866	0.833	0.766	0.533	0.566	0.600	0.600	0.466	0.733	0.433	0.366	0.333	0.500	0.533	0.533	0.400	0.466	0.566	0.600	0.566	0.600	0.466	0.566	0.566	0.533	0.466	0.733	0.733	0.566	0.733	0.566	0.866
MMMU_Agriculture	0.400	0.400	0.500	0.533	0.533	0.533	0.566	0.633	0.533	0.266	0.600	0.633	0.600	0.400	0.466	0.500	0.600	0.500	0.533	0.533	0.400	0.333	0.466	0.366	0.200	0.433	0.500	0.400	0.433	0.566	0.566	0.466	0.533	0.566	0.600	0.633	0.633	0.566	0.500	0.266	0.166	0.666
MMMUArchitectureand_Engineering	0.200	0.166	0.400	0.266	0.333	0.433	0.466	0.633	0.500	0.300	0.366	0.400	0.466	0.333	0.400	0.300	0.366	0.400	0.400	0.400	0.266	0.300	0.433	0.300	0.433	0.200	0.333	0.366	0.433	0.466	0.500	0.066	0.266	0.266	0.300	0.366	0.500	0.533	0.300	0.400	0.400	0.666
MMMUArtTheory	0.533	0.666	0.833	0.800	0.866	0.733	0.833	0.900	0.733	0.300	0.800	0.866	0.833	0.433	0.500	0.766	0.633	0.866	0.700	0.633	0.633	0.600	0.766	0.733	0.500	0.566	0.500	0.733	0.700	0.766	0.833	0.733	0.733	0.800	0.833	0.900	0.933	0.800	0.400	0.266	0.266	0.800
MMMU_Art	0.566	0.566	0.700	0.600	0.766	0.833	0.733	0.633	0.666	0.433	0.766	0.633	0.633	0.533	0.466	0.700	0.533	0.633	0.666	0.533	0.600	0.666	0.666	0.666	0.366	0.400	0.500	0.566	0.533	0.700	0.666	0.500	0.466	0.633	0.666	0.633	0.666	0.666	0.366	0.233	0.200	0.700
MMMUBasicMedical_Science	0.333	0.533	0.633	0.566	0.566	0.666	0.666	0.566	0.666	0.400	0.700	0.666	0.666	0.400	0.433	0.700	0.600	0.733	0.600	0.433	0.433	0.533	0.533	0.500	0.300	0.500	0.500	0.500	0.633	0.600	0.633	0.566	0.633	0.666	0.733	0.666	0.666	0.700	0.466	0.566	0.333	0.866
MMMU_Biology	0.300	0.166	0.300	0.366	0.366	0.600	0.566	0.600	0.500	0.266	0.500	0.500	0.466	0.333	0.233	0.300	0.400	0.400	0.433	0.366	0.333	0.233	0.366	0.433	0.200	0.233	0.233	0.400	0.500	0.533	0.466	0.366	0.433	0.400	0.500	0.500	0.500	0.466	0.433	0.466	0.633	0.666
MMMU_Chemistry	0.033	0.266	0.333	0.233	0.333	0.533	0.500	0.466	0.433	0.233	0.433	0.366	0.433	0.233	0.233	0.400	0.333	0.366	0.366	0.133	0.233	0.100	0.300	0.266	0.333	0.266	0.333	0.300	0.333	0.366	0.366	0.266	0.366	0.300	0.266	0.366	0.466	0.433	0.300	0.333	0.233	0.500
MMMUClinicalMedicine	0.066	0.466	0.533	0.533	0.600	0.633	0.766	0.666	0.466	0.300	0.633	0.633	0.633	0.533	0.500	0.600	0.633	0.633	0.733	0.500	0.466	0.466	0.533	0.566	0.266	0.466	0.433	0.733	0.700	0.566	0.700	0.533	0.533	0.666	0.666	0.566	0.866	0.733	0.266	0.366	0.133	0.766
MMMUComputerScience	0.400	0.466	0.466	0.500	0.600	0.666	0.700	0.666	0.533	0.233	0.533	0.500	0.500	0.366	0.300	0.333	0.333	0.400	0.433	0.433	0.366	0.433	0.566	0.533	0.300	0.300	0.300	0.500	0.466	0.466	0.466	0.266	0.466	0.400	0.500	0.566	0.666	0.533	0.666	0.533	0.466	0.666
MMMU_Design	0.633	0.766	0.733	0.833	0.766	0.733	0.800	0.833	0.800	0.466	0.866	0.766	0.833	0.633	0.600	0.733	0.566	0.666	0.800	0.666	0.633	0.533	0.633	0.800	0.400	0.666	0.633	0.766	0.700	0.766	0.833	0.533	0.766	0.766	0.800	0.800	0.833	0.766	0.433	0.333	0.200	0.800
MMMUDiagnosticsandLaboratoryMedicine	0.100	0.200	0.300	0.300	0.233	0.366	0.400	0.466	0.400	0.200	0.433	0.466	0.500	0.300	0.233	0.366	0.433	0.400	0.433	0.300	0.133	0.433	0.333	0.366	0.233	0.366	0.266	0.300	0.300	0.466	0.366	0.200	0.366	0.400	0.366	0.400	0.433	0.466	0.233	0.400	0.433	0.466
MMMU_Economics	0.466	0.533	0.500	0.566	0.600	0.700	0.733	0.800	0.733	0.366	0.766	0.766	0.700	0.666	0.566	0.600	0.700	0.666	0.766	0.433	0.700	0.566	0.500	0.500	0.466	0.500	0.433	0.666	0.633	0.700	0.466	0.600	0.800	0.533	0.766	0.733	0.700	0.766	0.600	0.733	0.566	0.866
MMMU_Electronics	0.066	0.133	0.233	0.233	0.400	0.466	0.433	0.400	0.333	0.166	0.466	0.466	0.466	0.400	0.200	0.433	0.366	0.400	0.400	0.266	0.333	0.233	0.366	0.300	0.166	0.100	0.266	0.366	0.333	0.366	0.433	0.066	0.166	0.233	0.266	0.266	0.366	0.266	0.366	0.333	0.300	0.433
MMMUEnergyand_Power	0.333	0.233	0.400	0.300	0.500	0.533	0.700	0.600	0.600	0.066	0.600	0.633	0.733	0.366	0.333	0.566	0.533	0.500	0.400	0.466	0.300	0.466	0.366	0.500	0.266	0.333	0.300	0.333	0.300	0.633	0.566	0.100	0.366	0.366	0.433	0.366	0.633	0.533	0.366	0.433	0.333	0.666
MMMU_Finance	0.333	0.333	0.466	0.433	0.533	0.566	0.600	0.566	0.566	0.100	0.500	0.433	0.500	0.433	0.366	0.533	0.533	0.500	0.466	0.366	0.400	0.366	0.466	0.500	0.366	0.233	0.333	0.433	0.533	0.466	0.366	0.300	0.500	0.500	0.500	0.500	0.566	0.566	0.533	0.600	0.466	0.666
MMMU_Geography	0.200	0.266	0.333	0.400	0.366	0.566	0.566	0.500	0.500	0.200	0.533	0.566	0.500	0.400	0.366	0.600	0.466	0.533	0.433	0.433	0.433	0.400	0.333	0.466	0.200	0.233	0.233	0.366	0.400	0.566	0.566	0.266	0.433	0.433	0.333	0.400	0.400	0.500	0.400	0.533	0.200	0.566
MMMU_History	0.566	0.633	0.733	0.833	0.800	0.766	0.866	0.766	0.733	0.433	0.800	0.666	0.633	0.466	0.400	0.633	0.666	0.500	0.700	0.566	0.500	0.666	0.600	0.733	0.466	0.433	0.700	0.633	0.566	0.833	0.800	0.566	0.600	0.666	0.700	0.766	0.833	0.700	0.600	0.366	0.366	0.833
MMMU_Literature	0.666	0.766	0.866	0.833	0.900	0.833	0.833	0.866	0.733	0.666	0.800	0.866	0.833	0.766	0.733	0.866	0.800	0.866	0.866	0.600	0.733	0.533	0.800	0.800	0.766	0.766	0.800	0.766	0.800	0.766	0.833	0.733	0.700	0.833	0.800	0.833	0.866	0.833	0.766	0.266	0.200	0.900
MMMU_Manage	0.233	0.333	0.333	0.433	0.500	0.500	0.533	0.500	0.500	0.233	0.500	0.466	0.500	0.433	0.466	0.533	0.566	0.466	0.566	0.400	0.400	0.466	0.400	0.466	0.400	0.300	0.366	0.333	0.400	0.600	0.533	0.200	0.400	0.466	0.433	0.366	0.600	0.466	0.533	0.533	0.466	0.633
MMMU_Marketing	0.333	0.400	0.466	0.566	0.666	0.866	0.933	0.766	0.766	0.266	0.800	0.766	0.700	0.666	0.566	0.633	0.733	0.633	0.700	0.466	0.466	0.433	0.666	0.500	0.266	0.233	0.266	0.566	0.533	0.800	0.700	0.533	0.733	0.733	0.833	0.900	0.833	0.900	0.666	0.800	0.700	0.866
MMMU_Materials	0.133	0.233	0.133	0.300	0.300	0.500	0.400	0.533	0.433	0.066	0.533	0.466	0.466	0.333	0.266	0.300	0.366	0.300	0.366	0.333	0.266	0.233	0.333	0.333	0.300	0.166	0.200	0.400	0.400	0.400	0.266	0.100	0.266	0.233	0.233	0.300	0.533	0.400	0.333	0.366	0.300	0.533
MMMU_Math	0.300	0.400	0.533	0.433	0.566	0.700	0.633	0.633	0.700	0.433	0.566	0.566	0.666	0.500	0.400	0.566	0.600	0.466	0.566	0.366	0.333	0.433	0.433	0.433	0.366	0.266	0.366	0.500	0.366	0.366	0.566	0.300	0.500	0.500	0.566	0.566	0.600	0.533	0.600	0.766	0.566	0.666
MMMUMechanicalEngineering	0.166	0.166	0.300	0.300	0.333	0.433	0.566	0.566	0.366	0.300	0.733	0.600	0.600	0.366	0.366	0.566	0.466	0.433	0.466	0.233	0.266	0.366	0.300	0.333	0.233	0.366	0.400	0.466	0.366	0.366	0.533	0.100	0.200	0.366	0.333	0.400	0.600	0.433	0.266	0.500	0.366	0.600
MMMU_Music	0.166	0.333	0.200	0.133	0.333	0.366	0.366	0.400	0.500	0.300	0.233	0.433	0.466	0.333	0.433	0.333	0.366	0.400	0.433	0.266	0.300	0.233	0.300	0.266	0.566	0.500	0.466	0.433	0.333	0.333	0.466	0.200	0.166	0.300	0.166	0.300	0.366	0.266	0.066	0.266	0.066	0.366
MMMU_Pharmacy	0.333	0.366	0.600	0.633	0.633	0.700	0.733	0.800	0.700	0.266	0.766	0.733	0.700	0.500	0.533	0.633	0.733	0.566	0.700	0.433	0.433	0.500	0.500	0.466	0.433	0.366	0.400	0.633	0.600	0.600	0.800	0.366	0.500	0.666	0.700	0.733	0.866	0.766	0.600	0.600	0.633	0.833
MMMU_Physics	0.166	0.300	0.433	0.466	0.600	0.766	0.866	0.833	0.733	0.266	0.666	0.666	0.700	0.500	0.600	0.533	0.600	0.433	0.600	0.400	0.466	0.400	0.500	0.466	0.333	0.433	0.500	0.466	0.400	0.666	0.766	0.333	0.566	0.800	0.800	0.766	0.866	0.833	0.733	0.833	0.633	0.900
MMMU_Psychology	0.366	0.433	0.500	0.566	0.566	0.666	0.700	0.633	0.600	0.266	0.566	0.633	0.566	0.433	0.366	0.666	0.533	0.633	0.466	0.533	0.500	0.433	0.600	0.600	0.400	0.433	0.500	0.566	0.500	0.566	0.533	0.466	0.533	0.666	0.633	0.566	0.666	0.666	0.433	0.300	0.400	0.733
MMMUPublicHealth	0.433	0.700	0.700	0.766	0.800	0.900	0.933	0.866	0.766	0.200	0.866	0.766	0.800	0.733	0.600	0.800	0.800	0.766	0.800	0.666	0.666	0.600	0.733	0.666	0.433	0.400	0.400	0.700	0.733	0.866	0.766	0.500	0.800	0.866	0.733	0.866	0.900	0.866	0.700	0.800	0.700	0.966
MMMU_Sociology	0.366	0.600	0.600	0.600	0.700	0.800	0.700	0.833	0.600	0.300	0.666	0.566	0.600	0.566	0.566	0.633	0.700	0.533	0.733	0.566	0.533	0.466	0.500	0.500	0.466	0.566	0.466	0.500	0.500	0.666	0.600	0.566	0.566	0.533	0.600	0.633	0.700	0.600	0.633	0.433	0.533	0.800
MMMU	0.318	0.407	0.487	0.494	0.558	0.635	0.663	0.657	0.590	0.287	0.628	0.611	0.615	0.463	0.435	0.557	0.552	0.535	0.575	0.438	0.430	0.425	0.493	0.496	0.365	0.381	0.413	0.508	0.501	0.580	0.585	0.375	0.497	0.537	0.553	0.571	0.660	0.610	0.471	0.478	0.394	0.708
MMMUPRO_Accounting	0.224	0.310	0.534	0.534	0.603	0.689	0.827	0.775	0.689	0.103	0.741	0.655	0.672	0.500	0.517	0.517	0.637	0.551	0.586	0.224	0.293	0.344	0.396	0.413	0.379	0.327	0.310	0.362	0.465	0.568	0.500	0.362	0.482	0.586	0.637	0.706	0.827	0.689	0.568	0.706	0.603	0.827
MMMUPRO_Agriculture	0.200	0.200	0.350	0.366	0.450	0.383	0.450	0.500	0.183	0.183	0.383	0.266	0.316	0.250	0.183	0.300	0.333	0.283	0.266	0.216	0.200	0.216	0.216	0.166	0.116	0.200	0.283	0.150	0.216	0.266	0.400	0.183	0.283	0.316	0.350	0.383	0.433	0.333	0.400	0.366	0.300	0.500
MMMUPROArchitectureand_Engineering	0.100	0.133	0.216	0.200	0.333	0.433	0.516	0.533	0.266	0.100	0.433	0.350	0.283	0.250	0.150	0.400	0.383	0.316	0.366	0.200	0.250	0.233	0.266	0.350	0.183	0.100	0.183	0.383	0.216	0.300	0.166	0.066	0.183	0.166	0.266	0.233	0.383	0.433	0.250	0.333	0.300	0.516
MMMUPROArtTheory	0.472	0.490	0.636	0.636	0.709	0.490	0.654	0.781	0.563	0.163	0.672	0.709	0.709	0.345	0.418	0.654	0.654	0.618	0.527	0.509	0.527	0.509	0.545	0.618	0.345	0.436	0.400	0.600	0.654	0.636	0.763	0.527	0.436	0.672	0.636	0.672	0.727	0.618	0.618	0.454	0.509	0.781
MMMUPRO_Art	0.396	0.452	0.547	0.622	0.622	0.528	0.566	0.660	0.509	0.207	0.603	0.622	0.622	0.415	0.415	0.452	0.490	0.471	0.528	0.452	0.528	0.433	0.528	0.566	0.283	0.415	0.339	0.433	0.528	0.490	0.641	0.471	0.471	0.509	0.528	0.547	0.641	0.509	0.490	0.377	0.415	0.641
MMMUPROBasicMedical_Science	0.269	0.250	0.384	0.307	0.442	0.500	0.557	0.538	0.480	0.230	0.596	0.326	0.442	0.192	0.173	0.307	0.423	0.384	0.403	0.173	0.173	0.192	0.307	0.307	0.192	0.134	0.211	0.365	0.346	0.442	0.384	0.269	0.480	0.403	0.461	0.461	0.557	0.365	0.384	0.384	0.557	0.673
MMMUPRO_Biology	0.169	0.237	0.288	0.338	0.322	0.474	0.525	0.542	0.322	0.152	0.423	0.389	0.406	0.118	0.186	0.406	0.338	0.355	0.372	0.254	0.271	0.203	0.254	0.254	0.118	0.084	0.152	0.288	0.355	0.474	0.406	0.288	0.355	0.322	0.389	0.423	0.457	0.288	0.322	0.322	0.355	0.576
MMMUPRO_Chemistry	0.200	0.266	0.333	0.266	0.350	0.450	0.500	0.533	0.383	0.200	0.366	0.350	0.466	0.266	0.216	0.300	0.433	0.383	0.450	0.150	0.300	0.300	0.283	0.250	0.316	0.250	0.283	0.333	0.333	0.300	0.450	0.266	0.216	0.300	0.350	0.366	0.583	0.383	0.416	0.333	0.283	0.566
MMMUPROClinicalMedicine	0.118	0.135	0.237	0.338	0.372	0.457	0.440	0.491	0.237	0.101	0.322	0.372	0.423	0.203	0.203	0.322	0.389	0.474	0.389	0.237	0.237	0.254	0.220	0.186	0.101	0.169	0.152	0.237	0.254	0.491	0.406	0.118	0.271	0.423	0.457	0.440	0.559	0.440	0.355	0.423	0.322	0.627
MMMUPROComputerScience	0.283	0.350	0.383	0.283	0.300	0.650	0.600	0.516	0.433	0.133	0.483	0.466	0.500	0.250	0.283	0.433	0.350	0.333	0.383	0.400	0.283	0.250	0.316	0.350	0.266	0.200	0.200	0.416	0.350	0.416	0.483	0.150	0.366	0.416	0.450	0.383	0.583	0.533	0.516	0.483	0.500	0.633
MMMUPRO_Design	0.433	0.500	0.533	0.500	0.616	0.666	0.633	0.766	0.500	0.150	0.616	0.700	0.666	0.400	0.350	0.583	0.566	0.533	0.616	0.383	0.383	0.500	0.616	0.616	0.350	0.533	0.416	0.550	0.533	0.683	0.750	0.450	0.516	0.633	0.650	0.733	0.700	0.616	0.500	0.516	0.450	0.650
MMMUPRODiagnosticsandLaboratoryMedicine	0.116	0.200	0.200	0.216	0.233	0.316	0.316	0.383	0.316	0.150	0.383	0.333	0.350	0.250	0.166	0.283	0.333	0.200	0.300	0.233	0.166	0.100	0.166	0.166	0.083	0.116	0.133	0.150	0.216	0.200	0.233	0.150	0.216	0.300	0.366	0.266	0.316	0.300	0.316	0.433	0.300	0.500
MMMUPRO_Economics	0.423	0.457	0.559	0.525	0.644	0.762	0.796	0.813	0.711	0.118	0.677	0.610	0.627	0.491	0.372	0.508	0.677	0.661	0.627	0.440	0.355	0.406	0.491	0.440	0.288	0.338	0.288	0.457	0.508	0.627	0.610	0.457	0.694	0.728	0.762	0.694	0.847	0.762	0.644	0.728	0.644	0.881
MMMUPRO_Electronics	0.233	0.316	0.350	0.350	0.316	0.633	0.700	0.683	0.500	0.050	0.616	0.516	0.483	0.316	0.316	0.516	0.650	0.600	0.600	0.366	0.366	0.300	0.516	0.483	0.200	0.200	0.200	0.450	0.433	0.533	0.466	0.033	0.250	0.250	0.366	0.316	0.633	0.400	0.416	0.533	0.500	0.716
MMMUPROEnergyand_Power	0.172	0.172	0.120	0.206	0.327	0.517	0.482	0.551	0.431	0.051	0.568	0.448	0.500	0.224	0.224	0.327	0.500	0.224	0.275	0.086	0.172	0.137	0.206	0.189	0.137	0.086	0.155	0.224	0.293	0.310	0.396	0.086	0.206	0.310	0.258	0.344	0.517	0.448	0.275	0.327	0.362	0.534
MMMUPRO_Finance	0.283	0.366	0.533	0.450	0.516	0.833	0.883	0.883	0.766	0.133	0.633	0.633	0.716	0.516	0.450	0.666	0.683	0.600	0.650	0.333	0.416	0.350	0.533	0.500	0.283	0.250	0.333	0.450	0.450	0.666	0.500	0.300	0.500	0.616	0.666	0.616	0.833	0.750	0.566	0.750	0.683	0.900
MMMUPRO_Geography	0.346	0.307	0.346	0.307	0.403	0.500	0.500	0.442	0.461	0.115	0.480	0.403	0.403	0.250	0.211	0.384	0.365	0.384	0.384	0.269	0.269	0.307	0.365	0.250	0.211	0.192	0.192	0.326	0.346	0.365	0.365	0.250	0.269	0.365	0.365	0.365	0.403	0.384	0.288	0.423	0.384	0.519
MMMUPRO_History	0.375	0.392	0.535	0.571	0.553	0.642	0.625	0.660	0.517	0.125	0.607	0.571	0.589	0.464	0.392	0.500	0.535	0.428	0.553	0.375	0.375	0.339	0.500	0.553	0.214	0.428	0.428	0.482	0.482	0.607	0.571	0.267	0.482	0.589	0.446	0.589	0.678	0.571	0.571	0.446	0.446	0.625
MMMUPRO_Literature	0.500	0.461	0.634	0.692	0.692	0.730	0.634	0.769	0.557	0.365	0.730	0.673	0.673	0.461	0.480	0.673	0.615	0.615	0.634	0.557	0.519	0.519	0.615	0.576	0.500	0.634	0.596	0.615	0.673	0.673	0.692	0.519	0.576	0.692	0.692	0.615	0.788	0.711	0.596	0.403	0.365	0.769
MMMUPRO_Manage	0.220	0.240	0.320	0.360	0.400	0.620	0.620	0.660	0.600	0.160	0.480	0.460	0.500	0.320	0.280	0.460	0.460	0.420	0.420	0.280	0.260	0.240	0.300	0.380	0.260	0.340	0.260	0.320	0.300	0.380	0.500	0.220	0.360	0.480	0.420	0.380	0.580	0.580	0.560	0.520	0.420	0.680
MMMUPRO_Marketing	0.288	0.305	0.440	0.508	0.440	0.711	0.813	0.762	0.745	0.101	0.627	0.627	0.677	0.491	0.423	0.508	0.593	0.525	0.593	0.254	0.288	0.322	0.406	0.389	0.338	0.271	0.322	0.474	0.389	0.593	0.559	0.423	0.627	0.661	0.711	0.711	0.711	0.745	0.610	0.762	0.745	0.898
MMMUPRO_Materials	0.083	0.133	0.150	0.133	0.250	0.433	0.366	0.466	0.383	0.083	0.316	0.333	0.266	0.166	0.150	0.266	0.300	0.166	0.300	0.216	0.166	0.116	0.183	0.200	0.100	0.083	0.100	0.216	0.250	0.216	0.216	0.083	0.216	0.250	0.316	0.266	0.483	0.333	0.250	0.350	0.183	0.416
MMMUPRO_Math	0.283	0.233	0.316	0.333	0.466	0.616	0.583	0.650	0.483	0.150	0.483	0.516	0.533	0.350	0.316	0.350	0.533	0.416	0.466	0.350	0.300	0.266	0.250	0.266	0.216	0.200	0.133	0.400	0.366	0.333	0.500	0.216	0.483	0.550	0.483	0.450	0.700	0.583	0.466	0.550	0.566	0.733
MMMUPROMechanicalEngineering	0.152	0.186	0.271	0.305	0.305	0.440	0.491	0.627	0.338	0.152	0.474	0.355	0.271	0.237	0.271	0.406	0.322	0.440	0.372	0.186	0.203	0.203	0.237	0.322	0.152	0.169	0.169	0.237	0.355	0.372	0.372	0.135	0.254	0.169	0.288	0.338	0.423	0.372	0.271	0.355	0.254	0.733
MMMUPRO_Music	0.216	0.250	0.266	0.150	0.233	0.300	0.316	0.233	0.250	0.183	0.183	0.266	0.233	0.266	0.266	0.200	0.216	0.183	0.233	0.283	0.300	0.266	0.316	0.233	0.200	0.150	0.200	0.200	0.283	0.266	0.283	0.233	0.150	0.216	0.150	0.300	0.233	0.250	0.150	0.233	0.116	0.333
MMMUPRO_Pharmacy	0.298	0.298	0.456	0.421	0.491	0.614	0.649	0.649	0.543	0.298	0.596	0.631	0.596	0.315	0.385	0.543	0.614	0.508	0.684	0.315	0.368	0.263	0.333	0.421	0.333	0.280	0.333	0.403	0.421	0.631	0.614	0.315	0.614	0.578	0.561	0.578	0.736	0.543	0.543	0.614	0.473	0.701
MMMUPRO_Physics	0.166	0.116	0.416	0.416	0.400	0.650	0.616	0.700	0.550	0.133	0.533	0.450	0.533	0.350	0.316	0.483	0.450	0.466	0.466	0.283	0.250	0.266	0.366	0.333	0.200	0.183	0.216	0.266	0.283	0.466	0.550	0.416	0.483	0.516	0.566	0.516	0.650	0.600	0.633	0.683	0.466	0.750
MMMUPRO_Psychology	0.366	0.333	0.300	0.333	0.383	0.566	0.600	0.466	0.450	0.200	0.500	0.450	0.400	0.316	0.200	0.483	0.450	0.350	0.416	0.283	0.283	0.300	0.316	0.333	0.200	0.200	0.266	0.316	0.300	0.383	0.416	0.216	0.450	0.483	0.516	0.450	0.483	0.466	0.350	0.400	0.316	0.600
MMMUPROPublicHealth	0.241	0.293	0.396	0.534	0.551	0.741	0.724	0.741	0.603	0.086	0.758	0.620	0.655	0.327	0.258	0.568	0.637	0.482	0.551	0.172	0.327	0.327	0.517	0.431	0.120	0.086	0.189	0.551	0.534	0.741	0.431	0.310	0.517	0.586	0.603	0.672	0.810	0.741	0.620	0.603	0.689	0.775
MMMUPRO_Sociology	0.333	0.462	0.574	0.518	0.629	0.648	0.574	0.703	0.500	0.314	0.592	0.555	0.537	0.592	0.462	0.444	0.537	0.574	0.518	0.407	0.425	0.407	0.500	0.407	0.333	0.425	0.314	0.481	0.407	0.611	0.555	0.407	0.611	0.462	0.518	0.629	0.648	0.611	0.518	0.500	0.388	0.703
MMMUPRO	0.263	0.293	0.384	0.387	0.442	0.565	0.584	0.615	0.473	0.154	0.527	0.487	0.500	0.328	0.300	0.440	0.481	0.430	0.463	0.294	0.306	0.294	0.367	0.363	0.232	0.246	0.256	0.369	0.382	0.466	0.470	0.270	0.398	0.449	0.473	0.480	0.596	0.510	0.447	0.478	0.429	0.658

DISCIPLINES NLP | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | MATH | 0.305 | 0.338 | 0.400 | 0.361 | 0.450 | 0.650 | 0.616 | 0.605 | 0.511 | 0.205 | 0.505 | 0.505 | 0.538 | 0.344 | 0.316 | 0.411 | 0.450 | 0.394 | 0.450 | 0.383 | 0.311 | 0.316 | 0.355 | 0.366 | 0.272 | 0.227 | 0.222 | 0.438 | 0.377 | 0.388 | 0.500 | 0.216 | 0.444 | 0.472 | 0.488 | 0.466 | 0.638 | 0.550 | 0.538 | 0.561 | 0.527 | 0.677 | SCIENCE | 0.178 | 0.218 | 0.318 | 0.329 | 0.378 | 0.514 | 0.521 | 0.570 | 0.418 | 0.173 | 0.452 | 0.414 | 0.443 | 0.273 | 0.260 | 0.369 | 0.400 | 0.354 | 0.400 | 0.267 | 0.271 | 0.233 | 0.305 | 0.285 | 0.204 | 0.209 | 0.256 | 0.298 | 0.329 | 0.398 | 0.432 | 0.267 | 0.351 | 0.380 | 0.423 | 0.432 | 0.547 | 0.438 | 0.423 | 0.425 | 0.342 | 0.592 | ENGINEERING | 0.239 | 0.272 | 0.337 | 0.337 | 0.409 | 0.532 | 0.574 | 0.624 | 0.445 | 0.154 | 0.563 | 0.507 | 0.501 | 0.331 | 0.302 | 0.472 | 0.476 | 0.442 | 0.463 | 0.299 | 0.304 | 0.310 | 0.387 | 0.411 | 0.237 | 0.257 | 0.279 | 0.400 | 0.387 | 0.467 | 0.478 | 0.161 | 0.306 | 0.337 | 0.387 | 0.409 | 0.550 | 0.472 | 0.344 | 0.409 | 0.355 | 0.631 | MEDICINE | 0.222 | 0.309 | 0.408 | 0.431 | 0.467 | 0.568 | 0.591 | 0.598 | 0.490 | 0.206 | 0.580 | 0.525 | 0.550 | 0.339 | 0.314 | 0.479 | 0.534 | 0.481 | 0.529 | 0.309 | 0.314 | 0.323 | 0.383 | 0.373 | 0.222 | 0.247 | 0.270 | 0.419 | 0.435 | 0.541 | 0.495 | 0.300 | 0.467 | 0.525 | 0.541 | 0.538 | 0.646 | 0.557 | 0.447 | 0.511 | 0.458 | 0.697 | HUMANITIES | 0.392 | 0.441 | 0.517 | 0.516 | 0.571 | 0.599 | 0.607 | 0.621 | 0.529 | 0.262 | 0.577 | 0.571 | 0.557 | 0.423 | 0.397 | 0.533 | 0.517 | 0.508 | 0.524 | 0.434 | 0.445 | 0.423 | 0.497 | 0.494 | 0.347 | 0.401 | 0.403 | 0.478 | 0.485 | 0.552 | 0.577 | 0.409 | 0.461 | 0.535 | 0.517 | 0.557 | 0.608 | 0.552 | 0.448 | 0.378 | 0.334 | 0.651 | BUSINESS | 0.309 | 0.360 | 0.477 | 0.486 | 0.550 | 0.706 | 0.768 | 0.750 | 0.681 | 0.169 | 0.653 | 0.619 | 0.639 | 0.495 | 0.444 | 0.550 | 0.619 | 0.552 | 0.603 | 0.346 | 0.373 | 0.369 | 0.456 | 0.451 | 0.344 | 0.314 | 0.327 | 0.449 | 0.465 | 0.591 | 0.534 | 0.378 | 0.559 | 0.598 | 0.635 | 0.616 | 0.738 | 0.701 | 0.587 | 0.692 | 0.600 | 0.821 | LAW | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | VISION | 0.498 | 0.500 | 0.628 | 0.688 | 0.714 | 0.702 | 0.727 | 0.818 | 0.826 | 0.635 | 0.780 | 0.815 | 0.813 | 0.720 | 0.746 | 0.806 | 0.782 | 0.790 | 0.773 | 0.805 | 0.807 | 0.800 | 0.811 | 0.811 | 0.780 | 0.805 | 0.808 | 0.816 | 0.858 | 0.810 | 0.777 | 0.736 | 0.800 | 0.801 | 0.848 | 0.818 | 0.831 | 0.790 | 0.803 | 0.839 | 0.810 | 0.824 | COMPOSITE AVERAGE AVG | 0.471 | 0.479 | 0.602 | 0.657 | 0.685 | 0.689 | 0.713 | 0.796 | 0.790 | 0.584 | 0.752 | 0.782 | 0.781 | 0.679 | 0.700 | 0.768 | 0.750 | 0.749 | 0.739 | 0.751 | 0.753 | 0.746 | 0.764 | 0.764 | 0.717 | 0.741 | 0.745 | 0.769 | 0.824 | 0.775 | 0.746 | 0.686 | 0.757 | 0.763 | 0.821 | 0.782 | 0.806 | 0.761 | 0.763 | 0.797 | 0.764 | 0.806 |

AUDIO MODEL TESTS:

MODEL	gemma-4-12B-it	gemma-4-12B-it-qat-q4_0-unquantized	Qwen2.5-Omni-3B	Qwen2.5-Omni-3B	Qwen2.5-Omni-3B	Qwen2.5-Omni-7B	Qwen2.5-Omni-7B	Qwen3-Omni-30B-A3B-Instruct	ultravox-v0_5-llama-3_1-8b	ultravox-v0_5-llama-3_1-8b	ultravox-v0_5-deepseek-r1-llama-3_1-8b	ultravox-v0_5-deepseek-r1-llama-3_1-8b	ultravox-v0_6-gemma-3-27b	ultravox-v0_6-qwen-3-32b	Voxtral-Mini-3B-2507	Voxtral-Mini-3B-2507	Voxtral-Small-24B-2507
params	11.91B	11.91B	3.09B	3.09B	3.09B	7.62B	7.62B	30.53B	8.03B	8.03B	8.03B	8.03B	27.01B	32.8B	4.01B	4.01B	23.57B
quant	Q4EH	Q4EH	Q4KH	Q6KH	Q80H	Q4KH	Q6KH	Q4KH	Q4KH	Q6KH	Q6_K	Q6KH	Q4KH	Q4KH	Q6_K	Q6KH	Q4KH
engine	llama.cpp version: 9524	llama.cpp version: 9632	llama.cpp version: 7154	llama.cpp version: 7154	llama.cpp version: 7003	llama.cpp version: 7130	llama.cpp version: 6937	llama.cpp version: 8786	llama.cpp version: 7310	llama.cpp version: 7310	llama.cpp version: 5869	llama.cpp version: 7484	llama.cpp version: 5853	llama.cpp version: 5853	llama.cpp version: 6014	llama.cpp version: 7410	llama.cpp version: 7330
TEST	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc	acc
BBAformalfallacies	0.956	0.980	0.564	0.580	0.624	0.516	0.516	0.876	0.548	0.596	0.768	0.872	0.640	0.996	0.544	0.528	0.616
BBA_navigate	1.000	1.000	0.660	0.636	0.676	0.760	0.748	0.976	0.728	0.688	0.988	0.992	0.716	0.976	0.664	0.636	0.688
BBAobjectcounting	0.936	0.940	0.652	0.656	0.692	0.676	0.648	0.980	0.736	0.844	0.924	1.000	0.800	0.984	0.596	0.560	0.524
BBAwebof_lies	0.828	0.916	0.648	0.636	0.608	0.536	0.592	0.856	0.716	0.756	0.932	0.880	0.464	0.784	0.576	0.636	0.668
BBA	0.930	0.959	0.631	0.627	0.650	0.622	0.626	0.922	0.682	0.721	0.903	0.936	0.655	0.935	0.595	0.590	0.624
BBHAformalfallacies	0.992	0.992	-	-	-	-	-	0.988	0.628	-	-	0.868	-	-	-	0.540	-
BBHA_navigate	1.000	1.000	-	-	-	-	-	0.996	0.652	-	-	0.992	-	-	-	0.688	-
BBHAobjectcounting	1.000	0.992	-	-	-	-	-	0.996	0.772	-	-	1.000	-	-	-	0.700	-
BBHAwebof_lies	0.996	0.996	-	-	-	-	-	0.988	0.844	-	-	0.952	-	-	-	0.760	-
BBHA	0.997	0.995	-	-	-	-	-	0.992	0.724	-	-	0.953	-	-	-	0.672	-

DISCIPLINES NLP | 0.892 | 0.948 | 0.606 | 0.608 | 0.616 | 0.526 | 0.554 | 0.866 | 0.632 | 0.676 | 0.850 | 0.876 | 0.552 | 0.890 | 0.560 | 0.582 | 0.642 | MATH | 0.968 | 0.970 | 0.656 | 0.646 | 0.684 | 0.718 | 0.698 | 0.978 | 0.732 | 0.766 | 0.956 | 0.996 | 0.758 | 0.980 | 0.630 | 0.598 | 0.606 | SCIENCE | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | ENGINEERING | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | MEDICINE | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | HUMANITIES | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | BUSINESS | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | LAW | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | AUDIO | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | COMPOSITE AVERAGE AVG | 0.930 | 0.959 | 0.631 | 0.627 | 0.650 | 0.622 | 0.626 | 0.922 | 0.682 | 0.721 | 0.903 | 0.936 | 0.655 | 0.935 | 0.595 | 0.590 | 0.624 |

MT MODEL TESTS:

MODEL	HY-MT1.5-7B	HY-MT2-7B	madlad400-7b-mt	madlad400-10b-mt	madlad400-10b-mt	plamo-2-translate	translategemma-4b-it	translategemma-12b-it	translategemma-27b-it
params	7.50B	7.50B	8.30B	10.71B	10.71B	9.53B	3.88B	11.77B	27.01B
quant	Q6EH	Q6EH	Q4KH	Q4KH	Q6KH	Q6KH	Q6KH	Q4EH	Q4KH
engine	llama.cpp version: 9745	llama.cpp version: 9761	llama.cpp version: 7885	llama.cpp version: 7830	llama.cpp version: 7772	llama.cpp version: 7762	llama.cpp version: 7760	llama.cpp version: 7779	llama.cpp version: 7789
TEST	acc	acc	acc	acc	acc	acc	acc	acc	acc
FLORES200deen	33.1	40.1	42.8	42.3	42.3	35.5	37.8	39.7	42.2
FLORES200ende	31.7	35.7	37.8	36.6	36.9	24.1	31.0	34.4	35.3
FLORES200enes	25.4	27.6	26.8	26.5	26.5	21.3	25.1	27.1	27.2
FLORES200enfr	37.1	43.4	50.0	49.7	49.8	36.5	40.9	44.7	45.2
FLORES200enja	30.0	34.1	22.8	22.2	22.3	28.2	27.7	30.2	31.0
FLORES200enru	22.7	28.6	29.5	28.7	28.8	18.2	25.4	27.9	28.2
FLORES200enzh	37.1	42.4	37.2	37.2	37.3	33.6	36.6	39.3	41.9
FLORES200esen	24.0	28.7	29.4	29.1	29.2	24.7	28.0	28.6	29.9
FLORES200fren	33.3	41.0	44.3	44.1	44.3	36.4	38.8	41.7	43.4
FLORES200jaen	21.0	24.8	26.8	25.9	25.9	23.4	21.5	23.7	26.6
FLORES200ruen	27.2	33.5	34.9	34.7	34.9	29.0	30.7	32.3	34.8
FLORES200zhen	23.2	27.7	28.3	27.3	27.3	23.4	23.5	25.3	27.8
FLORES200	28.8	34.0	34.2	33.7	33.8	27.9	30.6	32.9	34.4
OPUSdeen	26.5	31.3	35.6	35.6	27.1	21.3	28.2	28.9	30.5
OPUSende	23.9	28.0	33.2	32.9	30.7	21.1	25.5	26.1	27.1
OPUSenes	29.0	34.3	37.7	37.1	37.1	27.9	31.1	31.7	32.4
OPUSenfr	24.6	28.2	34.3	34.2	34.3	26.7	27.0	28.7	29.0
OPUSenja	10.5	11.5	16.0	16.0	15.9	11.9	9.8	10.5	11.7
OPUSenru	22.1	26.5	31.4	31.7	31.6	19.7	23.1	24.3	25.5
OPUSenzh	28.8	32.7	41.2	41.1	41.3	26.7	26.4	28.3	29.9
OPUSesen	29.2	35.4	40.6	40.2	40.3	26.7	33.2	34.6	36.4
OPUSfren	26.3	31.3	36.4	35.8	35.8	29.6	29.2	30.7	32.3
OPUSjaen	14.8	16.5	19.4	18.9	18.8	16.0	15.1	16.1	16.8
OPUSruen	25.0	30.2	35.2	34.7	34.8	27.1	27.1	28.0	30.2
OPUSzhen	26.5	31.2	39.4	38.5	38.8	23.8	22.1	24.6	27.6
OPUS	23.9	28.1	33.4	33.1	32.2	23.2	24.8	26.0	27.4
DE_EN	28.7	34.2	38.0	37.8	32.1	26.0	31.4	32.5	34.4
EN_DE	26.5	30.6	34.7	34.1	32.7	22.1	27.3	28.8	29.8
ES_EN	27.4	33.1	36.8	36.4	36.5	26.0	31.4	32.5	34.2
EN_ES	27.7	32.0	34.0	33.5	33.5	25.6	29.0	30.1	30.6
FR_EN	28.6	34.5	39.0	38.6	38.6	31.9	32.4	34.3	36.0
EN_FR	28.8	33.2	39.5	39.4	39.5	30.0	31.6	34.0	34.4
RU_EN	25.7	31.2	35.1	34.7	34.8	27.7	28.2	29.4	31.7
EN_RU	22.3	27.1	30.7	30.7	30.6	19.1	23.8	25.5	26.3
JA_EN	16.8	19.2	21.8	21.2	21.2	18.4	17.2	18.6	20.1
EN_JA	17.0	19.0	18.2	18.0	18.0	17.3	15.8	17.1	18.1
ZH_EN	25.3	29.9	35.6	34.7	34.9	23.6	22.5	24.8	27.6
EN_ZH	31.6	35.9	39.8	39.8	39.9	29.0	29.8	32.0	33.9

COMPOSITE AVERAGE AVG | 25.5 | 30.0 | 33.6 | 33.2 | 32.7 | 24.7 | 26.7 | 28.3 | 29.8 |