kth8/gemma-4-E2B-it-GPQA-Diamond-benchmark
Benchmark of google/gemma-4-E2B-it against GPQA-Diamond dataset. Results are averaged over 4 runs to reduce variance. Accuracy: 39.5% with Python tool. Metric Value Correct 313 Incorrect 478 Errors 1 Total samples 792 Python tool calls 209 Python tool errors 19 Total completion tokens 1,984,858
Benchmark of google/gemma-4-E2B-it against GPQA-Diamond dataset. Results are averaged over 4 runs to reduce variance.
Accuracy: 39.5% with Python tool. | Metric | Value | |----------------------|---------------| | Correct | 313 | | Incorrect | 478 | | Errors | 1 | | Total samples | 792 | | Python tool calls| 209 | | Python tool errors| 19 | | Total completion tokens | 1,984,858 |