CoolFace
Modelpublic

ericflo/Llama-3.2-1B-Instruct-RLHF-v0.1

sourceHugging Faceapache-2.0updated 2y agoView on Hugging Face
0likes93downloads
Model Card

This model's benchmark results

TasksVersionFiltern-shotMetricValueStderr
tinyBenchmarksN/A
- tinyArc0none25acc_norm↑0.4253±N/A
- tinyGSM8k0flexible-extract5exact_match↑0.3768±N/A
strict-match5exact_match↑0.3768±N/A
- tinyHellaswag0none10acc_norm↑0.5379±N/A
- tinyMMLU0none0acc_norm↑0.4483±N/A
- tinyTruthfulQA0none0acc↑0.4217±N/A
- tinyWinogrande0none5acc_norm↑0.5366±N/A

Original meta-llama/Llama-3.2-1B-Instruct benchmark results

TasksVersionFiltern-shotMetricValueStderr
tinyBenchmarksN/A
- tinyArc0none25acc_norm↑0.4145±N/A
- tinyGSM8k0flexible-extract5exact_match↑0.3412±N/A
strict-match5exact_match↑0.3412±N/A
- tinyHellaswag0none10acc_norm↑0.5335±N/A
- tinyMMLU0none0acc_norm↑0.4298±N/A
- tinyTruthfulQA0none0acc↑0.4288±N/A
- tinyWinogrande0none5acc_norm↑0.5366±N/A

Below is a side-by-side comparison of the two result sets. For each task, the higher value (i.e., “better” on that metric) is highlighted in bold:

TaskthisorigBetter?
tinyArc (acc_norm)0.42530.4145v1 higher
tinyGSM8k (exact_match)0.37680.3412v1 higher
tinyHellaswag (acc_norm)0.53790.5335v1 higher
tinyMMLU (acc_norm)0.44830.4298v1 higher
tinyTruthfulQA (acc)0.42170.4288v2 higher
tinyWinogrande (acc_norm)0.53660.5366tie

Observations

  1. 1.Ours outperforms the original on four tasks (tinyArc, tinyGSM8k, tinyHellaswag, tinyMMLU).
  2. 2.The original outperforms ours on one task (tinyTruthfulQA).
  3. 3.One task is a tie (tinyWinogrande).

Given these comparisons, our results are stronger overall because it has higher scores on the majority of tasks. The only exception is on tinyTruthfulQA, where the original scores slightly better, and on tinyWinogrande, both versions tie.