sethuiyer/CodeCalc-Mistral-7B

1---2language:3- en4license: apache-2.05library_name: transformers6tags:7- mergekit8- merge9base_model:10- uukuguy/speechless-code-mistral-7b-v1.011- upaya07/Arithmo2-Mistral-7B12pipeline_tag: text-generation13model-index:14- name: sethuiyer/CodeCalc-Mistral-7B15  results:16  - task:17      type: text-generation18      name: Text Generation19    dataset:20      name: AI2 Reasoning Challenge (25-Shot)21      type: ai2_arc22      config: ARC-Challenge23      split: test24      args:25        num_few_shot: 2526    metrics:27    - type: acc_norm28      value: 61.9529      name: normalized accuracy30    source:31      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=sethuiyer/CodeCalc-Mistral-7B32      name: Open LLM Leaderboard33  - task:34      type: text-generation35      name: Text Generation36    dataset:37      name: HellaSwag (10-Shot)38      type: hellaswag39      split: validation40      args:41        num_few_shot: 1042    metrics:43    - type: acc_norm44      value: 83.6445      name: normalized accuracy46    source:47      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=sethuiyer/CodeCalc-Mistral-7B48      name: Open LLM Leaderboard49  - task:50      type: text-generation51      name: Text Generation52    dataset:53      name: MMLU (5-Shot)54      type: cais/mmlu55      config: all56      split: test57      args:58        num_few_shot: 559    metrics:60    - type: acc61      value: 62.7862      name: accuracy63    source:64      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=sethuiyer/CodeCalc-Mistral-7B65      name: Open LLM Leaderboard66  - task:67      type: text-generation68      name: Text Generation69    dataset:70      name: TruthfulQA (0-shot)71      type: truthful_qa72      config: multiple_choice73      split: validation74      args:75        num_few_shot: 076    metrics:77    - type: mc278      value: 47.4979    source:80      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=sethuiyer/CodeCalc-Mistral-7B81      name: Open LLM Leaderboard82  - task:83      type: text-generation84      name: Text Generation85    dataset:86      name: Winogrande (5-shot)87      type: winogrande88      config: winogrande_xl89      split: validation90      args:91        num_few_shot: 592    metrics:93    - type: acc94      value: 78.395      name: accuracy96    source:97      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=sethuiyer/CodeCalc-Mistral-7B98      name: Open LLM Leaderboard99  - task:100      type: text-generation101      name: Text Generation102    dataset:103      name: GSM8k (5-shot)104      type: gsm8k105      config: main106      split: test107      args:108        num_few_shot: 5109    metrics:110    - type: acc111      value: 63.53112      name: accuracy113    source:114      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=sethuiyer/CodeCalc-Mistral-7B115      name: Open LLM Leaderboard116---117# CodeCalc-Mistral-7B118 119<p align="center">120  <img src="https://huggingface.co/sethuiyer/CodeCalc-Mistral-7B/resolve/main/codecalc.webp" height="128px" alt="CodeCalc">121</p>122 123 124### Configuration125 126The following YAML configuration was used to produce this model:127 128```yaml129 130base_model: uukuguy/speechless-code-mistral-7b-v1.0131dtype: bfloat16132merge_method: ties133models:134- model: uukuguy/speechless-code-mistral-7b-v1.0135- model: upaya07/Arithmo2-Mistral-7B136  parameters:137    density:  [0.25, 0.35, 0.45, 0.35, 0.25]138    weight: [0.1, 0.25, 0.5, 0.25, 0.1]139parameters:140  int8_mask: true141 142```143 144 145### Evaluation146 147| T  | Model                                       | Average | ARC  | HellaSwag | MMLU  | TruthfulQA | Winogrande | GSM8K |148|----|---------------------------------------------|---------|------|-----------|-------|------------|------------|-------|149| 🔍  | sethuiyer/CodeCalc-Mistral-7B               | 66.33   | 61.95| 83.64     | 62.78 | 47.79      | 78.3       | 63.53 |150| 📉  | uukuguy/speechless-code-mistral-7b-v1.0     | 63.6    | 61.18| 83.77     | 63.4  | 47.9       | 78.37      | 47.01 |151 152The merge appears to be successful, especially considering the substantial improvement in the GSM8K benchmark while maintaining comparable performance on other metrics.153 154 155## Usage156 157Alpaca Instruction Format and [Divine Intellect](https://raw.githubusercontent.com/oobabooga/text-generation-webui/ae8cd449ae3e0236ecb3775892bb1eea23f9ed68/presets/Divine%20Intellect.yaml) preset.158 159```160You are an intelligent programming assistant.161 162### Instruction:163Implement a linked list in C++164 165### Response:166```167 168Preset:169 170```text171temperature: 1.31172top_p: 0.14173repetition_penalty: 1.17174top_k: 49175```176 177 178# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)179Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_sethuiyer__CodeCalc-Mistral-7B)180 181|             Metric              |Value|182|---------------------------------|----:|183|Avg.                             |66.33|184|AI2 Reasoning Challenge (25-Shot)|61.95|185|HellaSwag (10-Shot)              |83.64|186|MMLU (5-Shot)                    |62.78|187|TruthfulQA (0-shot)              |47.79|188|Winogrande (5-shot)              |78.30|189|GSM8k (5-shot)                   |63.53|190 191