ed001/datascience-coder-6.7b

1---2language:3- en4license: cc-by-nc-sa-4.05tags:6- code7- data science8datasets:9- ed001/ds-coder-instruct-v110pipeline_tag: text-generation11model-index:12- name: datascience-coder-6.7b13  results:14  - task:15      type: text-generation16      name: Text Generation17    dataset:18      name: AI2 Reasoning Challenge (25-Shot)19      type: ai2_arc20      config: ARC-Challenge21      split: test22      args:23        num_few_shot: 2524    metrics:25    - type: acc_norm26      value: 34.6427      name: normalized accuracy28    source:29      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=ed001/datascience-coder-6.7b30      name: Open LLM Leaderboard31  - task:32      type: text-generation33      name: Text Generation34    dataset:35      name: HellaSwag (10-Shot)36      type: hellaswag37      split: validation38      args:39        num_few_shot: 1040    metrics:41    - type: acc_norm42      value: 53.8343      name: normalized accuracy44    source:45      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=ed001/datascience-coder-6.7b46      name: Open LLM Leaderboard47  - task:48      type: text-generation49      name: Text Generation50    dataset:51      name: MMLU (5-Shot)52      type: cais/mmlu53      config: all54      split: test55      args:56        num_few_shot: 557    metrics:58    - type: acc59      value: 37.9660      name: accuracy61    source:62      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=ed001/datascience-coder-6.7b63      name: Open LLM Leaderboard64  - task:65      type: text-generation66      name: Text Generation67    dataset:68      name: TruthfulQA (0-shot)69      type: truthful_qa70      config: multiple_choice71      split: validation72      args:73        num_few_shot: 074    metrics:75    - type: mc276      value: 44.8277    source:78      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=ed001/datascience-coder-6.7b79      name: Open LLM Leaderboard80  - task:81      type: text-generation82      name: Text Generation83    dataset:84      name: Winogrande (5-shot)85      type: winogrande86      config: winogrande_xl87      split: validation88      args:89        num_few_shot: 590    metrics:91    - type: acc92      value: 55.7293      name: accuracy94    source:95      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=ed001/datascience-coder-6.7b96      name: Open LLM Leaderboard97  - task:98      type: text-generation99      name: Text Generation100    dataset:101      name: GSM8k (5-shot)102      type: gsm8k103      config: main104      split: test105      args:106        num_few_shot: 5107    metrics:108    - type: acc109      value: 24.94110      name: accuracy111    source:112      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=ed001/datascience-coder-6.7b113      name: Open LLM Leaderboard114---115 116# The Data Science Coder117 118Data Science coder is a group of fine tuned models designed to help with coding for data science applications. It comes in 2 variants: 1.3b and 6.7b. Models are fine tuned from DeepSeek Coder instruct versions. Fine tuning was performed on the [ed001/ds-coder-instruct-v1](https://huggingface.co/datasets/ed001/ds-coder-instruct-v1) dataset which is constructed by filtering publicly available datasets on HuggingFace.119 120## Usage121 122```python123from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline124 125def build_instruction_prompt(instruction):126    return '''127    You are the Data Science Coder, a helpful AI assistant created by a man named Ed.128    You help people with data science coding and you answer questions about data science in a helpful manner.129    ### Instruction:130    {}131    ### Response:132    '''.format(instruction.strip()).lstrip()133 134tokenizer = AutoTokenizer.from_pretrained("ed001/datascience-coder-6.7b", trust_remote_code=True)135model = AutoModelForCausalLM.from_pretrained("ed001/datascience-coder-6.7b", trust_remote_code=True).cuda()136pipe = pipeline(task="text-generation", model=model, tokenizer=tokenizer, max_length=1024, top_p=0.95)137result = pipe(build_instruction_prompt("Perform EDA on the Iris dataset"))138print(result[0]['generated_text'])139```140 141## Training Details142lora_r: 16  143lora_alpha: 8  144lora_dropout: 0.05  145target_modules: q, k, v, o, gate_proj, down_proj, up_proj, lm_head  146weight_decay: 0  147optmizer: paged_adamw_32bit  148lr: 1e-4  149lr_scheduler: cosine  150max_seq_len: 4096  151batch_size: 4  152max_grad_norm: 0.5  153warmup_ratio: 0.05  154num_epochs: 1  155 156The model was trained on the python susbet of the ds-coder-instruct dataset.157 158## Samples159<img src="https://cdn-uploads.huggingface.co/production/uploads/62618f3e6dae705b2567fb13/0H8lj26xLOfLuCD0yVmER.png" width="90%"/>160 161<img src="https://cdn-uploads.huggingface.co/production/uploads/62618f3e6dae705b2567fb13/8W62qr1cPSLsq6lLfLCib.png" width="90%"/>162 163<img src="https://cdn-uploads.huggingface.co/production/uploads/62618f3e6dae705b2567fb13/XNLclcr4KQqtPseGg2Gzn.png" width="90%"/>164 165## Contact166GitHub: [Ea0011](https://github.com/Ea0011)167# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)168Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_ed001__datascience-coder-6.7b)169 170|             Metric              |Value|171|---------------------------------|----:|172|Avg.                             |41.99|173|AI2 Reasoning Challenge (25-Shot)|34.64|174|HellaSwag (10-Shot)              |53.83|175|MMLU (5-Shot)                    |37.96|176|TruthfulQA (0-shot)              |44.82|177|Winogrande (5-shot)              |55.72|178|GSM8k (5-shot)                   |24.94|179 180