JetBrains/CodeLlama-7B-KStack-clean

1---2license: apache-2.03datasets:4- JetBrains/KStack-clean5base_model: meta-llama/CodeLlama-7b-hf6results:7- task:8    type: text-generation9  dataset:10    name: MultiPL-HumanEval (Kotlin)11    type: openai_humaneval12  metrics:13  - name: pass@114    type: pass@115    value: 37.8916tags:17- code18---19 20# Model description21 22This is a repository for the **CodeLlama-7b** model fine-tuned on the [KStack-clean](https://huggingface.co/datasets/JetBrains/KStack-clean) dataset with rule-based filtering, in the *Hugging Face Transformers* format. KStack-clean is a small subset of [KStack](https://huggingface.co/datasets/JetBrains/KStack), the largest collection of permissively licensed Kotlin code, automatically filtered to include files that have the highest "educational value for learning algorithms in Kotlin".23 24# How to use25 26```python27from transformers import AutoModelForCausalLM, AutoTokenizer28 29# Load pre-trained model and tokenizer30model_name = 'JetBrains/CodeLlama-7B-KStack-clean'31tokenizer = AutoTokenizer.from_pretrained(model_name)32model = AutoModelForCausalLM.from_pretrained(model_name).to('cuda')33 34# Create and encode input35input_text = """\36This function takes an integer n and returns factorial of a number:37fun factorial(n: Int): Int {\38"""39input_ids = tokenizer.encode(40    input_text, return_tensors='pt'41).to('cuda')42 43# Generate44output = model.generate(45    input_ids, max_length=60, num_return_sequences=1, 46    pad_token_id=tokenizer.eos_token_id47)48 49# Decode output50generated_text = tokenizer.decode(output[0], skip_special_tokens=True)51print(generated_text)52```53 54As with the base model, we can use FIM. To do this, the following format must be used: 55```56'<PRE> ' + prefix + ' <SUF> ' + suffix + ' <MID>'57```58 59# Training setup60 61The model was trained on one A100 GPU with following hyperparameters:62 63|         **Hyperparameter**           |             **Value**              |64|:---------------------------:|:----------------------------------------:|65|        `warmup`            |           100 steps            |66|        `max_lr`        |          5e-5          |67|        `scheduler`        |          linear          |68|        `total_batch_size`        |        32 (~30K tokens per step)          |69|        `num_epochs`        |          2          |70 71More details about fine-tuning can be found in the technical report (coming soon!).72 73# Fine-tuning data74 75For tuning the model, we used 25K exmaples from the [KStack-clean](https://huggingface.co/datasets/JetBrains/KStack-clean) dataset, selected from the larger [KStack](https://huggingface.co/datasets/JetBrains/KStack) dataset according to educational value for learning algorithms. In total, the dataset contains about 23M tokens. 76 77# Evaluation 78 79For evaluation, we used the [Kotlin HumanEval](https://huggingface.co/datasets/JetBrains/Kotlin_HumanEval) dataset, which contains all 161 tasks from HumanEval translated into Kotlin by human experts. You can find more details about the pre-processing necessary to obtain our results, including the code for running, on the [datasets's page](https://huggingface.co/datasets/JetBrains/Kotlin_HumanEval).80 81Here are the results of our evaluation:82 83|         **Model name**           |             **Kotlin HumanEval Pass Rate**              |84|:---------------------------:|:----------------------------------------:|85|           `CodeLlama-7B`            |           26.89            |86|        `CodeLlama-7B-KStack-clean`        |          **37.89**        |87 88# Ethical Considerations and Limitations89 90CodeLlama-7B-KStack-clean is a new technology that carries risks with use. The testing conducted to date has not covered, nor could it cover all scenarios. For these reasons, as with all LLMs, CodeLlama-7B-KStack-clean's potential outputs cannot be predicted in advance, and the model may in some instances produce inaccurate or objectionable responses to user prompts. The model was fine-tuned on a specific data format (Kotlin tasks), and deviation from this format can also lead to inaccurate or undesirable responses to user queries. Therefore, before deploying any applications of CodeLlama-7B-KStack-clean, developers should perform safety testing and tuning tailored to their specific applications of the model.