RichardErkhov/CodeGPTPlus_-_deepseek-coder-1.3b-typescript-8bits
Quantization made by Richard Erkhov.
deepseek-coder-1.3b-typescript - bnb 8bits
- Model creator: https://huggingface.co/CodeGPTPlus/
- Original model: https://huggingface.co/CodeGPTPlus/deepseek-coder-1.3b-typescript/
Original model description: --- license: other base_model: deepseek-ai/deepseek-coder-1.3b-base tags:
- axolotl
- generatedfromtrainer model-index:
- name: deepseek-coder-1.3b-typescript results: [] datasets:
- bigcode/the-stack-dedup widget:
- text: "class Person {\n constructor(public name:" example_title: "class"
- text: "function quickSort" example_title: "function" ---
<p align="center"> <img width="1000px" alt="CodeGPT: DeepSeek Coder - Typescript" src="codegpt-deepseek-typescript.png?raw=true"> </p> <p align="center"><a href="https://codegpt.co/">[CodeGPT.co]</a> | <a href="https://ollama.ai/codegpt/deepseek-coder-1.3b-typescript">[🦙 Ollama]</a> | <a href="https://discord.gg/fKyyJX5pne">[Discord]</a> | <a href="https://marketplace.visualstudio.com/items?itemName=DanielSanMedium.dscodegpt">[VSCode Extension]</a> </p> <hr>
<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/> <details><summary>See axolotl config</summary>
axolotl version: 0.3.0
base_model: deepseek-ai/deepseek-coder-1.3b-base
model_type: AutoModelForCausalLM
trust_remote_code: true
load_in_8bit: false
load_in_4bit: false
strict: false
datasets:
- path: CodeGPTPlus/typescript-0-500000-seq1024
type: completion
field: text
val_set_size: 0.001
output_dir: ./fft-out
sequence_len: 1024
adapter:
lora_model_dir:
lora_r:
lora_alpha:
lora_dropout:
lora_target_linear:
lora_fan_in_fan_out:
lora_modules_to_save:
wandb_project: deepseek_1.3_fft
wandb_entity:
wandb_watch:
wandb_name: aws_a10g
wandb_log_model: end
gradient_accumulation_steps: 2
micro_batch_size: 20
num_epochs: 1
optimizer: adamw_bnb_8bit
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 0.000001
max_grad_norm: 1.0
weight_decay: 0.1
lr_scheduler: cosine
learning_rate: 0.00002
train_on_inputs: false
group_by_length: false
bf16: true
fp16: false
tf32: false
gradient_checkpointing: true
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true
loss_watchdog_threshold: 5.0
loss_watchdog_patience: 3
hub_model_id: CodeGPTPlus/deepseek_coder_1.3b_typescript
hub_strategy: every_save
warmup_ratio: 0.01
evals_per_epoch: 20
saves_per_epoch: 3
debug:
deepspeed:
fsdp:
fsdp_config:
special_tokens:
bos_token: "<｜begin▁of▁sentence｜>"
eos_token: "<｜end▁of▁sentence｜>"
pad_token: "<｜end▁of▁sentence｜>"</details><br>
deepseek-coder-1.3b-typescript
CodeGPTPlus/deepseek-coder-1.3b-typescript, emerges as a fine-tuned iteration of deepseek-ai/deepseek-coder-1.3b-base, meticulously crafted by the CodeGPT team to excel in generating expert code in TypeScript. With specific fine-tuning for TypeScript and a dataset of 0.5B tokens, this model excels in producing precise and efficient solutions in this programming language.
The 16K window size and an additional fill-in-the-middle task are employed to deliver project-level code completion.
This new model stands as the ideal choice for those seeking a specialized code generator for TypeScript, backed by the expertise of the CodeGPT team.
It achieves the following results on the evaluation set:
- Loss: 0.7681
Model Developers CodeGPT Team
Variations 1.3B
Input Models input text only.
Output Models generate text only.
How to Use
This model is for completion purposes only. Here give some examples of how to use the model.
Running the model on a GPU
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("CodeGPTPlus/deepseek-coder-1.3b-typescript",
trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("CodeGPTPlus/deepseek-coder-1.3b-typescript",
trust_remote_code=True).cuda()
input_text = """<｜fim▁begin｜>function quickSort(arr: number[]): number[] {
if (arr.length <= 1) {
return arr;
}
const pivot = arr[0];
const left = [];
const right = [];
<｜fim▁hole｜>
return [...quickSort(left), pivot, ...quickSort(right)];
}<｜fim▁end｜>"""
inputs = tokenizer(input_text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_length=256)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))Running with Ollama
Model: https://ollama.ai/codegpt/deepseek-coder-1.3b-typescript
### Running with Ollama and CodeGPT Autocomplete in VSCode
**Documentation:** https://docs.codegpt.co/docs/tutorial-features/code_autocompletion
Select "Ollama - codegpt/deepseek-coder-1.3b-typescript" in the autocomplete model selector.
Then, write any code or comment in the vscode text editor, and the model will provide you with code suggestions through the CodeGPT code autocomplete.
<img width="1000px" alt="CodeGPT: DeepSeek Coder - Typescript" src="ollama_autocomplete_codegpt.gif">
### Fill In the Middle (FIM)<｜fim▁begin｜>function quickSort(arr: number[]): number[] { if (arr.length <= 1) { return arr; } const pivot = arr[0]; const left = []; const right = []; <｜fim▁hole｜> return [...quickSort(left), pivot, ...quickSort(right)]; }<｜fim▁end｜>
## Training procedure
### Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 20
- eval_batch_size: 20
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 40
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-06
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 261
- num_epochs: 1
### Training results
| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:-----:|:---------------:|
| 1.0745 | 0.0 | 1 | 0.8681 |
| 1.2267 | 0.05 | 1308 | 0.8130 |
| 1.1594 | 0.1 | 2616 | 0.8018 |
| 0.7674 | 0.15 | 3924 | 0.7942 |
| 0.6443 | 0.2 | 5232 | 0.7889 |
| 0.9155 | 0.25 | 6540 | 0.7847 |
| 0.7501 | 0.3 | 7848 | 0.7819 |
| 0.8835 | 0.35 | 9156 | 0.7792 |
| 0.7261 | 0.4 | 10464 | 0.7769 |
| 0.9746 | 0.45 | 11772 | 0.7748 |
| 0.6884 | 0.5 | 13080 | 0.7734 |
| 0.6104 | 0.55 | 14388 | 0.7722 |
| 0.8876 | 0.6 | 15696 | 0.7710 |
| 0.9567 | 0.65 | 17004 | 0.7703 |
| 0.6915 | 0.7 | 18312 | 0.7696 |
| 0.8874 | 0.75 | 19620 | 0.7691 |
| 0.6124 | 0.8 | 20928 | 0.7686 |
| 0.8147 | 0.85 | 22236 | 0.7684 |
| 0.8021 | 0.9 | 23544 | 0.7683 |
| 0.8665 | 0.95 | 24852 | 0.7681 |
### Framework versions
- Transformers 4.37.0.dev0
- Pytorch 2.0.1+cu118
- Datasets 2.16.1
- Tokenizers 0.15.0