coderian/axiom-python-1.5B

1---2language:3- tr4- en5license: apache-2.06base_model: Qwen/Qwen2.5-1.5B7tags:8- axiom9- qwen10- qwen211- fine-tuned12- lora13- sft14- trl15- code16- python17- text-generation18pipeline_tag: text-generation19model_type: qwen220library_name: transformers21---22 23# Axiom Python 1.5B24 25**Axiom Python 1.5B** is a text generation (causal language model) fine-tuned on [Qwen/Qwen2.5-1.5B](https://huggingface.co/Qwen/Qwen2.5-1.5B) with a focus on Python programming and code generation.26 27The model was trained using **LoRA + SFT** with the [TRL](https://github.com/huggingface/trl) library on the [CodeAlpaca_20K](https://huggingface.co/datasets/HuggingFaceH4/CodeAlpaca_20K) and [PythonCodeInstruct_18K](https://huggingface.co/datasets/iamtarun/python_code_instructions_18k_alpaca) datasets.28 29## Model Details30 31| Property | Value |32|---|---|33| Base Model | [Qwen/Qwen2.5-1.5B](https://huggingface.co/Qwen/Qwen2.5-1.5B) |34| Architecture | Qwen2ForCausalLM |35| Parameters | ~1.5B |36| Hidden Layers | 28 |37| Hidden Size | 1536 |38| Attention Heads | 12 |39| KV Heads | 2 |40| Vocabulary Size | 151936 |41| Max Context Length | 131072 |42| Weight Dtype | float16 (FP16) |43| Training Method | LoRA (r=16, alpha=32) + SFT |44| Datasets | CodeAlpaca_20K + PythonCodeInstruct_18K |45| Languages | Turkish and English (code-focused) |46 47## Installation48 49Install the following packages to get started:50 51```bash52pip install transformers torch53```54 55> If you are using a GPU, make sure you have installed a CUDA-compatible PyTorch version.56 57## Usage58 59### 1. Using `pipeline` (Simplest Way)60 61```python62from transformers import pipeline63 64generator = pipeline(65    "text-generation",66    model="coderian/axiom-python-1.5B",67    device_map="auto",68    torch_dtype="auto",69)70 71prompt = """### Instruction:72Write a Python function that reverses the elements of a list.73 74### Answer:75"""76 77output = generator(78    prompt,79    max_new_tokens=256,80    temperature=0.7,81    top_p=0.9,82    do_sample=True,83)84 85print(output[0]["generated_text"])86```87 88### 2. Using `AutoModelForCausalLM`89 90```python91import torch92from transformers import AutoModelForCausalLM, AutoTokenizer93 94model_id = "coderian/axiom-python-1.5B"95 96tokenizer = AutoTokenizer.from_pretrained(model_id)97model = AutoModelForCausalLM.from_pretrained(98    model_id,99    torch_dtype=torch.float16,100    device_map="auto",101)102 103model.eval()104 105prompt = """### Instruction:106Write a Python function that adds two numbers.107 108### Answer:109"""110 111inputs = tokenizer(prompt, return_tensors="pt").to(model.device)112 113with torch.no_grad():114    outputs = model.generate(115        **inputs,116        max_new_tokens=256,117        temperature=0.7,118        top_p=0.9,119        do_sample=True,120        pad_token_id=tokenizer.eos_token_id,121    )122 123response = tokenizer.decode(124    outputs[0][inputs["input_ids"].shape[1]:],125    skip_special_tokens=True,126)127 128print(response)129```130 131### 3. Using the Chat Template132 133Since the Qwen2.5 tokenizer supports the ChatML format, you can also use the model for chat-style conversations:134 135```python136import torch137from transformers import AutoModelForCausalLM, AutoTokenizer138 139model_id = "coderian/axiom-python-1.5B"140 141tokenizer = AutoTokenizer.from_pretrained(model_id)142model = AutoModelForCausalLM.from_pretrained(143    model_id,144    torch_dtype=torch.float16,145    device_map="auto",146)147 148messages = [149    {"role": "system", "content": "You are Axiom, a helpful Python coding assistant."},150    {"role": "user", "content": "Write a Python function to check if a number is prime."},151]152 153text = tokenizer.apply_chat_template(154    messages,155    tokenize=False,156    add_generation_prompt=True,157)158 159inputs = tokenizer(text, return_tensors="pt").to(model.device)160 161with torch.no_grad():162    outputs = model.generate(163        **inputs,164        max_new_tokens=256,165        temperature=0.7,166        top_p=0.9,167        do_sample=True,168        pad_token_id=tokenizer.eos_token_id,169    )170 171response = tokenizer.decode(172    outputs[0][inputs["input_ids"].shape[1]:],173    skip_special_tokens=True,174)175 176print(response)177```178 179### Recommended Generation Parameters180 181| Parameter | Suggested Value | Description |182|---|---|---|183| `max_new_tokens` | `512` | Maximum number of new tokens to generate |184| `temperature` | `0.7` | Lower values produce more deterministic output |185| `top_p` | `0.9` | Nucleus sampling ratio |186| `do_sample` | `True` | Enable/disable sampling |187| `repetition_penalty` | `1.05` | Reduces repetitive output |188 189## Training Details190 191| Setting | Value |192|---|---|193| Base Model | Qwen/Qwen2.5-1.5B |194| LoRA Rank (r) | 16 |195| LoRA Alpha | 32 |196| LoRA Dropout | 0.05 |197| Target Modules | q_proj, v_proj |198| Batch Size | 32 (2 x 4 grad. accumulation) |199| Training Epochs | 1 |200| Learning Rate | 2e-4 |201| Optimizer | AdamW (fused) |202| Precision | FP16 |203| Steps | 4000 |204| Max Sequence Length | 256 |205| Adapter Location | `axiom-python-1.5B/checkpoint-4000` |206 207After training, the LoRA adapter was merged into the base model and released as a single file. You can also load the adapter directly using the peft library:208 209```python210from peft import PeftModel211from transformers import AutoModelForCausalLM, AutoTokenizer212 213base = AutoModelForCausalLM.from_pretrained(214    "Qwen/Qwen2.5-1.5B",215    torch_dtype="auto",216    device_map="auto",217)218 219model = PeftModel.from_pretrained(base, "path/to/adapter")220```221 222## Limitations223 224- It is a small 1.5B parameter model and may make mistakes on very complex and long code generation tasks.225- It was trained only on Python-focused datasets; performance in other languages is limited.226- The training data has a maximum length of 256 tokens; consistency may degrade in very long contexts.227- Generated code may not always be correct or safe. Review it before running.228- It may contain known limitations inherited from the training data regarding bias and harmful content.229 230## Intended Usage Tips231 232- It performs best on single-line and medium-complexity Python functions.233- Lower the `temperature` value if you want stable output for code generation.234- Since the model was trained in a completion format, the `### Instruction:` / `### Answer:` template yields the highest quality output.235- For batched inference, remember to set `tokenizer.pad_token = tokenizer.eos_token`.236 237## License238 239The base model Qwen2.5 is released under the Apache-2.0 license, and this model is also shared under the **Apache-2.0** license.240 241## Resources242 243- Base Model: [Qwen/Qwen2.5-1.5B](https://huggingface.co/Qwen/Qwen2.5-1.5B)244- Training Library: [TRL](https://github.com/huggingface/trl)245- Dataset 1: [HuggingFaceH4/CodeAlpaca_20K](https://huggingface.co/datasets/HuggingFaceH4/CodeAlpaca_20K)246- Dataset 2: [iamtarun/python_code_instructions_18k_alpaca](https://huggingface.co/datasets/iamtarun/python_code_instructions_18k_alpaca)