fivetech/Harbour

1---2base_model: Qwen/Qwen3.5-35B-A3B3library_name: peft4pipeline_tag: text-generation5tags:6- harbour7- fivewin8- fwh9- lora10- sft11- transformers12- trl13- unsloth14- code-generation15- xbase16- clipper17language:18- en19- es20license: apache-2.021---22 23# Harbour/FWH Coder — Qwen3.5-35B-A3B LoRA v224 25> **Training code:** https://github.com/FiveTechSoft/finetune26 27LoRA adapter fine-tuned on **5,004 compilable Harbour and FiveWin (FWH) examples** for code generation. Built on top of [Qwen3.5-35B-A3B](https://huggingface.co/Qwen/Qwen3.5-35B-A3B), a 35B Mixture-of-Experts model with 256 experts (8 active per token).28 29## What's New in v230 31| | v1 | **v2** |32|---|---|---|33| **Dataset** | 996 entries (7 categories) | **5,004 entries** (8 categories) |34| **Training time** | 3h 43min | **~8h** |35| **Eval loss** | 0.5957 | **0.4790** (↓20%) |36| **Train loss** | 0.6456 | **0.4211** (↓35%) |37| **Format** | messages (chat) | instruction/output |38| **Learning rate** | 1e-4 | 8e-5 (conservative) |39| **Epochs** | 3 | 2 |40 41### Key improvements42 43- **5x more training data** — expanded from 996 to 5,004 unique, compilable examples44- **Better loss convergence** — 35% lower train loss, 20% lower eval loss45- **More conservative training** — lower learning rate preserves base model capabilities46- **FiveWin (FWH) coverage** — added FiveWin GUI framework examples47- **Verified code** — all examples verified with Harbour v3.2.0dev compiler48 49## Dataset50 51Training data sourced from the [Harbour](https://harbour.github.io/) project — an open-source Clipper-compatible compiler — and [FiveWin](https://fivewin.com/) (FWH) GUI framework.52 53### Categories54 55| Category | Count | Description |56|---|---|---|57| contrib | 583 | Contribution libraries (network, database, graphics, security...) |58| rtl | 80 | Harbour Runtime Library |59| include | 59 | Header files with constants/macros |60| tests | 225 | Test programs |61| extras | 25 | Extra libraries |62| utils | 13 | Utility programs |63| fwh | ~500+ | FiveWin GUI framework examples |64| low-level C | 500+ | HB_FUNC C extension wrappers |65 66### Format67 68```json69{70  "instruction": "Write a Harbour function that creates a 2D array...",71  "input": "",72  "system": "You are an expert Harbour programmer...",73  "output": "FUNCTION CreateTable()\n   LOCAL aTable := {}\n   ...",74  "task_type": "code_generation"75}76```77 78## Training Details79 80### Hardware81 82- **Device:** NVIDIA GB10 Grace Blackwell Superchip (DGX Spark)83- **Architecture:** ARM aarch64 (10 NVIDIA Grace CPU cores + Blackwell GPU)84- **RAM:** 121 GB unified memory (CPU + GPU shared)85- **OS:** Ubuntu 24.04.4 LTS (aarch64)86- **Training time:** 7h 49min (564 steps, 2 epochs over 5,004 samples)87 88### Hyperparameters89 90| Parameter | Value |91|---|---|92| Base model | Qwen3.5-35B-A3B (MoE, 256 experts) |93| Method | QLoRA (4-bit) |94| LoRA rank | 8 |95| LoRA alpha | 16 |96| LoRA targets | q/k/v/o/gate/up/down_proj |97| Epochs | 2 |98| Learning rate | 8e-5 |99| LR scheduler | cosine |100| Warmup ratio | 0.05 |101| Batch size | 1 (effective: 16 via grad accum) |102| Max seq length | 1024 |103| Optimizer | adamw_8bit |104 105### Framework106 107- Unsloth 2026.6.8108- Transformers 5.5.0109- PEFT 0.19.1110- PyTorch 2.12.1111 112## How to Use113 114### With PEFT + Transformers115 116```python117from transformers import AutoModelForCausalLM, AutoTokenizer118from peft import PeftModel119 120base_model = AutoModelForCausalLM.from_pretrained(121    "Qwen/Qwen3.5-35B-A3B",122    load_in_4bit=True,123    device_map="auto",124)125model = PeftModel.from_pretrained(base_model, "fivetech/Harbour")126tokenizer = AutoTokenizer.from_pretrained("fivetech/Harbour")127 128prompt = "Write a Harbour function that splits a CSV string into an array."129messages = [130    {"role": "system", "content": "You are an expert Harbour programmer. Write compilable code."},131    {"role": "user", "content": prompt},132]133text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)134inputs = tokenizer([text], return_tensors="pt").to(model.device)135 136output = model.generate(**inputs, max_new_tokens=1500, temperature=0.2)137print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))138```139 140### With Ollama (merge + quantize first)141 142```bash143# Export to GGUF144python -m unsloth.save_pretrained_gguf model_output/ ./tokenizer/ q4_k_m145 146# Then use with Ollama147ollama create harbour-coder -f Modelfile148```149 150## Evaluation151 152Evaluated on 100 Harbour programming tests (Arrays, OOP, Functions, Database, File I/O, Control flow):153 154- **Compilation pass rate:** TBD (running test battery)155- **Categories tested:** Arrays (48), OOP (22), Other (9), Functions (8), Database (7), File I/O (4), Control (2)156 157## License158 159Apache 2.0160 161## Model Card Contact162 163fivetech — https://github.com/fivetechsoft/finetune164