fairleap-ai/fairleap-v1-clm-qwen3.5-4b-adapter

<p align="center"> <img src="assets/logo.png"/> <h1 align="center">Fairleap v1 CLM Qwen3.5-4B Adapter</h1> </p>

📘 Model Overview

A QLoRA adapter that turns `Qwen/Qwen3.5-4B` into a driver-welfare assistant for Gojek/GOTO partners in Indonesia, built for the Fairleap AI project — a platform addressing income uncertainty and wellbeing for ride-hailing drivers.

It is trained for three behaviours: answering from context it was handed rather than asking for data, calling a forecasting tool when a question needs arithmetic a language model cannot do, and declining what falls outside the product. Training data is `fairleap-ai/fairleap-driver-chat-sft-43k`.

[!IMPORTANT] Trained entirely on synthetic conversations that no human reviewed, and no held-out benchmark has been run — the evaluation below is a 10-conversation probe, not a score. The model also over-calls its forecasting tool, reaching for an earnings prediction on questions about fatigue and traffic. Read Limitations & Biases before putting it in front of anyone.

🚀 Usage

This repository holds a LoRA adapter only — no base weights. load_model.py handles the three things that otherwise look like broken weights:

from load_model import load, chat, build_system_prompt, PREDICT_EARNINGS_TOOL

model, tokenizer = load()          # base + adapter, 4-bit

system = build_system_prompt(
    today="2026-08-24", city="Bekasi", vehicle="motor", risk="sedang",
    wellness_score=62, period="2026-08-18 s/d 2026-08-24",
    totals={"Total penghasilan": "Rp1.482.000", "Total order": "88",
            "Hari kerja": "6 dari 7 hari", "Rata-rata per hari kerja": "Rp247.000"},
)
print(chat(model, tokenizer, [
    {"role": "system", "content": system},
    {"role": "user", "content": "berapa penghasilan saya minggu ini?"},
]))

Offer the tool only when a forecast is plausibly needed — see Limitations:

reply = chat(model, tokenizer, messages, tools=[PREDICT_EARNINGS_TOOL])

from load_model import parse_tool_call
parse_tool_call(reply)
# ('predict_earnings', {'start': '2026-08-25', 'end': '2026-08-27', 'wellness_score': 62})

Three traps, handled for you

1. 1.The base is a vision-language model. Qwen3_5ForConditionalGeneration — so from_pretrained returns a Qwen3VLProcessor, not a tokenizer, and its __call__ reads the first positional argument as an image source: processor("halo") raises Incorrect image source. get_tokenizer() extracts the inner text tokenizer.
2. 2.Adapter-only directory. AutoModelForCausalLM.from_pretrained(".") fails and PEFT then retries the local path as a Hub repo id, reporting HFValidationError: Repo id must be in the form 'repo_name'…, which reads as a path bug.
3. 3.Tool calls come back as Qwen's XML block, not the JSON the corpus stored: <tool_call><function=predict_earnings><parameter=start>…. parse_tool_call() recovers it. `fairleap-api` must parse this shape.

The system prompt carries the driver

The service this was trained for is stateless and identity-blind: every fact about a driver arrives in the request. Training prompts ran ~4,400 characters — persona, style, prohibitions, then city, vehicle, BPJS status, risk tolerance, a 7-day summary and up to 14 daily rows. Give it less and it has less to be correct about; give it nothing and it has nothing to read.

🗂️ Model Details

The vision tower is untouched. Every adapted module sits under model.language_model.*. The projection names are shared across both stacks, so an unscoped target list would silently adapt a tower that never sees an image — the training script asserts the scoping rather than assuming it, and the shipped safetensors was re-checked: 256 tensors, none outside the text decoder.

📊 Training

The 12,000 sample preserves the corpus mix to within 0.04 percentage points on scenario, language and source, and keeps all 17 scenarios and all 5 language registers.

Loss fell monotonically and was still falling at the end:

Final train loss 1.029, so the train/eval gap stayed ≈0.05 — no overfitting, and headroom for more epochs or more data rather than less.

training/ reproduces the run end to end.

📈 Evaluation

There is no held-out benchmark. A full scoring pass over the 427-conversation test split was started and stopped on cost grounds. What follows is a 10-conversation probe plus six qualitative prompts. Treat every number here as indicative, not measured.

eval_model.py ships in this repo and scores the failure modes that matter for this product — invented Rupiah figures, out-of-scope tools, malformed or unsolicited tool calls, language drift, refusal erosion. To run it properly:

python eval_model.py --backend unsloth --model . --test fairleap_test.jsonl

10-conversation probe, tool offered on every turn:

Argument formation is the strong result: 8 calls, zero malformed, and zero overruns — generation stops at the call instead of inventing the forecast it was about to request.

Six qualitative prompts (verbatim output in the session that produced this adapter): grounded recall quoted every figure exactly from the stuffed context and invented none; wellness advice reproduced the rest/hydration/clinic guidance; financial advice respected the stated risk tolerance; the out-of-scope prompt declined to draft a divorce petition and redirected.

⚠️ Limitations & Biases

It over-calls the forecasting tool. In the 10-conversation probe it called predict_earnings on 8 of 10 conversations, including wellness ("badan saya capek terus"), traffic_route and data_absent — where a forecast is simply the wrong response. It declined to call only on clarification and out_of_scope. Root cause is a distribution mismatch: under 10% of training conversations carried a tool, so "tool offered, not needed" is under-represented. Mitigation: offer `PREDICT_EARNINGS_TOOL` only on turns where a forecast is plausible, and treat a call on a wellness or routing question as a bug in the caller, not a signal from the driver. Quantifying this properly needs the full evaluation.

No held-out score. Everything above is 10 conversations and 6 prompts. Rates below ~10% cannot be distinguished from zero at that sample size.

Javanese drift. Asked a question in Javanese, it answered in casual Indonesian — correct content, wrong register — despite jv being 10.2% of the corpus. Score jv and su separately; a corpus-level average hides this.

It embellishes beyond context. Asked for a weekly summary it rendered 2026-08-21 as Minggu (Sunday) when that date is a Friday — a weekday that was never in the prompt. Figures were grounded; the decoration around them was not.

Date-range interpretation is loose. Asked about minggu depan ("next week") it requested a three-day window, consistently across probes. Validate start/end before executing the call.

Not converged. Trained on 12,000 of 41,461 available conversations, and eval loss was still improving at 2 epochs. This is a v1, not a finished model.

Inherits every dataset limitation — fully synthetic, unreviewed advice, teacher-model errors about Indonesian financial products, simulated tool results. See the dataset card.

Not professional advice. It discusses debt, insurance selection and investment for a financially vulnerable population, and none of it was expert-reviewed. Not suitable for estimating real driver income, informing platform or labour policy, or any claim about actual gig-economy conditions in Indonesia.

Vision is untrained. The base accepts images; this adapter never saw one. Image input is out of scope and unvalidated.

🛠️ Tech Stacks

• —peft: The Hugging Face library implementing LoRA and other parameter-efficient finetuning methods.
• —unsloth: A finetuning library giving roughly 2x faster training and much lower VRAM use than stock peft+trl.
• —trl: The Hugging Face supervised-finetuning and RL library that ran the training loop.
• —transformers: The Hugging Face library providing the base model, processor and chat template.
• —bitsandbytes: The 4-bit NF4 quantisation backend that lets a 4B model train on one 40 GB card.
• —torch: The deep learning framework everything above runs on.

📚 Citation

@misc{fairleap_v1_clm_qwen35_4b_adapter,
  title  = {Fairleap v1 CLM Qwen3.5-4B Adapter},
  author = {Fairleap AI},
  year   = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/fairleap-ai/fairleap-v1-clm-qwen3.5-4b-adapter}}
}

📝 License

This adapter is licensed under the MIT License. The base model Qwen/Qwen3.5-4B carries its own license (Apache 2.0); using this adapter means loading that model too.