fairleap-ai/fairleap-v1-clm-sahabatai-8b-adapter

<p align="center"> <img src="assets/logo.png"/> <h1 align="center">Fairleap v1 CLM Sahabat-AI 8B Adapter</h1> </p>

📘 Model Overview

A QLoRA adapter that turns `GoToCompany/llama3-8b-cpt-sahabatai-v1-instruct` into a driver-welfare assistant for Gojek/GOTO partners in Indonesia, built for the Fairleap AI project — a platform addressing income uncertainty and wellbeing for ride-hailing drivers.

It is trained for three behaviours: answering from context it was handed rather than asking for data, calling a forecasting tool when a question needs arithmetic a language model cannot do, and declining what falls outside the product. Training data is `fairleap-ai/fairleap-driver-chat-sft-43k`.

Sahabat-AI is itself a continued-pretrain of Llama-3 8B on Indonesian, Javanese and Sundanese, which is why this adapter reaches a lower loss than its Qwen3.5-4B sibling on the same 12,000 conversations.

[!IMPORTANT] Trained entirely on synthetic conversations that no human reviewed. In held-out evaluation it invents a week-over-week earnings comparison in half of all earnings answers — quoting the figures it was given correctly, then appending a prior-period total that was never supplied. Read Limitations & Biases before putting it in front of anyone.

🚀 Usage

This repository holds a LoRA adapter only — no base weights. load_model.py handles the four things that otherwise look like broken weights:

from load_model import load, chat, build_system_prompt, PREDICT_EARNINGS_TOOL

model, tokenizer = load()          # base + adapter, 4-bit

system = build_system_prompt(
    today="2026-08-24", city="Bekasi", vehicle="motor", risk="sedang",
    wellness_score=62, period="2026-08-18 s/d 2026-08-24",
    totals={"Total penghasilan": "Rp1.482.000", "Total order": "88",
            "Hari kerja": "6 dari 7 hari", "Rata-rata per hari kerja": "Rp247.000",
            # Supply this and it quotes it instead of inventing one.
            "Total 7 hari sebelumnya": "Rp1.301.000"},
)
print(chat(model, tokenizer, [
    {"role": "system", "content": system},
    {"role": "user", "content": "berapa penghasilan saya minggu ini?"},
]))

Offer the tool only when a forecast is plausibly needed — see Limitations:

reply = chat(model, tokenizer, messages, tools=[PREDICT_EARNINGS_TOOL])

from load_model import parse_tool_call
parse_tool_call(reply)
# ('predict_earnings', {'start': '2026-08-25', 'end': '2026-09-01', 'wellness_score': 62})

Four traps, handled for you

1. 1.The base model's own chat template silently destroys tool calls. Sahabat-AI ships the plain Llama-3 template, which renders every message as content | trim and ignores both tool_calls and the tools argument. An assistant tool-call turn carries content: "", so it renders as an empty assistant reply — no error anywhere, roughly 10% of the corpus gone. The tool-capable template is chat_template.jinja in this repository, and from_pretrained on this directory picks it up. Load the tokenizer from the base repo instead and tool use breaks with no warning.
2. 2.Adapter-only directory. AutoModelForCausalLM.from_pretrained(".") fails and PEFT then retries the local path as a Hub repo id, reporting HFValidationError: Repo id must be in the form 'repo_name'…, which reads as a path bug.
3. 3.The base's generation config disagrees with its own template. It names <|end_of_text|> (128001) as eos while every turn ends with <|eot_id|> (128009), so generation never stops. chat() passes eos explicitly.
4. 4.Tool calls come back as one line of JSON keyed `parameters`, not the arguments an OpenAI-shaped parser looks for: {"name": "predict_earnings", "parameters": {…}}. `fairleap-api` must parse this shape — it differs from the XML block the Qwen adapter emits.

The system prompt carries the driver

The service this was trained for is stateless and identity-blind: every fact about a driver arrives in the request. Training prompts ran ~4,400 characters — persona, style, prohibitions, then city, vehicle, BPJS status, risk tolerance, a 7-day summary and up to 14 daily rows. Give it less and it has less to be correct about; give it nothing and it has nothing to read.

🗂️ Model Details

Every adapted module sits inside model.layers.* — 32 layers × 7 projections × 2 matrices — verified in the shipped safetensors, not just asserted at train time.

📊 Training

The same 12,000 rows and the same seed as the Qwen sibling, so the two differ by base model rather than by data.

Unlike the Qwen run, this curve has largely flattened — the last two passes gained 0.012 and 0.003. The remaining lever is more data, not more epochs.

Loss is not comparable across tokenizers. Llama-3 spends ~1.12× more tokens on this corpus than Qwen does, and more tokens each carrying less information mechanically deflates per-token loss. Qwen's 0.979 against this 0.7499 is a real margin but a smaller one than it looks — roughly 0.84 once corrected.

training/ reproduces the run end to end.

📈 Evaluation

All 427 held-out test conversations, 0 generation errors. Every reply and its per-record flags ship in eval_results.jsonl, so the figures below can be checked rather than trusted:

python eval_model.py --backend unsloth --model . --test fairleap_test.jsonl

Tool use is the strong result

Perfect recall and perfect precision on the distribution it was trained for. The evaluator's expected tool, none = 11 is not a miss: those are exactly the 11 forecast conversations where no tool was offered.

Refusals and grounding

Auditing the 4 flagged grounding failures by hand: one is correct arithmetic the checker cannot do (Rp3.989.500 ÷ 19 = 209,974), two are small slips on derived averages (off by 105 and by 1,000), and one is a genuine fabrication.

Qualitatively, it gets calendar facts right that the Qwen adapter got wrong — asked for a weekly summary it rendered 2026-08-21 as Jumat, and that date is a Friday — and it reads "minggu depan" as a sensible seven-day window rather than three days.

⚠️ Limitations & Biases

It invents week-over-week comparisons, in half of all earnings answers. This is the defect that matters most.

scenario             invented  total   rate
earnings_qa                27     53  50.9%
earnings_forecast           1     32   3.1%
multi_intent                1     19   5.3%
ALL                        29    427   6.8%

Always the same template, with a baseline that was never in the prompt:

"Dibanding 7 hari sebelumnya (Rp928.500), penghasilan Pak naik Rp390.000 atau sekitar 42 persen."

The context figures around it are quoted perfectly, and the delta and percentage are computed from the invention, so the whole sentence is internally consistent. That is what makes it hard to catch. Mitigation: put the previous period's total in the system prompt — then it quotes a real number instead of manufacturing one — and treat any prior-period figure in a reply as unverified unless you supplied it.

Note that check_reply_grounding caught only 1 of these 29. It accepts any figure within 2% of a context number times one of 26 multipliers, which lets almost anything through. check_period_comparison in eval_model.py was added for this and is what the table reports.

Small arithmetic slips on derived averages. Monthly total ÷ working days came out 105 low in one audited reply and 1,000 low in another. Figures it quotes directly are exact; figures it computes are approximately right.

Javanese and Sundanese drift to Indonesian. Asked in either, it answers in fluent casual Indonesian — correct content, wrong register — despite jv being 10.2% of the corpus. The language drift 0/427 above measures CJK leakage only, a Qwen-era concern; it cannot see this and should not be read as evidence against it.

It over-calls when you offer the tool on a turn that does not need one. On the natural distribution it never called unbidden (0 unsolicited in 427). But forced — tool offered on a wellness question, or on a recall question whose answer was already in context — it produced a predict_earnings call both times. The caller decides, not the model.

Not converged. Trained on 12,000 of 41,461 available conversations. The loss curve has flattened for this data volume, so more epochs will not help; more data might. This is a v1.

Inherits every dataset limitation — fully synthetic, unreviewed advice, teacher-model errors about Indonesian financial products, simulated tool results. See the dataset card.

Not professional advice. It discusses debt, insurance selection and investment for a financially vulnerable population, and none of it was expert-reviewed. Not suitable for estimating real driver income, informing platform or labour policy, or any claim about actual gig-economy conditions in Indonesia.

🛠️ Tech Stacks

• —peft: The Hugging Face library implementing LoRA and other parameter-efficient finetuning methods.
• —unsloth: A finetuning library giving roughly 2x faster training and much lower VRAM use than stock peft+trl.
• —trl: The Hugging Face supervised-finetuning and RL library that ran the training loop.
• —transformers: The Hugging Face library providing the base model, tokenizer and chat template.
• —bitsandbytes: The 4-bit NF4 quantisation backend that lets an 8B model train on one 40 GB card.
• —torch: The deep learning framework everything above runs on.

📚 Citation

@misc{fairleap_v1_clm_sahabatai_8b_adapter,
  title  = {Fairleap v1 CLM Sahabat-AI 8B Adapter},
  author = {Fairleap AI},
  year   = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/fairleap-ai/fairleap-v1-clm-sahabatai-8b-adapter}}
}

📝 License

The base model is Llama-3 derived, so the Llama 3 Community License governs use of this adapter too — you cannot run it without loading those weights. Its Acceptable Use Policy applies, and attribution ("Built with Meta Llama 3") is required for derivative works. The code in this repository (load_model.py, eval_model.py, training/) is MIT.