fotapol/qwen3-1.7b-financial-rag-lora-v3

Qwen3-1.7B Financial RAG LoRA v3

This is an experimental PEFT LoRA adapter for grounded question answering over retrieved financial-document context. It is the adapter used by the document-rag pet-project MVP.

The adapter is intentionally published separately from the base weights. It must be loaded on Qwen/Qwen3-1.7B at the pinned revision:

~~~ 70d244cc86ccca08cf5af4e1e306ecf908b1ad5e ~~~

V3 was trained to improve behavior that the earlier benchmark-oriented adapter did not learn reliably:

• —preserve currency, scale, percentage, and other answer units;
• —cite supplied context using labels such as [Source 1];
• —show a normalized calculation trace for reasoning questions;
• —return an exact refusal when the supplied context is insufficient;
• —treat retrieved document text as data rather than instructions.

Intended use

The intended input is a grounded chat prompt containing:

1. 1.a system instruction requiring context-only answers;
2. 2.up to five retrieved financial-document chunks labeled [Source N];
3. 3.one user question.

The intended output is a short answer with units and source citations, or exactly:

~~~ I cannot answer this question from the supplied context. ~~~

This adapter is intended for demonstrations, learning, and experimentation. It is not suitable for autonomous financial, investment, legal, tax, or accounting decisions.

Loading the adapter

Install compatible versions:

~~~bash pip install "transformers>=4.57.6" "peft>=0.20.0" accelerate torch ~~~

Load the pinned base model and unmerged adapter:

~~~python import torch from peft import PeftModel from transformers import AutoModelForCausalLM, AutoTokenizer

BASEMODELID = "Qwen/Qwen3-1.7B" BASEMODELREVISION = "70d244cc86ccca08cf5af4e1e306ecf908b1ad5e" ADAPTER_ID = "fotapol/qwen3-1.7b-financial-rag-lora-v3"

tokenizer = AutoTokenizer.frompretrained(ADAPTERID) basemodel = AutoModelForCausalLM.frompretrained( BASEMODELID, revision=BASEMODELREVISION, dtype="auto", devicemap="auto", ) model = PeftModel.frompretrained( basemodel, ADAPTERID, is_trainable=False, ) model.eval()

messages = [ { "role": "system", "content": ( "You are a financial document question-answering assistant. " "Use only the retrieved context supplied by the user. " "If the context does not fully support an answer, respond exactly: " "I cannot answer this question from the supplied context. " "When an answer is supported, cite the relevant source labels." ), }, { "role": "user", "content": ( "Retrieved context:\n\n" "[Source 1 | page 4 | chunk_id example-1]\n" "Revenue was $14.1 million in 2025 and $12.4 million in 2024.\n\n" "Question:\n" "By how much did revenue increase from 2024 to 2025?" ), }, ]

text = tokenizer.applychattemplate( messages, tokenize=False, addgenerationprompt=True, enablethinking=False, ) inputs = tokenizer(text, returntensors="pt").to(model.device)

with torch.inferencemode(): generated = model.generate( **inputs, maxnewtokens=128, dosample=False, padtokenid=tokenizer.padtokenid or tokenizer.eostokenid, eostokenid=tokenizer.eostokenid, )

answer = tokenizer.decode( generated[0, inputs["inputids"].shape[1] :], skipspecial_tokens=True, ).strip() print(answer) ~~~

The deterministic settings above reproduce the project evaluation configuration. They are not a claim that greedy decoding is optimal for every Qwen3 use case.

Training

• —Method: 4-bit NF4 QLoRA
• —LoRA rank: 16
• —LoRA alpha: 32
• —LoRA dropout: 0.05
• —Target modules: q, k, v, o, gate, up, and down projections
• —Epochs: 1
• —Optimizer steps: 1,460
• —Effective batch size: 8
• —Maximum sequence length: 4,096
• —Thinking mode: disabled
• —Loss: assistant tokens only
• —Training records: 11,674
• —Validation records: 1,493
• —Test records reserved but not evaluated: 1,475

Training examples were derived from FinQA and DocFinQA and converted into production-shaped RAG conversations using frozen BM25+dense RRF contexts, source labels, units, visible calculations, and synthetic insufficient-context refusals. Report-level splitting prevents documents from crossing train, validation, and test.

See datasetmanifest.json and trainingmetadata.json for the exact configuration and counts.

Validation results

The complete 1,493-example schema-v5 validation split was evaluated with identical frozen prompts for the pinned base and adapter. Both conditions used thinking disabled, greedy decoding, a 4,096-token input limit, and a 128-token output limit.

These are project-specific validation metrics, not public benchmark or leaderboard results. The held-out schema-v5 test split was not run for this MVP release. Exact evaluation configuration is stored in evaluation/validation_metrics.json.

Limitations

• —The adapter is not a reliable calculator. It often emits a sensible calculation structure but an incorrect final arithmetic result.
• —Final-value accuracy is 23.70%; important figures must be checked against the cited document.
• —Accuracy declines substantially as retrieved context becomes longer.
• —Questions requiring evidence from three or more sources remain difficult.
• —Citation labels are usually valid, but the model sometimes omits part of the required evidence.
• —Unsupported refusal behavior is uneven: it is stronger on DocFinQA-shaped inputs and weaker on FinQA-shaped inputs.
• —The synthetic refusal rule requires every gold source-lineage element. Some contexts labeled insufficient still contain plausible alternative evidence, so a generated answer is not always a simple hallucination.
• —Training and evaluation cover English financial-report questions and do not establish performance in other domains or languages.

Always display retrieved context, page information, and chunk IDs beside generated answers.

Reproducibility

• —Training manifest SHA-256: 9ba6699401ef30cd7646cb0e86f15b88afa904b9232db5e1d75d74f162a3bec9
• —Adapter weights SHA-256: dcafae02812dbb479202bde0ffbe7758bf93380570775c4fe132d91b80136d8b
• —Validation evaluation archive SHA-256: 842da65320a8c6f9c6936215684368c11c9bb03116d372f7b4755db3a9f6010d

License

The adapter follows the Apache-2.0 license of the Qwen3-1.7B base model. Users are responsible for reviewing the licenses and terms of the base model and source datasets.