fotapol/qwen3-1.7b-financial-rag-lora-v4

Qwen3-1.7B Financial RAG LoRA v4

This is an unmerged LoRA adapter for grounded financial-document question answering. It is intended to run on top of the pinned Qwen/Qwen3-1.7B base model with retrieved document context. It was trained to preserve values and units, cite supplied sources, perform explicit calculations, and refuse questions that the context cannot support.

This is a pet-project research artifact, not a financial-advice system.

Exact base model

• —Model: Qwen/Qwen3-1.7B
• —Revision: 70d244cc86ccca08cf5af4e1e306ecf908b1ad5e
• —Adapter weights SHA-256: 6a89a40cb988567ee09e1ed049b19f09456ec84975738da9edec27613627b2e2

The adapter is not merged into the base model. adapter_config.json records the exact base revision.

Training

The schema-v6 training split contains 9,000 RAG-shaped chat examples:

• —40% direct lookup, including 3,287 table-lookup examples;
• —40% financial reasoning;
• —20% unsupported-context refusal.

Training used one epoch of 4-bit NF4 QLoRA on two NVIDIA T4 GPUs, LoRA rank 16, alpha 32, dropout 0.05, an effective batch size of 8, and a learning rate of 1e-4. All 1,125 optimizer steps completed. See training_metadata.json and dataset_manifest.json for reproducibility details.

Evaluation

On the complete 900-example schema-v6 validation split, using the same deterministic prompts for base and adapter:

Table-lookup overall accuracy improved from 60.13% to 98.73%. Across all examples, v4 improved 205 cases and regressed 54 relative to base.

A four-question frozen-context smoke test using financial_report_chunking_test.pdf passed 4/4 cases for direct lookup, large-table lookup, arithmetic reasoning, and an unsupported CEO question. Two contexts in this small diagnostic suite were oracle-augmented, so it is a generator smoke test rather than a complete end-to-end retrieval benchmark.

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_id = "Qwen/Qwen3-1.7B"
base_revision = "70d244cc86ccca08cf5af4e1e306ecf908b1ad5e"
adapter_id = "fotapol/qwen3-1.7b-financial-rag-lora-v4"

tokenizer = AutoTokenizer.from_pretrained(adapter_id)
base = AutoModelForCausalLM.from_pretrained(
    base_id,
    revision=base_revision,
    device_map="auto",
    dtype="auto",
)
model = PeftModel.from_pretrained(base, adapter_id, is_trainable=False)
model.eval()

Use the Qwen chat template with thinking disabled and deterministic decoding (do_sample=False). Supply retrieved context containing stable source labels such as [Source 1] and instruct the model to use only that context.

Limitations

• —Refusal behavior is weaker than the base model on the broader unsupported validation set.
• —Financial reasoning remains imperfect; reasoning overall accuracy was 18.06%.
• —Synthetic table examples are strongly represented and may not cover every real parser or table layout.
• —Citations indicate supplied context sources; they do not independently verify the source document.
• —Do not use generated answers as financial, legal, accounting, or investment advice.