durganani60/qwen2.5-0.5b-financial-adapted

Qwen2.5-0.5B Financial Domain Adapted LLM

This model is a domain-adapted version of Qwen2.5-0.5B specifically tailored for financial NLP tasks. It features a custom Byte-Level BPE Tokenizer integrated directly into the base architecture via Tokenizer Surgery and aligned using TRL SFTTrainer with PEFT/LoRA on Apple Silicon (MPS).

Model Details

Model Description

Standard LLM tokenizers often fragment specialized financial terminology (such as ticker symbols like $NVDA, key metrics like EBITDA/CAGR, and currency pairs like EUR/USD) into excessive subwords, leading to increased prompt inflation and potential [UNK] errors.

This model addresses these issues by:

1. 1.Extending the base vocabulary with custom financial domain tokens.
2. 2.Resizing the underlying embedding matrix ($E \in \mathbb{R}^{V_{new} \times d}$) and language model head.
3. 3.Conducting continued pre-training using LoRA to train the new token embeddings while preserving base model capabilities.

• —Developed by: Durga
• —Model type: Causal Language Model (Transformer Decoder)
• —Language(s) (NLP): English (Financial Domain)
• —License: Apache 2.0
• —Finetuned from model: Qwen/Qwen2.5-0.5B

Uses

Direct Use

• —Financial text generation and completion.
• —Processing financial filings (SEC 10-K, 10-Q), earnings transcripts, and analyst reports.
• —Reduced context fragmentation for domain-specific prompt engineering.

Out-of-Scope Use

• —Generating binding financial advice, automated trading decisions, or official financial audits without human oversight.

Bias, Risks, and Limitations

• —Domain Specificity: The model has been continued pre-trained on a focused financial corpus. It is designed for domain text compression and processing rather than general open-domain chat.
• —Accuracy: Outputs should be independently verified when used for quantitative or regulatory compliance tasks.

How to Get Started with the Model

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

# Load model and custom adapted tokenizer
model_id = "durganani60/qwen2.5-0.5b-financial-adapted"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, 
    torch_dtype=torch.float16, 
    device_map="auto"
)

# Test prompt containing financial domain tokens
prompt = "Q3 2024 financial report: $NVDA recorded EBITDA growth above 14.5% with expanding FCF margin."

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=50, do_sample=True, temperature=0.7)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))
'''