lateesha-bhatia/Llama-3.2-3B-SEC-Fintech-GGUF

Llama-3.2-3B-SEC-Fintech (4-bit GGUF)

A domain-adapted Small Language Model (SLM) optimized for high-precision information extraction from corporate SEC annual (10-K) and quarterly (10-Q) disclosures.

Fine-tuned from meta-llama/Llama-3.2-3B-Instruct using QLoRA (Unsloth) and quantized to 4-bit q4_k_m GGUF format for zero-cost, low-latency CPU inference.

Model Highlights

• —Base Architecture: Llama-3.2-3B-Instruct
• —Domain Adaptation: Fine-tuned on 5,000 SEC filing Q&A records.
• —Optimization: 4-bit Medium Quantization (q4_k_m) via llama.cpp.
• —Memory Footprint: ~2.0 GB RAM (runs on standard consumer CPUs without requiring a dedicated GPU).
• —Inference Runtime: Compatible with Ollama, llama-cpp-python, and LM Studio.

Training Configuration & Hyperparameters

• —Framework: Unsloth + TRL (SFTTrainer)
• —Hardware: 1x NVIDIA T4 GPU (Google Colab Free Tier)
• —PEFT Method: QLoRA (4-bit base weights)
• —LoRA Rank ($r$): 16
• —LoRA Alpha ($\alpha$): 16
• —LoRA Target Modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• —Effective Batch Size: 8 (Per-device batch size: 2, Gradient accumulation steps: 4)
• —Learning Rate: $2 \times 10^{-4}$ (Linear decay, AdamW 8-bit)
• —Convergence: Training completed at step 275–300 with cross-entropy loss stabilizing at ~0.45 – 0.55.

Deployment & Usage

Option 1: Run Locally with Ollama

1. 1.Download unsloth.Q4_K_M.gguf from this repository.
2. 2.Create a file named Modelfile in the same directory:

FROM ./unsloth.Q4_K_M.gguf

TEMPLATE """<|begin_of_text|><|start_header_id|>system<|end_header_id|>

{{ .System }}<|eot_id|><|start_header_id|>user<|end_header_id|>

{{ .Prompt }}<|eot_id|><|start_header_id|>assistant<|end_header_id|>

{{ .Response }}<|eot_id|>"""

SYSTEM """You are a financial analyst. Extract structured facts from the following SEC context."""

PARAMETER stop "<|begin_of_text|>"
PARAMETER stop "<|start_header_id|>"
PARAMETER stop "<|end_header_id|>"
PARAMETER stop "<|eot_id|>"
PARAMETER temperature 0.1
PARAMETER top_p 0.9