Shion1124/vlm-lora-agentic-rag

🎯 VLM + LoRA + Agentic RAG: Enterprise Document Structuring

A production-ready Vision Language Model with LoRA adaptation and Agentic Retrieval-Augmented Generation for intelligent document analysis and structured extraction.

🏆 Architecture Overview

This model combines three complementary technologies:

1️⃣ Vision Language Model (VLM)

• —Base: LLaVA-1.5-7B
• —Capability: Multi-modal document understanding
• —Purpose: Understand document layout, tables, charts, and visual elements
• —Quantization: 4-bit (bitsandbytes) for T4 GPU efficiency

2️⃣ LoRA Adaptation Layer

• —Method: Low-Rank Adaptation with r=64, alpha=128
• —Trainable Parameters: 0.1% of base model (lightweight)
• —Task: Fine-tuned for structured output accuracy
• —Efficiency: 8GB VRAM (T4 compatible)

3️⃣ Agentic RAG Engine

• —Strategy: Multi-strategy document retrieval
• —Keyword-based search
• —Semantic similarity search
• —Hybrid search with result verification
• —Purpose: Intelligent answer verification and re-search if needed
• —Embeddings: all-MiniLM-L6-v2 (sentence transformers)
• —Indexing: FAISS (vector search)

📊 Technical Details

Model Specifications

Training Configuration

• —Objective: Structured output accuracy (JSON/YAML/XML)
• —Task: Extract business documents → Structured JSON
• —Loss: Task-specific loss on final assistant output
• —Chain-of-Thought: Preserved (intermediate reasoning masked during loss)

🎯 Use Cases

Financial Document Analysis

• —Quarterly reports → Structured financials
• —Income statements → Parsed key metrics
• —Cash flow statements → Quantified outputs

Technical Documentation

• —System architecture docs → Structured schemas
• —API specs → Parsed endpoints
• —Requirements → Extracted constraints

Business Intelligence

• —Market analysis → Competitive summaries
• —Risk assessments → Structured risk factors
• —Strategic plans → Actionable metrics

📦 Installation & Usage

Quick Start

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

# Load base model
base_model_id = "liuhaotian/llava-v1.5-7b"
adapter_id = "Shion1124/vlm-lora-agentic-rag"

model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    torch_dtype=torch.float16,
    device_map="auto",
    load_in_4bit=True
)

# Load LoRA adapter
model = PeftModel.from_pretrained(model, adapter_id)
tokenizer = AutoTokenizer.from_pretrained(base_model_id)

Document Analysis Pipeline

from PIL import Image
import json

def analyze_document(image_path: str) -> dict:
    """Analyze document and extract structured JSON"""
    
    image = Image.open(image_path).convert("RGB")
    prompt = """Analyze this document and output ONLY valid JSON:
    {
      "title": "...",
      "summary": "...",
      "key_data": [...],
      "insights": "..."
    }"""
    
    # Process with VLM + LoRA
    # ... implementation ...
    
    return structured_json

Agentic RAG Search

from sentence_transformers import SentenceTransformer
import faiss

# Build document index
embedder = SentenceTransformer('all-MiniLM-L6-v2')
index = faiss.IndexFlatL2(vector_dim)
index.add(embeddings)

# Multi-strategy search with verification
def agentic_search(query: str, max_iterations: int = 3):
    for iteration in range(max_iterations):
        if iteration == 1:
            results = keyword_search(query)
        elif iteration == 2:
            results = semantic_search(query)
        else:
            results = hybrid_search(query)
        
        if verify_results(results):
            break
    
    return results

📈 Performance Metrics

🔧 Advanced Configuration

LoRA Parameters

from peft import LoraConfig, get_peft_model

lora_config = LoraConfig(
    r=64,                              # Rank
    lora_alpha=128,                    # Scaling
    target_modules=["q_proj", "v_proj"],  # Attention heads
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

Agentic RAG Strategies

• —Keyword Search: Fast, lexical matching
• —Semantic Search: Slow, semantic matching
• —Hybrid Search: Combined approach
• —Verification Loop: Confidence-based re-search

📚 Training Details

Dataset Sources

• —Structured output datasets (JSON/YAML/XML)
• —Financial documents with labeled extractions
• —Technical documentation with parsed schemas
• —Business reports with key metric annotations

Training Objective

Maximize accuracy of structured extraction while preserving:

• —Chain-of-thought reasoning
• —Intermediate explanation quality
• —Hallucination prevention through Agentic verification

⚠️ Limitations & Future Work

Current Limitations

• —Requires labeled training data for domain adaptation
• —OCR quality dependent on PDF preprocessing
• —Multi-page document coherence needs improvement
• —Multilingual support under development

Roadmap

• —[ ] Multilingual variant (EN, JA, ZH)
• —[ ] Larger model variant (13B, 70B)
• —[ ] Fine-grained attribute extraction
• —[ ] Real-time streaming inference
• —[ ] Knowledge graph generation

📄 License

Apache 2.0

🙏 Acknowledgments

• —LLaVA: Vision Language Model foundation
• —PEFT: LoRA implementation
• —Sentence Transformers: Embedding models
• —FAISS: Vector search infrastructure

📧 Contact & Support

For questions, issues, or collaborations:

• —GitHub: https://github.com/Shion1124/vlm-lora-agentic-rag
• —HuggingFace: Shion1124/vlm-lora-agentic-rag

Updated: 2026-03-20 Status: Production-Ready