Tushar9802/hybrid-summariser-crossmodal-lora

Hybrid-Summariser Cross-Modal LoRA (Phase 3)

Known limitation (July 2026): this checkpoint produces repetitive token loops on a substantial fraction of video inputs at greedy or unblocked decoding; the results below were measured with no_repeat_ngram_size=3, which masks but does not remove the artifact. This checkpoint is preserved as the artifact of the March 2026 conference paper. A follow-up study re-evaluating this pipeline is in preparation; a successor adapter will be linked here on publication.

LoRA adapter for Mistral-7B-v0.1 trained with a 3-phase curriculum framework that prevents catastrophic forgetting during cross-modal summarization. Produces domain-aware summaries of CS lecture videos without academic style contamination.

Paper: Follow-up to "Cross-Modal Transfer Learning in Domain-Adaptive Video Summarization" (IMPACT 2025 - Springer proceedings, 2026)

Repo: github.com/Tushar-9802/Hybrid-Dataset-Summariser

Results (vs zero-shot Mistral-7B, n=75 videos, p < 0.005)

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel
import torch

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)
model = AutoModelForCausalLM.from_pretrained(
    "mistralai/Mistral-7B-v0.1",
    quantization_config=bnb_config,
    device_map="auto",
)
model = PeftModel.from_pretrained(model, "Tushar9802/hybrid-summariser-crossmodal-lora")
tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1")

prompt = "Summarize the following text.\n\nText: {your_text}\n\nSummary:"
inputs = tokenizer(prompt, return_tensors="pt", truncation=True, max_length=1024).to("cuda")
outputs = model.generate(**inputs, max_new_tokens=256, num_beams=4, no_repeat_ngram_size=3)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Training

Three-phase curriculum on RTX 5070 Ti (16GB VRAM):

Dataset

4,324 CS samples: 2,368 arXiv papers + 738 YouTube lectures + 1,218 SBERT-mined cross-modal pairs. Kaggle

Hyperparameters

• —LoRA: r=32, alpha=64, dropout=0.1, targets=q,k,v,o,gate,up,down
• —Trainable: 83.9M params (1.16% of 7.24B total)
• —Optimizer: 8-bit AdamW, effective batch size 24
• —Quantization: 4-bit NF4, double quant, bfloat16 compute
• —Peak VRAM: 11.3 GB (Phase 3)

Training Procedure

Phase 1 trains on papers only to acquire domain vocabulary. Fisher Information (83.9M entries across 448 parameter matrices) and SVD (top-16 singular directions per module) computed at exit.

Phase 2 introduces videos and cross-modal pairs with OPLoRA orthogonal projection preventing subspace contamination, EWC preserving critical paper summarization parameters, and a 10% replay buffer from Phase 1.

Phase 3 shifts to video-heavy training with contrastive cross-modal alignment and increased elastic regularization.

Limitations

• —CrossCLR exhibited NaN on some pair batches (partially limiting contrastive alignment)
• —Single model (Mistral-7B) and domain (CS/engineering) — generalization untested
• —Video references generated by GPT-4o-mini, not human annotators
• —No human evaluation conducted

Citation

@inproceedings{jaju2025crossmodal,
  title={Cross-Modal Transfer Learning in Domain-Adaptive Video Summarization},
  author={Jaju, Tushar and Saharawat, Tanishka and Bhatia, Shruti and Rastogi, Shivansh},
  booktitle={Proc. IMPACT 2025},
  publisher={Springer},
  year={2025}
}

Authors

Tushar Jaju (training infrastructure, implementation, experiments), Tanishka Saharawat, Shruti Bhatia, Shivansh Rastogi

Guide: Dr. Neha Yadav — ABES Engineering College, Ghaziabad (AKTU)

Framework Versions

• —PEFT: 0.18.1
• —Transformers: 5.0.0rc3
• —PyTorch: 2.11.0.dev20260214+cu128
• —bitsandbytes: 0.49.2
• —Python: 3.11