Modelpublic

sascha-frank-ai-research/tsft-rag-gemma-3-1b-it

sourceHugging Facegemmaupdated 2mo agoView on Hugging Face

0likes89downloads

Model Card

TSFT-RAG Gemma-3-1B-IT

Task-Specific Full Fine-Tuning for Retrieval-Augmented Generation

Part of the TSFT-RAG model family for Retrieval-Augmented Generation across Gemma, Qwen and Llama architectures.

This repository contains TSFT-RAG Gemma-3-1B-IT, a full-parameter fine-tuned derivative of Google's Gemma-3-1B-IT developed within the TSFT-RAG research project. The objective is to investigate how task-specific full fine-tuning improves Retrieval-Augmented Generation (RAG), grounded question answering, unsupported-question detection and structured information extraction.

TSFT-RAG Model Series

Model	Parameters	Status
Gemma-3-1B-IT	1B	Released
Gemma-3-4B-IT	4B	Released
Gemma-3-12B-IT	12B	Released

Abstract

TSFT-RAG Gemma-3-1B-IT is a full-parameter fine-tuned derivative of Google's Gemma-3-1B-IT model developed for Retrieval-Augmented Generation (RAG). The objective of the TSFT-RAG project is to investigate whether relatively small language models can be specialized for reliable context-grounded answering, explicit abstention, structured output generation and information extraction through full-parameter supervised fine-tuning.

Research Background

Retrieval-Augmented Generation requires different optimization targets than general-purpose conversational assistants. Besides answering questions from retrieved evidence, practical RAG systems must reliably refuse unsupported questions, produce structured outputs and follow deterministic task instructions.

Model Details

Property	Value
Model family	TSFT-RAG
Base model	google/gemma-3-1b-it
Training implementation	unsloth/gemma-3-1b-it
Architecture	Gemma3ForCausalLM
Parameters	\~1B
Context length	32,768 tokens
Training sequence length	1,024 tokens
Precision	bfloat16

Training Procedure

Parameter	Value
Epochs	3
Learning rate	1e-6
Batch size	1
Gradient accumulation	16
Optimizer	Paged AdamW 8-bit
Scheduler	Cosine
Warmup ratio	0.08
Weight decay	0.01
Max gradient norm	1.0
Packing	disabled
Seed	42

Training runtime: 7 h 52 min

Final training loss: 1.1361

Evaluation

The benchmark contains 1,940 held-out German RAG evaluation instances covering grounded QA, unsupported questions, topic extraction, key-message extraction, argument extraction, structured JSON generation and citation-aware responses.

Metric	Base	TSFT-RAG	Difference
Aggregate score	0.328	0.605	+0.278
Hard-negative handling	0.020	0.815	+0.795
Grounded QA	0.509	0.395	-0.114
Analysis: main topic	0.550	0.546	-0.004
Analysis: key message	0.538	0.493	-0.046
Analysis: main arguments	0.404	0.546	+0.142

Discussion

The largest improvement is observed in unsupported-question handling. The fine-tuned model rejects unsupported questions substantially more reliably than the original base model while improving argument extraction. As expected for task-specific specialization, these improvements are accompanied by moderate reductions in grounded QA and key-message extraction. This reflects the intended optimization objective rather than a general capability increase.

Intended Use

—Local RAG systems
—Enterprise knowledge assistants
—University question answering
—Structured information extraction
—Research on task-specific fine-tuning

Reproducibility

The model was trained using the Unsloth implementation of Gemma with full-parameter supervised fine-tuning and the Hugging Face Transformers ecosystem.

Limitations

Performance outside the evaluated task formats has not been systematically studied. The model inherits limitations of the original Gemma base model and may generate incorrect or unsupported information.

License

This repository contains a modified derivative of Google's Gemma model. Use is subject to the Gemma Terms of Use. See LICENSE, NOTICE and MODIFICATIONS.md.

Associated Publication

This model was developed and evaluated as part of the following research:

Frank, S., & Singh, R. (2026). Task-Specific Full Fine-Tuning for Retrieval-Augmented Generation: A Multi-Family Evaluation Across Open-Weight Language Models.

Zenodo. DOI: 10.5281/zenodo.21638352

If you use this model in academic work, please cite the publication above.

bibtex

@misc{Frank2026TSFTRAG,
  author       = {Frank, Sascha and Singh, Rawel},
  title        = {Task-Specific Full Fine-Tuning for Retrieval-Augmented Generation:
                  A Multi-Family Evaluation Across Open-Weight Language Models},
  year         = {2026},
  month        = jul,
  version      = {v1},
  publisher    = {Zenodo},
  doi          = {10.5281/zenodo.21638352},
  url          = {https://doi.org/10.5281/zenodo.21638352},
  note         = {Preprint}
}

Project

The complete TSFT-RAG research project, including training scripts, evaluation pipeline, benchmark methodology and documentation, is available at:

Project repository https://github.com/frankmst/rag-task-specific-full-finetuning

TSFT-RAG model family https://huggingface.co/sascha-frank-ai-research

Author

Sascha Frank

Independent AI Researcher

ORCID https://orcid.org/0000-0002-0588-0081

GitHub https://github.com/frankmst

Hugging Face https://huggingface.co/sascha-frank-ai-research