Petermoyano/unsloth-gemma-7b-bnb-4bit-LoRA-Tipification-CausalLM-16R-16Alpha-1Epoch

Gemma2 Fine-Tuned LoRA Model

Overview

This is a LoRA (Low-Rank Adaptation) fine-tuned model based on the `unsloth/gemma-7b-bnb-4bit` base model. It has been adapted for a tipification analysis task similar to the Llama-3.2-3B-Instruct LoRA fine-tuning, where the model classifies text into categories such as "ESTAFA," "ROBO," "HURTO," and their "TENTATIVA DE" variations.

During fine-tuning, only specific adapter layers were trained (\~50 million parameters), while the rest of the base model was frozen. This approach allows parameter-efficient training, significantly reducing computational costs.

Key Features

• —Base Model: unsloth/gemma-7b-bnb-4bit
• —Task Type: Causal Language Modeling (CAUSAL_LM)
• —LoRA Parameters:
• —r: 16
• —lora_alpha: 16
• —lora_dropout: 0.0
• —Target Modules:
• —gate_proj, up_proj, down_proj, k_proj, q_proj, o_proj, v_proj
• —Number of Trainable Parameters: 50,003,968
• —Training Loss & Validation Loss:
• —Observed over 117 steps (1 epoch).
• —See table below for detailed step-by-step values.

Dataset Distribution

This model was fine-tuned on the same dataset as the Llama-3.2-3B-Instruct LoRA version, with the following category distribution:

Although the dataset has nearly 10K examples in this summary table, the fine-tuning run used an extended version (\~15K examples) for this particular training session.

Training Details

• —Hardware: Single GPU A100 40Gb
• —Num Examples: ~15,000
• —Epochs: 1
• —Batch Size per Device: 32
• —Gradient Accumulation Steps: 4
• —Effective Total Batch Size: 128
• —Total Steps: 117
• —Number of Trainable Parameters: 50,003,968

Training and Validation Loss

Below is a snapshot of how training and validation loss evolved during the single epoch (117 steps):

Final training concluded at step 117. We observe a steady decrease in both training and validation losses, indicating the model was converging throughout the single epoch.

Deployment Instructions

You can use this LoRA fine-tuned model with the Hugging Face Transformers library. Below is an example of how to load and run the model for text generation or classification-like tasks:

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Petermoyano/unsloth-gemma-7b-bnb-4bit-LoRA-Tipification-CausalLM-16R-16Alpha-1Epoch")
model = AutoModelForCausalLM.from_pretrained("Petermoyano/unsloth-gemma-7b-bnb-4bit-LoRA-Tipification-CausalLM-16R-16Alpha-1Epoch")

input_text = "TENTATIVA DE ESTAFA:"
inputs = tokenizer(input_text, return_tensors="pt")
outputs = model.generate(**inputs, max_length=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))