mariamoracrossitcr/Llama-3.1-8B-INBioCR-sp-DAPT-ft-INBioCR_plants_QA

Llama-3.1-8B-INBioCR-sp-DAPT-ft-INBioCRplantsQA

This model is a supervised fine tuning of mariamoracrossitcr/Llama-3.1-8B-INBioCR-sp-DAPT for Spanish question answering on biodiversity data related to Costa Rican plants.

It is intended as a domain adapted assistant for answering plant related biodiversity questions in Spanish, using concise and factual responses grounded in curated structured data transformed into question-answer pairs.

Model lineage

This model follows the lineage:

meta-llama/Llama-3.1-8B-Instruct → mariamoracrossitcr/Llama-3.1-8B-INBioCR-sp-DAPT → Llama-3.1-8B-INBioCR-sp-DAPT-ft-INBioCR_plants_QA

The immediate parent for this fine tuning stage is:

• —mariamoracrossitcr/Llama-3.1-8B-INBioCR-sp-DAPT

Intended use

This model is designed for:

• —Spanish biodiversity question answering
• —Costa Rican flora related queries
• —educational and research oriented biodiversity assistance
• —domain specific response generation from plant knowledge

Training data

The model was fine tuned using the dataset:

• —mariamoracrossitcr/INBioCR_plants_QA

The training script formats each example as a chat conversation with:

• —a system instruction in Spanish
• —a user question
• —an assistant answer

The system message used in training was:

Eres un asistente experto en biodiversidad y conservación en Costa Rica. Responde en español claro, preciso y basado en datos. Si la pregunta no se puede responder con la información disponible, dilo explícitamente.

Training procedure

This model was trained with PEFT LoRA using trl.SFTTrainer. Hardware: Kabre.cenat.ac.cr NVIDIA L40 48GB

Base model loading

• —Base model: mariamoracrossitcr/Llama-3.1-8B-INBioCR-sp-DAPT
• —Tokenizer: meta-llama/Llama-3.1-8B-Instruct
• —Precision: bfloat16

LoRA configuration

• —Rank r = 32
• —lora_alpha = 32
• —lora_dropout = 0.05
• —Bias: none
• —Task type: CAUSAL_LM

Target modules:

• —o_proj
• —q_proj
• —v_proj
• —k_proj
• —gate_proj
• —up_proj
• —down_proj

Training arguments

• —Per device train batch size: 2
• —Gradient accumulation steps: 8
• —Effective batch accumulation used for memory efficient fine tuning
• —Number of epochs: 3
• —Learning rate: 1e-4
• —Scheduler: cosine
• —Warmup ratio: 0.1
• —Weight decay: 0.05
• —Evaluation strategy: every 200 steps
• —Save strategy: every 200 steps
• —Logging steps: 20
• —bf16 = True
• —Gradient checkpointing enabled
• —load_best_model_at_end = True
• —Best model selected by lowest validation loss

Early stopping

• —Patience: 5
• —Threshold: 0.01

Training results

The training log shows a steady reduction in validation loss during fine tuning.

Observed validation losses from the provided log include:

• —Epoch 0.15: 1.4776
• —Epoch 0.30: 1.4007
• —Epoch 0.45: 1.1877
• —Epoch 0.60: 1.0944
• —Epoch 0.75: 0.9866 --- license: apache-2.0 language:
• —es base_model:
• —meta-llama/Llama-3.1-8B-Instruct pipeline_tag: question-answering ---- Epoch 0.90: 0.9541
• —Epoch 1.05: 0.7356
• —Epoch 1.20: 0.6595
• —Epoch 1.35: 0.5773
• —Epoch 1.50: 0.4970
• —Epoch 1.65: 0.4441
• —Epoch 1.80: 0.3958

Best validation loss observed in the provided output:

• —eval_loss = 0.3958 at epoch 1.80

Prompt format

This model was trained with chat formatted examples. A typical prompt structure is:

<system>
Eres un asistente experto en biodiversidad y conservación en Costa Rica. Responde en español claro, preciso y basado en datos. Si la pregunta no se puede responder con la información disponible, dilo explícitamente.

<user>
¿En qué tipo de bosques se puede encontrar con frecuencia Perrottetia longistylis?

<assistant>
...


Limitations
The model is specialized for the training domain and language.
It may hallucinate when the dataset does not provide enough evidence.
It should not be treated as a substitute for expert taxonomic or conservation judgment.
Performance may degrade for out of domain species or questions not represented in training.