somosnlp-hackathon-2025/llava-v1.6-mistral-7b-memes-chilenos-small

llava-v1.6-mistral-7b-memes-chilenos-small

A LoRA‑fine‑tuned version of LLaVA‑Next for explaining Chilean memes in colloquial Spanish, built for the Somos NLP Hackathon 2025.

<img src="./tralalelo-tralala-logo.png" alt="Banner" width="70%" />

Project Overview

Introduction

Memes are more than jokes—they are cultural micro-units.

According to James Gleick (2011), memes are:

“nothing more than ideas, images, slogans, melodies, stories, recipes, skills or abilities, legends, and systems that populate our minds.”

Ryan Milner (2012) defines them as:

“amateur media artifacts, extensively remixed and recirculated by different participants on social media networks”

These definitions highlight the richness and complexity behind meme culture. Yet, vision-language models like LLaVA often struggle to interpret them—especially when memes rely on local context, irony, or informal language.

This project adapts LLaVA‑Next to the Chilean cultural landscape via targeted fine‑tuning, enabling the model to explain memes in Chilean Spanish, grounded in the humor, expressions, and imagery found in real social media content.

Motivation

While vision-language models like LLaVA‑Next have shown impressive results in traditional tasks (e.g., image captioning, VQA), they often struggle with highly contextual and culturally specific content such as memes—especially those rooted in local humor, slang, and social cues.

In the Chilean context, memes frequently rely on implicit cultural knowledge, regional dialects, and shared national experiences that general-purpose models are not trained to recognize. Moreover, no vision-language model had been fine-tuned specifically to interpret memes in colloquial Chilean Spanish.

Recent work such as MemeGuard (Liu et al., 2024) highlights this challenge, noting that memes are inherently ambiguous, multi-layered, and culturally grounded, making them difficult even for strong multimodal baselines.

This motivated us to fine-tune LLaVA‑Next using a custom dataset of Chilean Instagram memes, enriched with instruction-based annotations and cultural explanations.

Impact

This work contributing to research on:

• —Cultural alignment of vision-language models.
• —Educational, social media, and discourse analysis applications.
• —Advanced fine-tuning strategies beyond standard captioning/VQA tasks.

It also offers a replicable methodology for developing culturally adapted models in other regions—especially across Latin America.

Model Details

Model Description

llava-v1.6-mistral-7b-memes-chilenos-small adapts LLaVA‑Next to the highly contextual humour found in Chilean memes. Because no public corpus of memes explained in Chilean Spanish existed, we scraped 1194 image‑text posts from popular Chilean Instagram meme pages, wrote human explanations, and expanded the corpus to 3 568 examples with GPT‑4‑o and command-r-plus (Cohere) using four instruction‑tuning templates:

1. 1.“Explica qué significa este meme en Chile, usando lenguaje coloquial…”
2. 2.“Explica cómo entendería este meme alguien que vive en Chile…”
3. 3.“Describe por qué este meme sería gracioso o significativo para un chileno…”
4. 4.“Imagina que le explicas este meme a alguien extranjero…”

The result is a compact model that can describe why a meme is funny for a Chilean audience, though it still struggles with extremely time‑sensitive or highly niche references.

Both during training and inference, we used the following system prompt to guide the model’s behavior and cultural framing: “Eres experto en memes chilenos. Observa la imagen y, si hay texto, interprétalo sin repetirlo. Analiza su sentido usando contexto cultural chileno. Responde según la instrucción.”

Bias, Risks & Limitations

• —Temporal drift – many memes reference current events; explanations may become dated quickly.
• —Subjectivity of humour – what is “funny” varies; the model reflects the curators’ viewpoint.
• —Dataset biases – Instagram accounts skew toward urban, younger demographics; regional slang may be under‑represented.
• —Vision safety – the model was not filtered for possibly offensive or unsafe imagery.

Recommendations

Always present model outputs with a disclaimer that humour is subjective and culturally bound. Human review is recommended before publishing explanations.

How to Get Started

from transformers import LlavaNextForConditionalGeneration, AutoProcessor
from peft import PeftModel
import torch
from PIL import Image

BASE_MODEL = "llava-hf/llava-v1.6-mistral-7b-hf"
ADAPTER_ID = "AndresSebad/llava-v1.6-mistral-7b-memes-chilenos-small"

device = "cuda"  # or "cpu" if no GPU is available

# Load the frozen base checkpoints in FP16 and attach the LoRA adapters
base_model = LlavaNextForConditionalGeneration.from_pretrained(
    BASE_MODEL,
    torch_dtype=torch.float16,
    device_map="auto"
)
model = PeftModel.from_pretrained(base_model, ADAPTER_ID)

model = model.eval()
model = model.to(device)

processor = AutoProcessor.from_pretrained(BASE_MODEL)
processor.tokenizer.padding_side = "right"

img = Image.open("meme.jpg")
conversation = [{
    "role": "user",
    "content": [
        {"type": "image"},
        {"type": "text", "text": "Eres experto en memes chilenos. Observa la imagen y, si hay texto, interprétalo sin repetirlo. Analiza su sentido usando contexto cultural chileno. Responde según la instrucción."},
        {"type": "text", "text": "Explica qué significa este meme en Chile, usando lenguaje coloquial chileno."}
    ]
}]

prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)
inputs = processor(text=prompt, images=[img], return_tensors="pt").to(device)

out_ids = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(out_ids[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0])

Training Details

Training data

• —1 194 real Chilean memes (image + caption + human explanation).
• —3 582 synthetic examples generated with GPT‑4‑o using the templates above.

Pre‑processing

• —Images resized to 336 × 336.
• —Mixed‑precision (fp16) training.

Hyper‑parameters

Environmental Impact

Estimated total location-based greenhouse gas emissions were 190.96 grams CO₂eq for fine-tuning.

Training was performed on a single NVIDIA L40S GPU over approximately 1.45 hours, tracked using the CodeCarbon library.

No renewable energy matching or carbon offsets were applied. The training took place in a datacenter estimated to be located in Virginia, USA based on IP geolocation.

Note: Results are approximate and based on available metadata.

Example 1 – “18 de Septiembre”

<img src="./perro18.png" alt="Example1" width="50%" />

Example 2 – “No estoy entendiendo...”

<img src="./perro.png" alt="Example2" width="50%" />

Technical Specifications

• —Architecture – Mistral‑7B decoder with multimodal projector; unchanged aside from LoRA adapters.
• —Compute – fine‑tuned on a single GPU; no distributed or multi‑node training.

Citation

BibTeX

@software{llava_memes_chile_2025,
  author       = {Somos NLP Hackathon 2025},
  title        = {llava-v1.6-mistral-7b-memes-chilenos-small},
  year         = 2025,
  publisher    = {Hugging Face},
  url          = {https://huggingface.co/somosnlp-hackathon-2025/llava-v1.6-mistral-7b-memes-chilenos-small}
}

Glossary

• —LoRA – Low‑Rank Adaptation; inserts lightweight trainable matrices.
• —Instruction‑tuning – generating diverse prompts to improve alignment.
• —Mistral‑7B – a 7‑billion‑parameter transformer decoder.

Hackathon: This model was developed for Somos NLP Hackathon 2025 – see the project page here.

Resources

• —🔗 GitHub repository – Source code, training scripts, and the Gradio demo.