bsana1/arrodeio-tucano-cordel-lora

arrodeio-tucano-cordel-lora

A LoRA adapter that teaches TucanoBR/Tucano-1b1-Instruct to write in the style of Brazilian cordel poetry and glosa (mote → glosa, the classic form where a poet expands a given saying into a rhyming stanza).

This is the "v2" model of [arrodeio-llm](https://github.com/bsana1/arrodeio-llm), a small hands-on project for learning how LLM pretraining, fine-tuning, and inference actually work — by building each phase from scratch on a from-scratch GPT, then repeating the fine-tuning/SFT phases on a real pretrained Portuguese instruct model (this adapter).

Try it live: arrodeio-llm-gradio Space (pick "v2 · Tucano + SFT" in the demo).

Model Details

Model Description

This repo contains only the LoRA adapter weights (~18 MB) — not a merged model. Load it on top of the base model with 🤗 PEFT (see below).

• —Developed by: Bernardo Sana (bsana1)
• —Model type: Causal decoder-only LLM, LoRA adapter (rank 16)
• —Language(s): Portuguese (pt-BR)
• —License: Apache 2.0 — inherited from the base model
• —Finetuned from model: TucanoBR/Tucano-1b1-Instruct (1.1B params, Llama2-style architecture, natively pretrained in Portuguese on GigaVerbo by the Tucano team at the University of Bonn)

Model Sources

• —Repository: github.com/bsana1/arrodeio-llm
• —Demo: arrodeio-llm-gradio Space

Uses

Direct Use

Instruction-following text generation in Portuguese, specifically for cordel/glosa-style poetry: given a mote (a proverb or saying) or a theme, generate a rhyming stanza in that tradition. Works through the base model's chat template (see usage example below).

Out-of-Scope Use

This is a small hobby/learning project, not a production model. It was fine-tuned on ~650 examples (50 hand-written + 600 synthetic) — it has not been evaluated for factual accuracy, safety, or any use outside creative Portuguese-language poetry generation, and shouldn't be used for anything that requires reliability guarantees.

Bias, Risks, and Limitations

Inherits the base model's general limitations (Tucano-1b1-Instruct, 1.1B params). The fine-tuning set is small and cordel poetry draws on 19th/20th century Northeastern Brazilian folk material, so outputs can reflect the period's language, social attitudes, and references. Not evaluated for bias.

How to Get Started with the Model

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = "TucanoBR/Tucano-1b1-Instruct"
adapter = "bsana1/arrodeio-tucano-cordel-lora"

tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, dtype=torch.bfloat16)
model = PeftModel.from_pretrained(model, adapter)
model.eval()

msg = [{"role": "user", "content": "Glose o mote: «Água mole em pedra dura.»"}]
ids = tok.apply_chat_template(msg, tokenize=True, add_generation_prompt=True, return_tensors="pt")
out = model.generate(ids, max_new_tokens=110, do_sample=True, temperature=0.7,
                      top_k=50, repetition_penalty=1.2, pad_token_id=tok.eos_token_id)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))

Training Details

Training Data

650 mote → glosa / tema → cordel instruction pairs:

• —50 hand-written by the project author (data/sft.jsonl)
• —600 synthesized via instruction backtranslation (data/sft_synth.jsonl) — real public-domain cordel stanzas (from pt.wikisource.org, long-dead authors only) paired with a fabricated instruction that would plausibly produce them, generated by scripts/make_sft_data.py

Training Procedure

LoRA fine-tuning with prompt-masked loss (loss computed only on the response tokens, via the base model's chat template).

Training Hyperparameters

• —LoRA rank (r): 16
• —LoRA alpha: 32
• —LoRA dropout: 0.05
• —Target modules: q_proj, k_proj, v_proj, o_proj
• —Learning rate: 2e-4
• —Epochs: 2
• —Training regime: bf16
• —Task type: CAUSAL_LM

Speeds, Sizes, Times

Trained on a free Google Colab T4 GPU in about 10 minutes. Adapter weights: ~18 MB.

Evaluation

No formal benchmark — evaluated qualitatively against the earlier stages of the same project (from-scratch model, fine-tuned GPT-2). See samples/comparison.md in the GitHub repo for the same prompts run through every stage side by side.

Environmental Impact

Training took ~10 minutes on a single shared T4 GPU (Google Colab free tier) — negligible compute. Inference in the live demo runs on Hugging Face's ZeroGPU (shared A10G, on-demand).

Technical Specifications

Model Architecture and Objective

Base: Tucano-1b1-Instruct, a Llama2-style causal decoder-only transformer (1.1B params) natively pretrained in Portuguese. This adapter adds LoRA matrices to the attention projections only (q/k/v/o_proj); all base weights stay frozen.

Compute Infrastructure

Hardware

Google Colab, free-tier T4 GPU.

Software

PyTorch, 🤗 Transformers, 🤗 PEFT.

• —PEFT 0.20.0

Model Card Contact

Questions/issues: open one on github.com/bsana1/arrodeio-llm.