inesctec/CitiLink-BERTimbau-large-Topic-Classification-pt

BaselineBERTimbau-large-TopicClassification-Council-PT: Multi-Label Topic Classification for Portuguese Council Discussion Subjects

Model Description

BERTimbau Large Topic Classification Council PT is a baseline implementation, consisting of a fine-tuned version of neuralmind/bert-large-portuguese-cased (BERTimbau Large) for multi-label topic classification to automatically identify and categorize discussion subjects from Portuguese municipal council meeting minutes, optimized with dynamic per-label thresholds to identify multiple simultaneous topics within municipal discussion subjects.

Key Features

• —🎯 Specialized for Municipal Topics: Fine-tuned on Portuguese council meeting minutes discussion subjects
• —🧠 Large Transformer Model: 335M parameters (24 layers, 1024 hidden dim, 16 attention heads)
• —📊 Multi-Label Classification: Identifies multiple co-occurring topics per subject
• —⚡ Dynamic Thresholds: Optimized per-label classification thresholds (not fixed 0.5)
• —🇵🇹 Portuguese-Native: Built on BERTimbau, pre-trained on Brazilian Portuguese corpus
• —🔄 End-to-End Learning: Direct fine-tuning on task-specific data

Model Details

• —Architecture: BERT Large (Transformer Encoder)
• —Base Model: neuralmind/bert-large-portuguese-cased (BERTimbau Large)
• —Parameters: ~335M
• —24 transformer layers
• —1024 hidden dimensions
• —16 attention heads
• —4096 intermediate size
• —Max Sequence Length: 512 tokens
• —Learning Rate: 5e-5
• —Warmup: 0.1
• —Batch Size: 16
• —Optimizer: AdamW
• —Weight Decay: 0.01
• —Classification Head: Linear layer (1024 → 22 labels)
• —Loss Function: BCEWithLogitsLoss (multi-label)
• —Optimization: Dynamic per-label thresholds (F1-maximization)
• —Framework: PyTorch + Hugging Face Transformers

How It Works

The model processes Portuguese municipal texts through a fine-tuned transformer architecture:

1. 1.Text Preprocessing
2. 2.Basic normalization (lowercasing, whitespace cleanup)
3. 3.Minimum text length filtering (>10 chars)
4. 4.Punctuation removal for noise reduction

1. 1.Tokenization
2. 2.WordPiece tokenization (BERTimbau vocabulary)
3. 3.Max length truncation at 512 tokens
4. 4.Padding to max length for batching

1. 1.Transformer Encoding
2. 2.24-layer BERT encoder processes token sequences
3. 3.Self-attention captures contextual dependencies
4. 4.[CLS] token representation used for classification

1. 1.Multi-Label Prediction
2. 2.Linear classification head outputs 22 logits
3. 3.Sigmoid activation for independent probabilities
4. 4.Dynamic per-label thresholds for final predictions

1. 1.Threshold Optimization
2. 2.Each label has optimal threshold (0.10-0.90 range)
3. 3.Optimized via F1-score grid search on validation set
4. 4.Handles class imbalance better than fixed 0.5 threshold

Usage

import torch
import numpy as np
from transformers import AutoTokenizer, AutoModelForSequenceClassification
from joblib import load

# Load model and tokenizer
model_path = "path/to/model"
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModelForSequenceClassification.from_pretrained(model_path)

# Load optimal thresholds and label encoder
thresholds = np.load("optimal_thresholds.npy")
mlb = load("mlb_encoder.joblib")

# Set device
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = model.to(device)
model.eval()

# Classify text
text = "A Câmara Municipal aprovou o orçamento de 2024 para obras públicas e educação."

# Tokenize
inputs = tokenizer(
    text,
    return_tensors="pt",
    max_length=256,
    truncation=True,
    padding="max_length"
).to(device)

# Predict
with torch.no_grad():
    outputs = model(**inputs)
    probs = torch.sigmoid(outputs.logits).cpu().numpy()[0]

# Apply optimized thresholds
predictions = (probs >= thresholds).astype(int)
predicted_labels = mlb.inverse_transform(predictions.reshape(1, -1))[0]

# Display results
print(f"Text: {text}")
print(f"\nPredicted Topics:")
for label in predicted_labels:
    idx = list(mlb.classes_).index(label)
    prob = probs[idx]
    thresh = thresholds[idx]
    print(f"  - {label}: {prob:.4f} (threshold: {thresh:.2f})")

Categories

The model classifies topics into 22 Portuguese administrative categories:

Evaluation Results

Comprehensive Performance Metrics

License

This project uses a custom dual-license based on AGPL v3.

See the full license terms here: LICENSE