AnjaliRuban/idiolex-bertin-es

IdioleX-ES — Style-Aware Spanish Sentence Embeddings

IdioleX-ES is a sentence encoder trained under the IDIOLEX framework for idiolectal representation learning — capturing how text is expressed rather than what it says. Embeddings encode stylistic and dialectal variation across 17 Spanish varieties, decoupled from semantic content.

Kantharuban et al., IDIOLEX: Unified and Continuous Representations for Idiolectal and Stylistic Variation (preprint, under review). Code: github.com/AnjaliRuban/IdioleX

Architecture

input_ids → BERTIN (RoBERTa-base) → layer-wise attention → mean pool
          → mean centering → L2 normalize → embedding

The scalar-mix weights are learned jointly with the encoder, following Rei et al. (2020).

Training

Data

Training data consists of Reddit comments from 17 regional Spanish-language subreddits, collected via the Pushshift archive through December 2024 and filtered for language and quality. Pre-training uses ~557k authors and ~49.5M sentences. Feature-supervised training uses 200 authors per dialect with LLM-annotated linguistic features.

Linguistic Features

41 binary dialectal features are extracted sentence-by-sentence using GPT-5-mini, covering: subject pronoun use (yo, tú, usted, vos, vosotros, ustedes, etc.), verbal morphology (voseo present/imperative suffixes, 2pl forms -áis/-éis/-ís), diminutive suffixes (-ito/a, -ico/a, -illo/a, -ino/a, -ete, etc.), clitic patterns (DOM a, accusative/dative doubling, preverbal clitics, clitic sequences), and orthographic markers (inverted punctuation, all-caps, repeated punctuation).

Objectives

Training proceeds in two stages:

Stage 1 — Ranking pre-training (full dataset): A margin ranking loss encourages sentences with higher hierarchical proximity to be closer in embedding space. Each batch of 16 is structured so every sentence has exactly one same-comment neighbor (r=3), two same-author neighbors (r=2), four same-dialect neighbors (r=1), and eight cross-dialect neighbors (r=0). The margin λ warms up linearly from 0 to 0.5 over 25k steps.

Stage 2 — Feature-aware training (annotated subset, α=0.5):

A VICReg regularizer (weight 0.25) enforces variance ≥ 1 per dimension and decorrelates embedding dimensions to prevent anisotropy.

Hyperparameters

Performance

Dialect Identification — DSL-ML 2024 (multi-label)

Authorship Attribution — PAN 2019 (open-set, cross-domain)

PAN 2019 is a cross-domain benchmark (test domain not seen during training), so these results demonstrate transfer of stylistic features beyond topical content.

Semantic Decoupling

Pearson correlation between IdioleX-ES idiolectal similarity scores and Multilingual-E5 semantic similarity scores on withheld Reddit test pairs: ρ = 0.09. Stylistic and semantic similarity are largely independent.

Usage

import torch
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained(
    "your-username/idiolex-bertin-spanish",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained("your-username/idiolex-bertin-spanish")

sentences = [
    "Che, ¿me tirás una mano con esto?",   # Argentinian
    "Oye, ¿me echas una mano con esto?",   # Peninsular
]

inputs = tokenizer(
    sentences,
    return_tensors="pt",
    truncation=True,
    padding=True,
    max_length=model.config.model_len,  # 512
)

with torch.no_grad():
    embeddings = model(**inputs)   # [2, 768], L2-normalized

# Cosine similarity (embeddings are L2-normalized, so dot product = cosine sim)
similarity = embeddings @ embeddings.T
print(similarity)

Config

Custom files

This model uses trust_remote_code=True. The following files are hosted in this repo:

Citation

@article{kantharuban2025idiolex,
  title   = {IDIOLEX: Unified and Continuous Representations for Idiolectal and Stylistic Variation},
  author  = {Kantharuban, Anjali and Srivastava, Aarohi and Faisal, Fahim and Ahia, Orevaoghene
             and Anastasopoulos, Antonios and Chiang, David and Tsvetkov, Yulia and Neubig, Graham},
  year    = {2025},
  note    = {Preprint, under review}
}