flaviussteff/base-ro-125m

Base-Ro-125M: Romanian Foundation Causal Language Model

Base-Ro-125M is a 124.8M-parameter causal language model based on the modern LLaMA architecture, trained from scratch ("Token Zero") purely on Romanian web text.

Model Architecture

Pretraining Details

• —Corpus: Cleaned Romanian web text comprising Romanian Wikipedia, 5-year news archives (HotNews, Digi24, G4Media), and FineWeb-2 Romanian web shards.
• —Tokens Trained: ~3.27 Billion tokens (50,000 additional steps, total 60,000 steps with effective batch size 64 $\times$ 1,024 context).
• —Hardware: Single NVIDIA GeForce RTX 3060 12GB GDDR6 (Mixed Precision BF16 / FlashAttention SDPA).
• —Training Time: ~50 hours continuous pretraining.
• —Final Pretraining Loss: 2.833 | Perplexity: 16.99.

Usage with Transformers

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "flaviussteff/base-ro-125m"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16 if torch.cuda.is_bf16_supported() else torch.float16,
    device_map="auto",
)

prompt = "În România contemporană, dezvoltarea economică"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=60,
    do_sample=True,
    temperature=0.7,
    top_p=0.9,
    repetition_penalty=1.15,
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Evaluation Benchmark

Evaluated on the parallel Romanian-English socio-cultural bias benchmark across 37 counterfactual diagnostic pairs:

• —Overall Stereotype Preference Metric ($SPM$): 56.76% (where 50.0% is neutral).
• —Gender & Occupation ($SPM$): 90.0% (absorbs prevailing web stereotypes regarding executive leadership, domestic roles, and caretaking).
• —Minority Framing ($SPM$): 62.5%.

Citation & Academic Context

Developed as part of academic research at the Faculty of Mathematics and Computer Science, University of Bucharest.