Polygl0t/portuguese-eval-logs-olmo2-smollm3

Evaluation Logs on Portuguese Benchmarks for OLMo-2 and SmolLM3

These logs contain benchmark results across a suite of Portuguese-language tasks. The data consists of recordings of the performance of various 3 different models at different checkpoints throughout their pretraining runs:

• —SmolLM3
• —OLMo-2-0425-1B
• —OLMo-2-1124-7B

Splits

Each split (smollm3_3b, olmo2_1b, olmo2_7b) contains rows for model checkpoints and columns for benchmark scores (e.g., ASSIN2 RTE, ENEM, BLUEX, OAB, etc.).

Data Format

The checkpoint column indicates the branch from which the checkpoint was taken (e.g., stage1-step-40000). step indicates the training step at which the checkpoint was saved. The remaining columns show the model's performance across various benchmarks.

{
  "Checkpoint": "stage1-step-40000",
  "Step": 40000,
  "ASSIN2 RTE": 0.5369919728357186,
  "ASSIN2 STS": 0.1356046511596823,
  "BLUEX": 0.2002781641168289,
  "ENEM": 0.1980405878236529,
  "FAQUAD NLI": 0.5373977569778228,
  "HateBR": 0.5875147596226018,
  "OAB": 0.2323462414578587,
  "PT Hate Speech": 0.5555178900597524,
  "TweetSentBR": 0.3303917299738176,
  "ARC Challenge": 0.288034188034188,
  "ASSIN2 ENT": 0.58075,
  "ASSIN2 PAR": 0.6365,
  "BELEBELE": 0.23,
  "CALAME": 0.5130057803468208,
  "Global PIQA": 0.65,
  "HellaSwag": 0.3650449669519991,
  "LAMBADA": 0.4723462060935377,
  "MMLU": 0.2284599219453617
}

~100 billion tokens separate each checkpoint. Still, due to differences in checkpoint saving frequency and batch size, the actual token counts between checkpoints may vary slightly.

How to Use

from datasets import load_dataset

# Loads the SmolLM3 split
ds = load_dataset("Polygl0t/portuguese-eval-logs-olmo2-smollm3", split="smollm3_3b")

Benchmarks

The benchmarks are sourced from lm-evaluation-harness (branch: polyglot_harness_portuguese) and lm-evaluation-harness-pt, which provide a standardized set of Portuguese-language tasks for LLM evaluation.

The following benchmarks are included in this log:

• —ENEM: Brazilian high-school exam, Q&A format (dataset).
• —BLUEX: University entrance exam questions (Unicamp/Fuvest), Q&A format (dataset).
• —OAB Exams: Brazilian Bar Association exam questions, Q&A format (dataset).
• —ASSIN2 RTE: Textual entailment / natural language inference (dataset).
• —ASSIN2 STS: Semantic textual similarity (dataset).
• —FAQUAD NLI: Entailment task based on Portuguese reading comprehension (dataset).
• —HateBR: Abusive language detection in Brazilian Portuguese social media (dataset).
• —PT Hate Speech: Hate speech detection in Portuguese tweets (dataset).
• —TweetSentBR: Sentiment analysis on Brazilian Portuguese tweets (dataset).
• —ARC Challenge: Multiple-choice grade-school science questions (Portuguese translation) (dataset).
• —ASSIN2 ENT: Textual entailment (natural language inference), not generative (dataset).
• —ASSIN2 PAR: Paraphrase detection from the ASSIN2 dataset (dataset).
• —BELEBELE: Multilingual reading comprehension (Portuguese subset)(dataset).
• —CALAME: Predict the last word of a passage — Portuguese version (similar to LAMBADA) (dataset).
• —Global PIQA: Physical commonsense reasoning (Brazilian Portuguese subset) (dataset).
• —HellaSwag: Commonsense inference (Portuguese translation) (dataset).
• —LAMBADA: Predict the last word of a passage (Portuguese translation) (dataset).
• —MMLU: Multitask language understanding (Portuguese translation) (dataset).

Usage and Purpose

• —Benchmark Analysis: Track how model performance evolves during pretraining.
• —Evaluation Research: Assess the reliability and signal quality of different benchmarks.
• —Model Comparison: Compare Portuguese language understanding across different LLMs and training regimes.

E.g, SmolLM3-3B Performance on the ENEM benchmark across checkpoints:

[image]

Other plots can be found in the `.plots` directory.

Citation Information

@misc{correa2026tucano2cool,
      title={{Tucano 2 Cool: Better Open Source LLMs for Portuguese}}, 
      author={Nicholas Kluge Corr{\^e}a and Aniket Sen and Shiza Fatimah and Sophia Falk and Lennard Landgraf and Julia Kastner and Lucie Flek},
      year={2026},
      eprint={2603.03543},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2603.03543}, 
}

Acknowledgments

Polyglot is a project funded by the Federal Ministry of Education and Research (BMBF) and the Ministry of Culture and Science of the State of North Rhine-Westphalia (MWK) as part of TRA Sustainable Futures (University of Bonn) and the Excellence Strategy of the federal and state governments.

We also gratefully acknowledge the granted access to the Marvin cluster hosted by University of Bonn along with the support provided by its High Performance Computing & Analytics Lab.

License

All data in this dataset is licensed under the Apache License 2.0.