Flaglab/esnlir-llm-predictions

ESNLIR-LLM — per-pair predictions

Per-pair predictions for every model evaluated in An Analysis of the Performance of Large Language Models in Spanish NLI Datasets with Causal Relationships (IBERAMIA 2026, to appear). Code in `Pacolas/NLI-via-LLM`; part of the **ESNLIR-LLM** collection.

These are the raw outputs behind the paper's tables, so results can be re-scored, sliced by genre or domain, or compared pair by pair without re-running any model.

Files

qwen25-7b/esnlir_test.csv        llama31-8b/esnlir_test.csv        xlmroberta-esnlir/esnlir_test.csv
qwen25-7b/human_validated.csv    llama31-8b/human_validated.csv    xlmroberta-esnlir/human_validated.csv
gpt4o-mini/esnlir_test.csv       gpt4o/human_validated.csv
gpt4o-mini/human_validated.csv

Accuracies are recomputed directly from these files and match the paper.

What is not here

GPT-4o has no full-test-set file. That model was run only on the small and human-validated sets, for cost reasons, which is why the paper reports n/a for it on the 80,213-pair split. Re-creating it means re-running the API notebooks in `Pacolas/NLI-via-LLM`, which costs money and will not reproduce exactly, since the endpoints are not deterministic.

Which run each file contains

Both proprietary models were called more than once over the validated pairs, at different times against a non-deterministic endpoint, so several prediction sets exist for each. They are genuine re-runs, not a bookkeeping error: across the 972 pairs no two runs share a single explanation string, so nothing is misaligned.

The paper reports, for every model, the run published here — so each row of its Table 3 recomputes from these files. Concretely:

• —GPT-4o-Mini: the full-test-set run, not the one recorded alongside the human annotations. The two disagree on 336 of the 972 pairs (0.5556 against 0.4918 accuracy). Using the full-test run means a pair carries the same prediction whichever split you load it from.
• —GPT-4o: the run in this file, 0.6029 accuracy / 0.5727 macro F1.

Saved outputs inside the notebooks in `Pacolas/NLI-via-LLM` record earlier sessions and quote different runs in places; where they disagree, these files are authoritative.

The GPT-4o-Mini test file here is reconstructed from the raw run in that repository's ESNLIR_final_test/: 80,046 responses plus 123 repaired ones = 80,169, the base the paper reports for the proprietary models, minus the 30 ambiguous repeated pairs described below.

Verify every number yourself

metrics/verify_metrics.py recomputes each figure the paper reports, straight from the prediction files in this repo. Nothing is read from a saved result.

pip install numpy pandas huggingface_hub
python metrics/verify_metrics.py            # print the full report
python metrics/verify_metrics.py --out .    # also write csv/json

Its output is checked in beside it, so you can diff without running anything:

Conventions, stated because they change the digits:

• —Accuracy CI is the Wilson score interval, not the normal approximation, so it stays inside [0, 1] and behaves at the small sample sizes.
• —Macro F1 CI is a percentile bootstrap over 2,000 resamples, seed 0. Resampling rows is done by drawing the confusion matrix from a multinomial over its cells, which is equivalent because macro F1 depends on the rows only through those counts, and makes the bootstrap essentially free.
• —McNemar uses Yates' continuity correction, (|b - c| - 1)^2 / (b + c), matching the paper and the statsmodels default. --no-continuity gives the uncorrected form; it shifts chi-square by about 1 and changes no conclusion.
• —Unparseable predictions count as errors rather than being dropped, so accuracy is never inflated by discarding hard cases.

Two results worth noting that the paper does not spell out. On the validated subset GPT-4o-Mini and Qwen2.5-7B are statistically indistinguishable pair by pair (chi-square 0.3, p = 0.58), which corroborates the overlapping-interval argument with a paired test. GPT-4o does beat GPT-4o-Mini there (chi-square 6.8, p = 0.009) even though their intervals overlap, since the paired test uses information the marginal intervals discard.

Columns

Labels are contrasting / entailment / neutral / reasoning.

Two things to know before comparing

Row sets differ slightly between models. The open-weight models cover all 80,213 test pairs. XLM-RoBERTa covers 80,183 and GPT-4o-Mini 80,139: 30 pairs are dropped in both because their sentences occur more than once, so a prediction cannot be attributed to one row unambiguously, and GPT-4o-Mini additionally lost 44 pairs (40 of them Neutral) to Azure's content filter. Always join on row_idx and intersect before any pair-by-pair test.

The LLMs nearly abandon Entailment. Qwen2.5-7B reaches 0.54 precision on Entailment at 0.08 recall; Llama-3.1-8B is close to identical (0.41, 0.08). Both effectively collapse the four-way task into a three-way one. XLM-RoBERTa shows no such collapse. This is a finding, not a bug in the files.

Usage

Each model is a named config, so the files are browsable in the dataset viewer and load directly:

from datasets import load_dataset

qwen = load_dataset("Flaglab/esnlir-llm-predictions", "qwen25-7b", split="esnlir_test")

Or a DataFrame per file, which is easier for pair-by-pair work:

import pandas as pd
from huggingface_hub import hf_hub_download

def load(model, split):
    p = hf_hub_download("Flaglab/esnlir-llm-predictions", f"{model}/{split}.csv",
                        repo_type="dataset")
    return pd.read_csv(p)

qwen = load("qwen25-7b", "esnlir_test")
xlmr = load("xlmroberta-esnlir", "esnlir_test")

print((qwen.gold == qwen.pred).mean())          # 0.41048

# pair-by-pair, on the rows both systems cover
both = qwen.merge(xlmr, on="row_idx", suffixes=("_qwen", "_xlmr"))
print(len(both))                                 # 80,183
print(((both.pred_xlmr == both.gold_xlmr) & (both.pred_qwen != both.gold_qwen)).sum())

Citation

These predictions were produced for the following paper:

@InProceedings{perez2026llmspanishnlicausal,
  author    = {P{\'e}rez, Nicol{\'a}s and Portela, Johan R. and Manrique, Ruben},
  title     = {An Analysis of the Performance of Large Language Models in Spanish
               NLI Datasets with Causal Relationships},
  booktitle = {Advances in Artificial Intelligence -- IBERAMIA 2026},
  year      = {2026},
  publisher = {Springer Nature Switzerland},
  address   = {Cham},
  note      = {To appear},
}

The pairs being predicted come from ESNLIR, released here under CC BY 4.0 — attribution to the source corpus is a condition of that licence:

@InProceedings{portela2025esnlirspanishmultigenredataset,
  author    = {Portela, Johan R. and P{\'e}rez-Ter{\'a}n, Nicol{\'a}s and Manrique, Rub{\'e}n},
  editor    = {Florez, Hector and Peluffo-Ordo{\~{n}}ez, Diego},
  title     = {{ESNLIR}: Expanding Spanish {NLI} Benchmarks with Multi-genre and Causal Annotation},
  booktitle = {Applied Informatics},
  year      = {2026},
  publisher = {Springer Nature Switzerland},
  address   = {Cham},
  pages     = {345--361},
  isbn      = {978-3-032-07175-0},
  doi       = {10.1007/978-3-032-07175-0_23},
}