nishan-chatterjee/aspect-based-sentiment-analysis

AspectBench: reusable document-level ABSA inference

[!NOTE] Separately downloaded AspectBench model contributions / trained heads are CC BY-NC 4.0: attribution is required, and commercial use requires separate permission. This software toolkit uses PolyForm Noncommercial 1.0.0, with its stated institutional-use permissions. Third-party assets, datasets and earlier grants retain their own terms. See Licensing.

This directory provides one inference interface for seven AspectBench model families in HBS and Slovenian. It supports single and batched predictions, masked and unmasked aspect handling, and uncertainty estimates from Monte Carlo dropout.

• —Model collection overview
• —Reusable inference toolkit
• —GitHub repository

Licensing

This repository contains the shared PolyForm Noncommercial 1.0.0 software toolkit, not the fine-tuned weight files. Its license does not grant commercial rights to the separately downloaded AspectBench model contributions.

All seven AspectBench families' checkpoint contributions / trained heads are CC BY-NC 4.0: noncommercial research and redistribution with attribution are allowed; commercial use requires separate permission. Consult each model card and LICENSE for exact scope and upstream attribution. Third-party base models, tokenizers and code retain their own licenses; the frozen BGE-M3 encoder remains upstream MIT while the trained AspectBench MLP heads are CC BY-NC 4.0.

The fine-tuned SloBERTa checkpoints are released under CC BY-NC 4.0 with permission from the SloBERTa owners, confirmed on 17 September 2026. The original SloBERTa release retains its upstream CC BY-SA 4.0 license. See the Slavic-specific model card.

Noncommercial is about the purpose of use, not academic/company affiliation. Dataset access terms are separate. The current project-owned toolkit is source-available under the unmodified PolyForm license, not OSI-approved open source. Its permitted-purpose clauses include educational and public-research institutions regardless of funding. Third-party terms and earlier MIT grants remain unaffected. The software license does not change any separately downloaded model license.

Input contract

Every article must mark the target using literal <aspect>...</aspect> tags:

Tokom šestonedeljnog testiranja, redakcija je više puta kontaktirala
<aspect>Primer Grupu</aspect> zbog nove usluge. Prvi odgovor
<aspect>Primer Grupe</aspect> stigao je istog dana, a tehnički tim je zatim
otklonio prijavljenu grešku bez dodatnih troškova. U završnom upitniku većina
korisnika ocenila je podršku kao jasnu i pouzdanu.

Keep these tags in the input for both inference modes:

• —masked: each tagged span and its paired aspect are replaced by [ASPECT]. The checkpoint sees the target location but not its name.
• —unmasked: tags are removed during preprocessing and the checkpoint sees the target text. An optional aspect field can select a target explicitly; otherwise the first tagged span is used.

The optional gold sentiment is -1 (negative), 0 (neutral), or 1 (positive). It is echoed in the result and is never used to make a prediction.

Batch JSON can be a top-level list or {"records": [...]}:

{
  "records": [
    {
      "article": "Kupci so v šesttedenskem preizkusu uporabljali dostavo podjetja <aspect>Modri Gaj</aspect>. Večina paketov je prispela pravočasno, podpora <aspect>Modrega Gaja</aspect> pa je manjkajoči naslov dopolnila še isti dan.",
      "sentiment": 1
    }
  ]
}

Choose a model

Start with masked XLM-R. It has the best three-run test Macro-F1 among the seven released single-model families in both languages: 73.80 for Slovenian and 82.21 for HBS. Add masked Longformer when retaining more of a long article matters, and masked HAN-XLM-R when you want a structurally different hierarchical expert. The GitHub toolkit can run these three together and return individual predictions, majority voting, confidence voting, and uncertainty.

Public model repositories

Both HBS and Slovenian model weights are released publicly with project-partner approval. Model repositories contain inference artifacts only—not training records, optimizer state, cached embeddings, logs, or row-level predictions.

The following test-set scores are means over three fixed train/validation splits; ± is the standard deviation. Precision and recall are macro-averaged, and all metrics except QWK are percentages. XLM-R unmasked is the paper's Truncated strategy. XLM-R masked is Truncated + Masked: it was completed after the accepted-manuscript table was assembled and is reported in the preserved final-results analysis.

Slovenian released-model results

HBS released-model results

Getting started

For a new portable environment:

conda env create -f environment.yml
conda activate aspectbench

environment-full.yml is the complete export of the development absa environment. On the existing cluster, the current environment remains valid:

module load Anaconda3/2024.02-1
source /opt/easybuild/software/Anaconda3/2024.02-1/etc/profile.d/conda.sh
conda deactivate 2>/dev/null || true
conda activate absa
hash -r
which python
python -c 'import sys, torch; print(sys.executable); print(torch.__version__)'

The module load command can replace PATH while an older shell still has CONDA_PREFIX=.../envs/absa. In that state the prompt says (absa), uv finds packages in the environment, but python is the EasyBuild base interpreter. Deactivating and reactivating repairs the shell. The supplied validation launchers avoid this ambiguity entirely by using the environment interpreter by absolute path. Override it when necessary with:

export ABSA_PYTHON="$HOME/.conda/envs/absa/bin/python"

To use pip instead:

python -m pip install -r requirements.txt

Authenticate and download all available model repositories into the expected layout:

hf auth login
python scripts/download.py

Each model repository includes its tokenizer and architecture configuration in LANGUAGE/base_model/. Inference therefore works without a separate clone of the training repository or a populated Transformers cache.

Directory layout

huggingface/
├── models/
│   └── MODEL_NAME/
│       ├── hbs/
│       │   ├── base_model/
│       │   ├── masked.pt
│       │   └── unmasked.pt
│       ├── slovenian/
│       │   ├── base_model/
│       │   ├── masked.pt
│       │   └── unmasked.pt
│       ├── availability.json
│       └── README.md
├── examples/
├── scripts/
├── environment.yml
├── environment-full.yml
└── requirements.txt

Only checkpoint filenames whose original trained artifacts exist are present. The .pt files are model-only PyTorch state dictionaries; optimizer, scheduler, and gradient-scaler state is excluded.

Python and Jupyter usage

Download the toolkit and one family directly from a notebook if needed:

from pathlib import Path
from huggingface_hub import snapshot_download

ROOT = Path("huggingface")
snapshot_download(
    repo_id="nishan-chatterjee/aspect-based-sentiment-analysis",
    local_dir=ROOT,
)
snapshot_download(
    repo_id="nishan-chatterjee/aspectbench-mdeberta-v3",
    local_dir=ROOT / "models" / "mdeberta-v3",
)

Load once and predict many times:

from pathlib import Path
import sys

ROOT = Path("huggingface").resolve()
sys.path.insert(0, str(ROOT / "scripts"))

from inference import InferenceEngine

engine = InferenceEngine(
    model_name="mdeberta-v3",
    language="slovenian",
    mode="unmasked",
    model_root=ROOT / "models",
    device="cuda",  # or "cpu"
)

single = engine.predict(
    {
        "article": "Kupci so v šesttedenskem preizkusu uporabljali dostavo podjetja <aspect>Modri Gaj</aspect>. Večina paketov je prispela pravočasno, podpora <aspect>Modrega Gaja</aspect> pa je manjkajoči naslov dopolnila še isti dan.",
        "sentiment": 1,
    },
    mc_passes=10,
)

batch = engine.predict_batch(
    [
        {"article": "Kupci so več tednov uporabljali <aspect>Modri Gaj</aspect>. Podpora <aspect>Modrega Gaja</aspect> je vse prijave rešila pravočasno.", "sentiment": 1},
        {"article": "Pritožbe glede <aspect>Drugega sistema</aspect> niso rešene.", "sentiment": -1},
    ],
    batch_size=2,
    mc_passes=10,
)

Single prediction on an interactive SLURM node

cd huggingface

python scripts/predict.py \
  --model-name xlmr \
  --language hbs \
  --mode masked \
  --model-root models \
  --device cuda \
  --mc-passes 10 \
  --article 'Tokom šestonedeljnog testiranja redakcija je više puta kontaktirala <aspect>Primer Grupu</aspect>. Odgovor <aspect>Primer Grupe</aspect> stigao je istog dana, a prijavljena greška otklonjena je bez dodatnih troškova.' \
  --sentiment 1

--base-model-root remains available for legacy layouts, but is unnecessary when the bundled base_model/ assets are present.

Ten-record Slovenian batch

python scripts/predict_batch.py \
  --model-name mdeberta-v3 \
  --language slovenian \
  --mode unmasked \
  --input examples/sl-tagged-synthetic-examples.json \
  --output sl-predictions.json \
  --batch-size 4 \
  --model-root models \
  --device cuda \
  --mc-passes 10

The command reports the number of written predictions and the absolute output path. Use examples/hbs-tagged-examples.json for the ten HBS examples. Both files contain machine-generated paragraphs intended only for quick inference checks; they are not dataset samples and must not be used for evaluation. The HBS file contains six broadly Serbo-Croatian, three Croatian, and one Bosnian example.

Output fields

Set --mc-passes 0 for deterministic inference. A value of 1 is rejected because it is neither deterministic inference nor a meaningful MC sample.

Checkpoint availability

The intended grid is 7 repository families × 2 languages × 2 modes = 28 inference slots. All 28 trained checkpoint files are now available:

The formerly absent two XLM-R unmasked heads, two HAN-XLM-R unmasked heads, and four BGE-M3 MLP heads were retrained across three splits, selected by validation Macro-F1, packaged with their required base/tokenizer metadata, and then validated from fresh Hugging Face downloads. No masked or different-family checkpoint was relabeled as a substitute. The complete four-GPU release smoke passed 28/28 slots for model loading, single and batch prediction, probability invariants, and MC-dropout uncertainty. Exact remote revisions and reports are preserved locally under the ignored validation-runs/hf-release-smoke/ directory.

Validate single and batched inference

The validator loads every available slot, performs one-record and true batched inference, and checks the output schema and probability invariants:

bash scripts/run_validation_slurm.sh

This selects $HOME/.conda/envs/absa/bin/python when the activated shell points at the wrong interpreter. Every available language/mode slot reports both single=completed and batch=completed; missing checkpoint artifacts report SKIP, and real errors report FAIL with the failing stage. The default batch contains all ten examples for the relevant language. validation-report.json retains the complete single result and all batch result dictionaries—including class probabilities and uncertainty fields—for every tested slot.

Interactive validation on 1, 2, or 4 GPUs

After opening an interactive allocation, run the validator directly from its shell. The argument is the number of GPUs to use:

cd huggingface

# Choose exactly one of these commands.
ABSA_MC_PASSES=10 bash scripts/run_validation_interactive.sh 1
ABSA_MC_PASSES=10 bash scripts/run_validation_interactive.sh 2
ABSA_MC_PASSES=10 bash scripts/run_validation_interactive.sh 4

The launcher assigns model families across the requested GPUs; each GPU handles its assigned families sequentially so several large models are not loaded onto the same device at once. It respects an existing CUDA_VISIBLE_DEVICES from the interactive allocation. To select devices yourself, the number of comma-separated IDs must match the positional argument:

GPU_IDS=0,2 ABSA_MC_PASSES=10 bash scripts/run_validation_interactive.sh 2

To check only selected families during a quick diagnostic run:

ABSA_MODELS=xlmr,longformer ABSA_MC_PASSES=2 \
  bash scripts/run_validation_interactive.sh 2

Each available checkpoint is tested with one single prediction and one genuine ten-record batch prediction. Progress is labelled by GPU and model family. Per-family reports and logs are saved below a unique timestamped validation-runs/ directory, while their complete merged report is written to validation-report.json. The process exits nonzero if any available checkpoint fails; intentionally absent checkpoints are reported as SKIP.

If activation leaves python pointing at the EasyBuild base interpreter, the launcher automatically tries the active Conda prefix and $HOME/.conda/envs/absa/bin/python. An explicit interpreter always wins:

ABSA_PYTHON="$HOME/.conda/envs/absa/bin/python" \
  bash scripts/run_validation_interactive.sh 2

Queued SLURM validation

To submit the complete test as a one-GPU SLURM job:

bash scripts/submit_validation_slurm.sh

Logs are written to logs/aspectbench-validate-JOB_ID.{out,err} and the full machine-readable result is validation-report.json. Override defaults with, for example, PARTITION=gpu-a40 TIME_LIMIT=06:00:00 ABSA_MC_PASSES=10.

Or run it directly:

python scripts/validate_all.py \
  --model-root models \
  --examples-root examples \
  --device cuda \
  --batch-size 10 \
  --mc-passes 2 \
  --output validation-report.json

Add --require-complete-matrix when documented missing checkpoint artifacts should also cause a nonzero exit status.

Why masked and unmasked weights are separate

A masked checkpoint must remain masked at inference. Exposing target text to a model trained only with [ASPECT] creates a train/inference distribution mismatch; it does not create an unmasked model. Separate files are therefore published only for independently trained variants.

The public repositories are linked in the table near the top of this card and grouped in the AspectBench model collection. scripts/download.py restores the complete repository layout automatically.

Citation

Download citation.bib for Overleaf and cite with \cite{chatterjee2026}.

Please cite the accompanying article when using AspectBench:

@article{chatterjee2026,
  title     = {Evaluating fine-tuned, embedding-based, and zero-shot models for aspect-based sentiment analysis in {South Slavic} news},
  author    = {Chatterjee, Nishan and Koloski, Boshko and Doucet, Antoine and Pollak, Senja and Purver, Matthew},
  journal   = {Frontiers in Artificial Intelligence},
  volume    = {9},
  year      = {2026},
  publisher = {Frontiers Media SA},
  issn      = {2624-8212},
  doi       = {10.3389/frai.2026.1844418},
  url       = {https://www.frontiersin.org/journals/artificial-intelligence/articles/10.3389/frai.2026.1844418}
}