nsr51324/DeepX-AI-Hackathon-ABSA

DeepX-AI-Hackathon-ABSA

A multi-dialect, multilingual Aspect-Based Sentiment Analysis (ABSA) system for Arabic reviews (Modern Standard Arabic, dialectal Arabic, Franco-Arabic, English, and French).

An ensemble of 4 independently fine-tuned transformer backbones, combined with a language-aware routing system, purpose-built for real-world user reviews (e.g. Google Maps, Play Store, food delivery apps) written in mixed scripts and dialects — including Egyptian, Gulf, and Levantine Arabic, and Franco-Arabic (Arabic written with Latin letters and digits).

📌 Overview

This project is a complete two-stage pipeline that extracts, for every review:

1. 1.Aspects mentioned in the review (e.g. food, service, price, cleanliness…).
2. 2.Sentiment associated with each individual aspect (positive / negative / neutral).

The final output for each review is a set of (aspect, sentiment) tuples:

{
  "review_id": 12345,
  "aspects": [
    {"aspect": "food", "sentiment": "positive"},
    {"aspect": "service", "sentiment": "negative"}
  ]
}

Why an ensemble of 4 models?

Real-world Arabic reviews are far from uniform — the same user might write in Modern Standard Arabic, a regional dialect, Franco-Arabic (Latin letters + digits standing in for Arabic sounds), or plain English/French. To handle this diversity, four different backbones were fine-tuned, each with a different strength, and their outputs are combined via a language-routed, weighted soft-voting ensemble.

🧠 Architecture

Stage 1 — Aspect Detection (Multi-Label Classification)

For every review text, the model predicts which of 9 aspect categories are mentioned:

Model architecture:

Review text → Backbone (BERT / RoBERTa) → [CLS] embedding
                                                ↓
                Metadata Fusion (concatenated with text embedding):
                - Star rating       → Embedding
                - Business category → Embedding
                - Platform (Google Maps / Play Store) → Embedding
                                                ↓
                        MLP + LayerNorm + Dropout
                                                ↓
                Linear layer → 9 logits (multi-label)
                                                ↓
                    Sigmoid + per-class decision threshold

Stage 2 — Sentiment Classification (per aspect)

Once the aspects are extracted, for every (review, aspect) pair an aspect-aware question is built:

• —Arabic backbones: [نجوم=X] <text> [SEP] ما رايك فى <aspect> ؟
• —Latin-script backbone (XLM-R): [stars=X] <text> [SEP] what about the <aspect> ?

The pair is then classified into one of 3 classes: positive / negative / neutral.

(Text + aspect question) → Backbone → [CLS]
                                          ↓
                Metadata Fusion + Aspect Embedding
                                          ↓
                    MLP (128) → 3 logits
                                          ↓
                                    Softmax

🔀 Language Router

Before any modeling, every text is automatically routed into one of the following categories using regex heuristics and marker-word dictionaries:

Important fix (V2): Very short texts (1–2 words) without a clear sentiment word (e.g. "Up" or "Shady") are not routed to Franco — they're routed to empty_rating_only instead, since model predictions on such short strings are less reliable than a simple star-based rule.

Which backbones handle which route?

ROUTE_BACKBONES = {
    'arabic':  ['marbert', 'arabert', 'camelbert', 'xlmr'],
    'mixed':   ['marbert', 'camelbert', 'xlmr'],
    'franco':  ['marbert', 'arabert', 'camelbert', 'xlmr'],
    'latin':   ['xlmr'],
    'other_script': ['xlmr'],
    'empty_rating_only': [],   # handled entirely by a hand-written star-based rule
}

🏗️ Backbones

Training data notes:

• —MARBERT and XLM-R were trained on an expanded dataset: real Arabic reviews + synthetically generated Franco-Arabic variants (see below).
• —AraBERT and CAMeLBERT were trained on real Arabic data only.

🎲 Synthetic Franco-Arabic Data Augmentation

Since real Franco-Arabic examples are scarce in the training set, an automatic Arabic → Franco-Arabic converter was built, based on:

1. 1.A common-word dictionary (60+ frequent Arabic words/phrases mapped to their typical Franco-Arabic spellings, e.g. ممتاز → mumtaz/momtaz, مش → mesh/mish).
2. 2.A character-level transliteration map for the remaining words (e.g. ح → 7, ع → 3, غ → 8).
3. 3.Controlled randomness (75% chance of using the common-word dictionary) to mimic natural spelling variation.

This expanded the MARBERT/XLM-R training set from 1,971 to 3,809 samples.

⚖️ Ensemble & Weight Tuning

After training the four backbones, a weighted soft-voting ensemble combines their probability outputs, weighted per model, and thresholded per aspect class.

Weights (from a grid search over 625 combinations on the validation set):

The fixed weights actually used for final test-set inference were: {'marbert': 1.2, 'arabert': 1.2, 'camelbert': 1.0, 'xlmr': 0.7} — see `ensemble_weights.json`.

Per-aspect decision thresholds

Saved in `thresholds.npy`:

📏 Post-Processing Rules

1. 1.Empty / rating-only reviews (empty_rating_only) skip the models entirely and are classified directly from the star rating:
2. 2.⭐ ≥ 4 → general: positive
3. 3.⭐ ≤ 2 → general: negative
4. 4.⭐ = 3 → none: neutral
5. 5.Max 6 aspects per review (kept by highest predicted probability).
6. 6.If none co-occurs with other aspects, none is dropped (a specific aspect and "no aspect" together are contradictory).
7. 7.If no aspect crosses its threshold, none: neutral is used as a default.
8. 8.If a sentiment prediction is unavailable for a given aspect, the star rating is used as a fallback.

📊 Results (Validation Set — 1,971 reviews)

Main metric: Tuple F1 (aspect + sentiment must both match)

Performance by route

Performance by aspect

Sentiment confusion matrix (when the aspect is correctly detected)

📁 Repository Contents

DeepX-AI-Hackathon-ABSA/
├── models/                         # Weights for all 4 backbones (Stage 1 + Stage 2)
├── ensemble_weights.json           # Final ensemble weights
├── thresholds.npy                  # Per-aspect decision thresholds (9 values)
├── submission.json                 # Predictions on the unlabeled set
├── submission_test.json            # Predictions on the hidden test set
├── __huggingface_repos__.json      # Repository metadata
└── README.md                       # This file

🚀 Usage

⚠️ This is not a single model loadable with AutoModel and a standard pipeline(). It is an ensemble of 4 backbones plus custom language-routing, preprocessing, and post-processing logic. Running inference requires the full inference code (from the original training notebook), not just the saved weights.

Inference outline:

from transformers import AutoTokenizer, AutoModel
import torch, json, numpy as np

BACKBONES = {
    'marbert':   'UBC-NLP/MARBERTv2',
    'arabert':   'aubmindlab/bert-base-arabertv02-twitter',
    'camelbert': 'CAMeL-Lab/bert-base-arabic-camelbert-da',
    'xlmr':      'FacebookAI/xlm-roberta-base',
}

# 1. Load the Stage-1 (aspect) and Stage-2 (sentiment) checkpoints from models/
# 2. Route each text via detect_language()
# 3. Preprocess it according to its route via preprocess_by_route()
# 4. Run the backbones listed in ROUTE_BACKBONES[route]
# 5. Combine outputs with the weights in ensemble_weights.json
# 6. Apply the per-aspect thresholds in thresholds.npy
# 7. Apply the post-processing rules to build the final prediction

For the complete code (model definitions, helper functions, preprocessing, and ensembling logic), see the original training notebook shipped alongside this project.

🎯 Intended Use

• —Multi-dialect customer review analysis (Google Maps / Play Store / food-delivery platforms).
• —Extracting per-aspect strengths and weaknesses (food, service, price, cleanliness, …) for business owners.
• —Sentiment dashboards for restaurants, hotels, delivery apps, clinics, and e-commerce.

⚠️ Limitations

• —Performance on very short / empty reviews (empty_rating_only) is comparatively weaker (F1 = 0.61) since it relies purely on a star-rating rule rather than the model.
• —The none aspect has weaker performance (F1 = 0.68), reflecting the difficulty of distinguishing "no clear aspect" from a generic "general" comment.
• —Trained on only 1,971 labeled reviews — a relatively small dataset, which may limit generalization to domains not well represented in training (e.g. medical or real-estate reviews).
• —Franco-Arabic training examples are synthetically generated rather than fully authentic, which may reduce accuracy on unusual real-world Franco-Arabic spelling patterns.

🏆 Context

This model was developed as part of the DeepX AI Hackathon, addressing an Aspect-Based Sentiment Analysis (ABSA) task on multilingual, multi-dialect Arabic reviews.

📄 License

MIT

For questions about this model, please open a Discussion on the Hugging Face repository page.