FerasMad/arabic-complaints-classifier

Arabic Restaurant Complaints Classifier

Single-label classification of Arabic restaurant complaints into 8 actionable categories. Specialized for Saudi-Gulf dialect; trained on ~98K reviews from Saudi delivery platforms (HungerStation, Jahez, Mrsool, Talabat) plus production data.

This model card covers the production 8-class model (deployed) and the v5 ambience experiment (research-stage 9-class model with a re-introduced الجو والمكان class).

Live demo: <https://huggingface.co/spaces/FerasMad/arabic-complaints-classifier> Source: <https://github.com/FerasMad/NLP-complaints-system> Companion dataset (v5 work): <https://huggingface.co/datasets/FerasMad/arabic-restaurant-ambience>

Categories

Performance

Held-out test set of 13,986 real reviews, never seen in training:

The keyword-rescue layer in the deployed Space trades 0.93% test accuracy for a 15-point gain on a 34-case behavioral audit (85% → 100%). See the GitHub repo for the rescue logic and audit set.

Intended use

Triage and routing of Arabic restaurant feedback for Saudi/Gulf operators. Use cases:

• —Customer-feedback dashboards: tag each complaint and route to the right team
• —Product analytics: aggregate complaint volume per category over time
• —Quality programs: prioritize improvement areas by complaint share

Out-of-scope / known limitations

• —Single-label only. The model picks one category. Multi-aspect complaints ("food was cold AND staff was rude") are not natively decomposed — the deployed Space adds a heuristic display layer for these. A true multi-label retrain is the next planned improvement.
• —Dialect bias. Trained almost entirely on Saudi/Gulf dialect. Cross-dialect canary scores (out of ~50 single-aspect probes per dialect):
• —Saudi: ~67%
• —Levantine: ~60%
• —Egyptian / MSA: ~50%
• —For non-Gulf dialects, treat predictions as advisory.
• —Domain bound to restaurants. The model was trained on restaurant complaints only. Don't apply to other product categories without retraining.
• —No ambience category in production model. v3 had a "الجو والمكان" class; v4 dropped it after a manual audit found ~99% of the gold labels were noise. The production 8-class model does not predict ambience. A 9-class v5 experiment exists that re-introduces ambience — see "V5 ambience experiment" section below. The v5 model achieves 89.22% ambience F1 on a hand-crafted adversarial fixture but is research-stage, not deployed.
• —No abstain mechanism in the raw model. The deployed Space adds short-input abstain (length < 3 chars) and reframes "عامة" predictions as "no specific aspect detected." If you call the model directly, you won't get those guards.
• —PII handling is the caller's responsibility. The model has no built-in PII scrubbing.

How to use

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

REPO = "FerasMad/arabic-complaints-classifier"
LABELS = [
    "التوصيل", "السعر والقيمة", "النظافة", "جودة الطعام",
    "خدمة الموظفين", "دقة الطلب", "عامة", "وقت الانتظار",
]

tok = AutoTokenizer.from_pretrained(REPO)
model = AutoModelForSequenceClassification.from_pretrained(REPO).eval()

text = "الاكل بايخ ومالح والطبخ مو متقن"
inputs = tok(text, return_tensors="pt", truncation=True, max_length=192)
with torch.no_grad():
    probs = model(**inputs).logits.softmax(-1)[0]
top_idx = int(probs.argmax())
print(f"{LABELS[top_idx]}: {probs[top_idx]:.2%}")
# جودة الطعام: 98.4%

For temperature-scaled probabilities, divide logits by T = 1.523 before softmax. For the keyword-rescue layer and aspect-extraction interpretability, use the deployed Space or copy hf_space/app.py from the GitHub repo.

Training details

• —Base model: CAMeL-Lab/bert-base-arabic-camelbert-mix
• —Architecture: BERT base (110M params) + classification head over 8 labels
• —Sequence length: 192 tokens
• —Training data: ~98K Arabic complaints from Saudi delivery platforms + production data, manually labeled by the AI Club NLP team
• —Augmentation: EDA (Easy Data Augmentation) for under-represented classes (دقة الطلب, عامة) only
• —Calibration: Temperature scaling with T=1.523, fit on the validation set by NLL minimization

The deployed system uses an ensemble of 4 BERT variants (CAMeLBERT-mix × 2 seeds, MARBERT, AraBERTv02) — see the GitHub repo. This single model is the lightest deployable variant and is what powers the public HF Space (free-tier memory constraint).

Evaluation methodology

• —Held-out test set of 13,986 reviews, sampled per-source and stratified per-class
• —Bootstrap 95% CI computed over 1000 resamples
• —Calibration assessed via ECE on the test set
• —Cross-dialect canary set written by hand to probe dialect generalization
• —Behavioral audit: 34 hand-written test cases (single + multi-aspect) covering all 8 categories — this is what the deployed rescue layer optimizes for

V5 ambience experiment (9-class, research-stage)

After the 8-class production model shipped, the v3-dropped ambience class was re-attempted from scratch with new infrastructure: PySarf morphology, weak-labeled real data from public HF datasets, MLM continued pre-training on 45K Arabic restaurant + hotel reviews, focal loss, and threshold-tuned inference. Three models exist (all research-stage, not deployed):

All three apply a post-hoc decision threshold (ambience_threshold ∈ [0.005, 0.02]) and an abstain wrapper (length gate + restaurant-domain OOD gate) at inference.

V5 evaluation methodology

• —Adversarial fixture: 183 hand-written cases across 13 attack types (clean, boundary, negation, sarcasm, mixed dialect, multi-aspect, short, veryshort, long, outof_domain, typo, emoji, adversarial). Released as part of the companion dataset.
• —Real-world val/test: 15% / 15% stratified splits of the combined baseline (95K rows) + 5,000 weak-labeled HARD ambience candidates.

V5 ambience class metrics (Run 4, threshold 0.01 + abstain)

V5 known limitations

• —Multi-aspect cases stuck at 40% — single-label classification can't decompose "الاكل بارد والمكيف خربان". Multi-label retraining is the only structural fix.
• —Out-of-domain handling at 50% — abstain wrapper catches half. A dedicated OOD classifier as a preprocessing gate would close more.
• —Trained on weak-labeled real data, not gold-labeled. The 1,256 qaym + HARD ambience candidates have ~50-80% precision per sampling. Hand-labeling would push F1 higher.

Reproducing v5

Full waves 7-10 documented at:

• —docs/V5_TRAINING_RUN_1_RESULTS.md — initial baseline (78.57% F1)
• —docs/V5_TRAINING_RUN_2_THRESHOLD_TUNING.md — threshold to 85.56%
• —docs/V5_TRAINING_RUN_3_REAL_DATA_AND_ABSTAIN.md — 24K real candidates harvested
• —docs/V5_TRAINING_RUN_4_PRETRAINED.md — MLM + weak labels = 89.22%
• —docs/V5_TRAINING_RUN_5_HARD_DATA.md — 5x data, mixed signal

Credits

Built by the NLP team at AI Club:

• —Feras Madkhali — team lead, training (Phase 3), evaluation, deployment
• —Lana — text cleaning pipeline (clean())
• —Khowla — labeling and split decisions
• —Rima & Mohammed — schema design, baseline trials
• —Meshal — deployment side

Special thanks to Rashidbm for PySarf — the Arabic morphology engine used in the deployed Space's aspect-extraction layer.

License

MIT. Upstream model licenses (CAMeLBERT, MARBERT, AraBERTv02) apply to the base architecture — see the GitHub repo's NOTICES file.

Citation

@misc{madkhali2026arcomplaints,
  title  = {Arabic Restaurant Complaints Classifier},
  author = {Madkhali, Feras and the AI Club NLP Team},
  year   = {2026},
  url    = {https://github.com/FerasMad/NLP-complaints-system},
  note   = {Saudi-Gulf dialect specialization. 8-class production model
            + 9-class v5 ambience experiment (89.22% adversarial F1).
            Companion dataset at huggingface.co/datasets/FerasMad/arabic-restaurant-ambience}
}