CoolFace
Modelpublic

imaneumabderahmane/Arabertv02-classifier-FA

sourceHugging Faceapache-2.0updated 6mo agoView on Hugging Face
1likes18downloads
Model Card

Model Card for Model ID

<!-- Provide a quick summary of what the model is/does. -->

FA-AraBert is an Arabic binary text classification model designed to detect whether a user query is related to first-aid. The classifier serves as the intent detection and safety filtering component of an MSA first-aid chatbot pipeline. Two models are developed and evaluated in this project: FA-AraBERTv2 and FA-AraBERTv0.2, both fine-tuned on the FALAH-Mix dataset (1,028 question–answer (QA) pairs, including 924 non first-aid pairs and 104 first-aid pairs) based on AraBERT base models. These classifiers were systematically compared under multiple training configurations to identify the most suitable model for deployment.

Model Details

Model Description

<!-- Provide a longer summary of what this model is. -->

  • —Model name: FA-AraBERTv0.2
  • —Developed by: MABROUK Imane
  • —Supervised by: Dr. Rana R. Malhas (bigIR Research Group, Qatar University) & Dr. Imane Chlioui (INSEA)
  • —Task: Binary text classification
  • —Domain: First Aid / Emergency Care
  • —Labels: <p align="center"> <img src="./labels.png" width="300"/> </p>
  • —Funded by: Self-funded academic project (Academic graduation project)

The FA-AraBERT classifier was developed as part of the PFE project titled:

  • —Towards Building an Arabic First-Aid Chatbot using FA-AraBERT Classifier and FALAH Dataset.
  • —Shared by: MABROUK Imane
  • —Model type: Transformer-based text classification model (BERT architecture)
  • —Language(s) (NLP): MSA(Modern Standard Arabic)
  • —License: Apache 2.0
  • —Finetuned from model: AraBERTv02

Model Sources

<!-- Provide the basic links for the model. -->

  • —Repository: https://github.com/Imymab/FA-AraBERT-Classifier

Uses

<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->

Direct Use

<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->

This model can be used directly for:

  • —First-aid and emergency query detection
  • —Binary text classification (First-aid/Non First-aid)

Downstream Use

<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->

The model can be integrated into:

  • —Medical conversational agents
  • —Healthcare assistance tools for emergency guidance

Out-of-Scope Use

<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->

This model is not suitable for:

  • —Use in high-stakes clinical decision-making without human supervision
  • —Non-Arabic text processing
  • —Tasks requiring deep medical reasoning or long-context understanding

Bias, Risks, and Limitations

<!-- This section is meant to convey both technical and sociotechnical limitations. -->

  • —The model is trained on a limited dataset (FALAH-Mix), which may not fully represent all medical domains.
  • —Class imbalance in the dataset may affect performance on underrepresented categories.
  • —The model may reflect biases present in the existing arabic medical datasets (AHD, MAQA).
  • —It is not a substitute for professional medical advice.

Recommendations

<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->

  • —Try to explore additional techniques to improve performance, particularly by addressing the class imbalance in the FALAH-Mix dataset (10% first-aid, 90% non–first-aid).

How to Get Started with the Model

Use the code below to get started with the model:

python
from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

model_name = "imaneumabderahmane/Arabertv02-classifier-FA"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)

text = "ما هي الإسعافات الأولية لحروق الدرجة الأولى؟"
inputs = tokenizer(text, return_tensors="pt", truncation=True)

with torch.no_grad():
    outputs = model(**inputs)
    prediction = torch.argmax(outputs.logits, dim=-1).item()

print(prediction)

Training Details

Training Data

<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. --> The FA-AraBERTv02 classifier was trained and evaluated on the FALAH-Mix dataset, which contains 1,028 Arabic question–answer pairs (924 non–first-aid QA pairs and 104 first-aid QA pairs). The dataset exhibits a strong class imbalance, with approximately 90% non–first-aid queries and 10% first-aid queries. The data was split into training, development, and test sets while preserving the original class distribution.

For more details about the FALAH and FALAH-Mix datasets, developed as part of the PFE project, please refer to: https://huggingface.co/datasets/imaneumabderahmane/FALAH

To mitigate class imbalance, the training set was augmented with additional first-aid samples from external datasets, including the Mayo Clinic First-Aid dataset (374 first-aid QA pairs) and the 68 first-aid QA pairs from the AHD dataset. This resulted in a balanced training set of 1,184 samples, while the development and test sets remained unchanged.

The following table presents the FALAH-Mix dataset before balancing the training set:

<p align="center"> <img src="./distribution of the FALAH-Mix dataset.png" width="900"/> </p>

The following table presents the FALAH-Mix dataset after balancing the training set:

<p align="center"> <img src="./table FALAH-Mix training set balanced.png" width="700"/> </p>

Note: The FALAH-Mix dataset was split according to the emergency labels. As a result, the training, development, and test sets each contain approximately 90% non–first-aid QA pairs and 10% first-aid QA pairs. The balancing strategy was applied only to the training set in order to evaluate its impact on the classifier.

Training Procedure

<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->

Preprocessing
  • —Arabic text normalization
  • —Translating questions from Arabic Dialect into Modern Standard Arabic using this model: https://huggingface.co/HamzaNaser/Dialects-to-MSA-Transformer
  • —Tokenization using AraBERT tokenizer
Training Hyperparameters

The models were fine-tuned using supervised learning with the following configuration:

  • —Optimizer: AdamW
  • —Learning rate: 3 × 10⁻⁵
  • —Batch size: 16
  • —Epochs: 3
  • —Loss function: Cross-entropy
  • —Mixed-precision training enabled <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
Speeds, Sizes, Times

Evaluation

<!-- This section describes the evaluation protocols and provides the results. -->

Testing Data, Factors & Metrics

Testing Data

<!-- This should link to a Dataset Card if possible. -->

The test split from the FALAH-Mix dataset: https://huggingface.co/datasets/imaneumabderahmane/FALAH

Metrics

<!-- These are the evaluation metrics being used, ideally with a description of why. -->

  • —Macro F1-score, chosen to account for class imbalance

Results

The following Table summarizes the Macro F1 scores obtained by FA-AraBERTv2 and FA-AraBERTv0.2 under different training configurations. <p align="center"> <img src="./table results classifier arabert.png" width="800"/> </p> The best performance was achieved when fine-tuning on the balanced FALAH-Mix training set with class weighting. FA-AraBERTv2 achieved a Macro F1-score of 0.6379, slightly outperforming FA-AraBERTv0.2. Due to this consistent advantage, FA-AraBERTv2 was selected for deployment in the final chatbot system. It should be noted that statistical significance testing was not performed due to time constraints related to the project.