FatimahEmadEldin/Isnad-AI-Identifying-Islamic-Citation

Isnad AI: AraBERT for Ayah & Hadith Span Detection in LLM Outputs

<p align="center"> <img src="https://placehold.co/800x200/dbeafe/3b82f6?text=Isnad+AI+-+Islamic+Citation+Detection" alt="Isnad AI - Islamic Citation Detection"> </p>

This repository contains the official fine-tuned model for the Isnad AI system, the submission to the [IslamicEval 2025 Shared Task 1A](https://sites.google.com/view/islamiceval-2025). The model is designed to identify character-level spans of Quranic verses (Ayahs) and Prophetic sayings (Hadiths) within text generated by Large Language Models (LLMs).

By: Fatimah Emad Eldin

Cairo University

   

📜 Model Description

This model fine-tunes AraBERTv2 (aubmindlab/bert-base-arabertv2) on a specialized token classification task. Its purpose is to label tokens within a given Arabic text according to the BIO schema:

• —B-Ayah (Beginning of a Quranic verse)
• —I-Ayah (Inside a Quranic verse)
• —B-Hadith (Beginning of a Prophetic saying)
• —I-Hadith (Inside a Prophetic saying)
• —O (Outside of any religious citation)

The key innovation behind this model is a novel rule-based data generation pipeline. This method was developed to address the critical lack of large, manually annotated datasets for this specific task. By programmatically creating a large-scale, high-quality training corpus from authentic religious texts, we completely eliminated the need for manual annotation. This approach proved highly effective, enabling the model to learn the contextual patterns of how LLMs cite Islamic sources, achieving an F1-score of 66.97% on the official blind test set.

🚀 How to Use

You can easily use this model with the transformers library pipeline for token-classification (or ner). For best results, use aggregation_strategy="simple" to group token pieces into coherent entities.

from transformers import pipeline

# Load the token classification pipeline
model_id = "FatimahEmadEldin/Isnad-AI-Identifying-Islamic-Citation"
islamic_ner = pipeline(
    "token-classification",
    model=model_id,
    aggregation_strategy="simple"
)

# Example text from an LLM response
text = "يوضح لنا الدين أهمية الصدق، ففي الحديث الشريف نجد أن النبي قال: عليكم بالصدق. كما أنزل الله في كتابه الكريم: يا أيها الذين آمنوا اتقوا الله وكونوا مع الصادقين."

# Get the identified spans
results = islamic_ner(text)

# Print the results
for entity in results:
    print(f"Entity: {entity['word']}")
    print(f"Label: {entity['entity_group']}")
    print(f"Score: {entity['score']:.4f}\n")

# Expected output:
# Entity: عليكم بالصدق
# Label: Hadith
# Score: 0.9876

# Entity: يا أيها الذين آمنوا اتقوا الله وكونوا مع الصادقين
# Label: Ayah
# Score: 0.9912

⚙️ Dataset Curation and Statistics

The model was trained exclusively on a synthetically generated dataset. The motivation was to overcome the data scarcity problem for this specialized task by creating realistic, contextualized examples that mimic how LLMs cite religious texts. The entire training and validation dataset was generated using a multi-stage pipeline.

Data Sources and Preprocessing

1. 1.Data Sourcing: Authentic texts were sourced from quran.json (containing all 6,236 Quranic verses) and a JSON file of the Six Major Hadith Collections (containing over 34,000 narrations).

1. 1.Text Splitting: To prevent sequence truncation during tokenization, any Quranic verse exceeding a 25-token length was split into two smaller, more manageable segments. This crucial step increased the number of unique Ayah texts from 6,236 to 6,910.

1. 1.Normalization and Augmentation (Tashkeel Removal): To improve the model's robustness against script variations, a duplicate version of every Ayah was created with all Arabic diacritics (Tashkeel) removed. This data augmentation step effectively doubled the number of unique Ayah texts to 13,820, training the model to recognize verses regardless of vocalization.

1. 1.Template-Based Generation: The core of the pipeline involves embedding these processed religious texts into contextual templates. Each unique text was used to generate multiple training examples by randomly combining it with different prefixes, suffixes, and neutral connecting sentences (detailed below). This simulates how these texts are typically cited in generated content.

Dataset Splits and Distribution

From the augmented collection of 45,137 unique religious texts, a 70/30 split was employed to create the training and validation sets. The template-based generation process was then applied to both sets to create the final corpus.

Template Component Examples

The following table details the components used to programmatically construct the training data.

Fine-Tuning

The aubmindlab/bert-base-arabertv2 model was fine-tuned with the following key hyperparameters:

• —Learning Rate: 2e-5
• —Epochs: 10 (with early stopping patience of 3)
• —Effective Batch Size: 16 (4 per device with 4 gradient accumulation steps)
• —Optimizer: AdamW
• —Weight Decay: 0.01
• —Warmup Steps: 500
• —Mixed Precision: fp16 enabled
• —Max Sequence Length: 512 tokens

📊 Evaluation Results

The model was evaluated using the official character-level Macro F1-Score metric for the IslamicEval 2025 shared task.

Official Test Set Results

The system achieved a final F1-score of 66.97% on the blind test set, demonstrating the effectiveness of the rule-based data generation approach.

Development Set Performance

A detailed evaluation on the manually annotated development set provided by the organizers shows a strong and balanced performance.

Final Macro F1-Score on Dev Set: 65.08%

Per-Class Performance (Character-Level)

(These results are from the official `scoring.py` script run on the development set).

⚠️ Limitations and Bias

• —Performance on Hadith: The model's primary challenge is identifying Hadith texts, which have significantly more linguistic and structural variety than Quranic verses. The F1-score for the Hadith class is lower than for Ayah, indicating it may miss or misclassify some prophetic sayings.
• —Template Dependency: The model's knowledge is based on the rule-based templates used for training. It may be less effective at identifying citations that appear in highly novel or unconventional contexts not represented in the training data.
• —Scope: This model identifies intended citations, as per the shared task rules. It does not verify the authenticity or correctness of the citation itself. An LLM could generate a completely fabricated verse, and this model would still identify it if it is presented like a real one.

✍️ Citation

If you use this model or the methodology in your research, please cite the paper:

  Coming soon