Datasetpublic

QCRI/LlamaLens-Hindi

LlamaLens: Specialized Multilingual LLM Dataset Overview LlamaLens is a specialized multilingual LLM designed for analyzing news and social media content. It focuses on 18 NLP tasks, leveraging 52 datasets across Arabic, English, and Hindi. LlamaLens This repo includes scripts needed to run our full pipeline, including data preprocessing and sampling, instruction dataset creation, model fine-tuning, inference and evaluation. Features… See the full description on the dataset page: https://huggingface.co/datasets/QCRI/LlamaLens-Hindi.

sourceHugging Facecc-by-nc-sa-4.0updated 2y agoView on Hugging Face

1likes117downloads

Dataset Card

LlamaLens: Specialized Multilingual LLM Dataset

Overview

LlamaLens is a specialized multilingual LLM designed for analyzing news and social media content. It focuses on 18 NLP tasks, leveraging 52 datasets across Arabic, English, and Hindi.

LlamaLens

This repo includes scripts needed to run our full pipeline, including data preprocessing and sampling, instruction dataset creation, model fine-tuning, inference and evaluation.

Features

—Multilingual support (Arabic, English, Hindi)
—18 NLP tasks with 52 datasets
—Optimized for news and social media content analysis

📂 Dataset Overview

Hindi Datasets

Task	Dataset	# Labels	# Train	# Test	# Dev
Cyberbullying	MC-Hinglish1.0	7	7,400	1,000	2,119
Factuality	fake-news	2	8,393	2,743	1,417
Hate Speech	hate-speech-detection	2	3,327	951	476
Hate Speech	Hindi-Hostility-Detection-CONSTRAINT-2021	15	5,718	1,651	811
NaturalLanguageInference	NaturalLanguageInference	2	1,251	447	537
Summarization	xlsum	--	70,754	8,847	8,847
Offensive Speech	OffensiveSpeechDetection	3	2,172	636	318
Sentiment	Sentiment_Analysis	3	10,039	1,259	1,258

Results

Below, we present the performance of L-Lens: LlamaLens , where "Eng" refers to the English-instructed model and "Native" refers to the model trained with native language instructions. The results are compared against the SOTA (where available) and the Base: Llama-Instruct 3.1 baseline. The Δ (Delta) column indicates the difference between LlamaLens and the SOTA performance, calculated as (LlamaLens – SOTA).

Task	Dataset	Metric	SOTA	Base	L-Lens-Eng	L-Lens-Native	Δ (L-Lens (Eng) - SOTA)
Factuality	fake-news	Mi-F1	--	0.759	0.994	0.993	--
Hate Speech Detection	hate-speech-detection	Mi-F1	0.639	0.750	0.963	0.963	0.324
Hate Speech Detection	Hindi-Hostility-Detection-CONSTRAINT-2021	W-F1	0.841	0.469	0.753	0.753	-0.088
Natural Language Inference	Natural Language Inference	W-F1	0.646	0.633	0.568	0.679	-0.078
News Summarization	xlsum	R-2	0.136	0.078	0.171	0.170	0.035
Offensive Language Detection	Offensive Speech Detection	Mi-F1	0.723	0.621	0.862	0.865	0.139
Cyberbullying Detection	MC_Hinglish1	Acc	0.609	0.233	0.625	0.627	0.016
Sentiment Classification	Sentiment Analysis	Acc	0.697	0.552	0.647	0.654	-0.050

File Format

Each JSONL file in the dataset follows a structured format with the following fields:

—id: Unique identifier for each data entry.
—original_id: Identifier from the original dataset, if available.
—input: The original text that needs to be analyzed.
—output: The label assigned to the text after analysis.
—dataset: Name of the dataset the entry belongs.
—task: The specific task type.
—lang: The language of the input text.
—instructions: A brief set of instructions describing how the text should be labeled.

Example entry in JSONL file:

{
    "id": "5486ee85-4a70-4b33-8711-fb2a0b6d81e1",
    "original_id": null,
    "input": "आप और बाकी सभी मुसलमान समाज के लिए आशीर्वाद हैं.",
    "output": "not-hateful",
    "dataset": "hate-speech-detection",
    "task": "Factuality",
    "lang": "hi",
    "instructions": "Classify the given text as either 'not-hateful' or 'hateful'. Return only the label without any explanation, justification, or additional text."
}

Model

**LlamaLens on Hugging Face**

Replication Scripts

**LlamaLens GitHub Repository**

📢 Citation

If you use this dataset, please cite our paper:

@article{kmainasi2024llamalensspecializedmultilingualllm,
  title={LlamaLens: Specialized Multilingual LLM for Analyzing News and Social Media Content},
  author={Mohamed Bayan Kmainasi and Ali Ezzat Shahroor and Maram Hasanain and Sahinur Rahman Laskar and Naeemul Hassan and Firoj Alam},
  year={2024},
  journal={arXiv preprint arXiv:2410.15308},
  volume={},
  number={},
  pages={},
  url={https://arxiv.org/abs/2410.15308},
  eprint={2410.15308},
  archivePrefix={arXiv},
  primaryClass={cs.CL}
}