KoalaAI/Text-Moderation-Multilingual

1---2license: apache-2.03task_categories:4- text-classification5tags:6- text-moderation7language:8  - en9  - de10  - fr11  - es12  - it13  - sv14  - fi15  - pl16  - cs17  - lv18  - zh19  - ja20  - ko21  - ru22  - uk23  - be24  - kk25---26 27# Text-Moderation-Multilingual28 29A comprehensive multilingual text moderation dataset combining multiple high-quality sources for training robust content moderation classifiers.30 31## Dataset Summary32 33This dataset aggregates text moderation data from multiple sources to create a large-scale, diverse training corpus for content moderation systems. It includes text samples labeled across multiple harmful content categories, supporting both multilingual and English-specific moderation use cases.34 35**Total Size:** ~1.7M entries  36**Languages:** Multilingual (primary focus on English)  37**Task:** Multi-label text classification for content moderation38 39## Dataset Structure40 41### Data Fields42 43- `prompt` (string): The input text to be classified44- `S` (int): Sexual content (0 = safe, 1 = harmful)45- `H` (int): Hate speech (0 = safe, 1 = harmful)  46- `V` (int): Violence (0 = safe, 1 = harmful)47- `HR` (int): Harassment (0 = safe, 1 = harmful)48- `SH` (int): Self-harm (0 = safe, 1 = harmful)49- `S3` (int): Sexual content involving minors (0 = safe, 1 = harmful)50- `H2` (int): Hate speech (alternative labeling) (0 = safe, 1 = harmful)51- `V2` (int): Violence (alternative labeling) (0 = safe, 1 = harmful)52 53### Data Splits54 55- **Train:** 1459350 samples56- **Validation:** 162150 samples57 58*Note: Split created with 90/10 train/validation ratio using random seed 42*59 60## Source Datasets61 62This dataset combines and harmonizes data from:63 64- **[ifmain's multilingual dataset](https://huggingface.co/datasets/ifmain/text-moderation-02-multilingual)** - Multilingual moderation examples65- **[OpenAI's English evaluation dataset](https://huggingface.co/datasets/mmathys/openai-moderation-api-evaluation)** - High-quality English evaluation samples  66- **[ifmain's English dataset](https://huggingface.co/datasets/ifmain/text-moderation-01)** - English moderation examples67 68## Intended Use69 70### Primary Use Cases71- Training text moderation classifiers72- Benchmarking content moderation systems73- Research into automated content moderation74- Multi-label classification model development75 76### Out-of-Scope Uses77- This dataset is **not intended** for any purpose other than training content moderation systems78- Should not be used to generate harmful content79- Not suitable for general text classification tasks outside of moderation80 81## Considerations for Using the Data82 83### Content Warning84This dataset contains examples of harmful content including hate speech, harassment, violence, and other potentially disturbing material. Users should exercise appropriate caution when working with this data.85 86### Bias and Limitations87- The dataset reflects the biases present in the source datasets88- Content moderation standards may vary across different platforms and cultures89- Label consistency across merged datasets may vary90- Primarily English-focused despite multilingual components91 92### Ethical Considerations93- This dataset should only be used to improve content moderation and safety systems94- Researchers and developers should implement appropriate safeguards when working with this data95- The goal is to reduce harmful content online, not to amplify it96 97## Example Usage98 99```python100from datasets import load_dataset101 102# Load the dataset103dataset = load_dataset("KoalaAI/Text-Moderation-Multilingual")104 105# Access splits106train_data = dataset["train"]107val_data = dataset["validation"]108 109# Example entry110print(train_data[0])111# {112#   'prompt': 'Example text...',113#   'S': 0, 'H': 0, 'V': 0, 'HR': 0, 114#   'SH': 0, 'S3': 0, 'H2': 0, 'V2': 0115# }116```117 118## Dataset Creation119 120### Curation Process1211. Source datasets were identified and downloaded1222. Data was harmonized to use consistent labeling schema1233. Entries were merged and deduplicated where appropriate1244. Train/validation split was created using stratified sampling125 126### Quality Control127- Labels were preserved from original high-quality sources128- Data integrity checks were performed during merging process129- Consistent schema applied across all entries130 131## License132 133Please refer to the licenses of the individual source datasets:134- Check ifmain datasets for their respective licensing terms135- OpenAI evaluation dataset licensing applies to that portion136- Usage should comply with all source dataset requirements137 138## Citation139 140If you use this dataset, please cite the original source datasets:141 142```bibtex143@misc{text-moderation-large,144  title={Text-Moderation-Multilingual: A Multilingual Text Moderation Dataset},145  author={[KoalaAI]},146  year={2025},147  note={Aggregated from ifmain's and OpenAI's moderation datasets}148}149```150 151## Contact152 153For questions about this dataset compilation, please open an issue on this repository.154 155---156 157**Disclaimer:** This dataset is provided for research and safety purposes only. Users are responsible for ensuring ethical use and compliance with applicable laws and regulations.