rayyanm86/hateful-memes

The Hateful Memes Challenge README

The Hateful Memes Challenge is a dataset and benchmark created by Facebook AI to drive and measure progress on multimodal reasoning and understanding. The task focuses on detecting hate speech in multimodal memes.

Please see the paper for further details:

The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes D. Kiela, H. Firooz, A. Mohan, V. Goswami, A. Singh, P. Ringshia, D. Testuggine

Dataset details

The files for this folder are arranged as follows:

img/ - the PNG images train.jsonl - the training set dev.jsonl - the development set test.jsonl - the “seen” test set

An additional “unseen” test set will be released at a later date under the NeurIPS 2020 competition. Please see https://ai.facebook.com/hatefulmemes. The competition rules are provided on the competition website.

The .jsonl format contains one JSON-encoded example per line, each of which has the following fields:

‘text’ - the text occurring in the meme ‘img’ - the path to the image in the img/ directory ‘label’ - the label for the meme (0=not-hateful, 1=hateful), provided for train and dev

The metric to use is AUROC. You may also report accuracy in addition, since this is more interpretable. To compute these metrics, we recommend the rocaucscore and accuracy_score methods in sklearn.metrics, with default settings.

Note on Annotator Accuracy

As is to be expected with a dataset of this size and nature, some of the examples in the training set have been misclassified. We are not claiming that our dataset labels are completely accurate, or even that all annotators would agree on a particular label. Misclassifications, although possible, should be very rare in the dev and seen test set, however, and we will take extra care with the unseen test set.

As a reminder, the annotations collected for this dataset were not collected using Facebook annotators and we did not employ Facebook’s hate speech policy. As such, the dataset labels do not in any way reflect Facebook’s official stance on this matter.

License

The dataset is licensed under the terms in the LICENSE.txt file.

Image Attribution

If you wish to display example memes in your paper, please provide the following attribution:

Image is a compilation of assets, including ©Getty Image.

Citations

If you wish to cite this work, please use the following BiBTeX:

@inproceedings{Kiela2020TheHM,
  title={The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes},
  author={Douwe Kiela and Hamed Firooz and Aravind Mohan and Vedanuj Goswami and Amanpreet Singh and Pratik Ringshia and Davide Testuggine},
  year={2020}
}

Contact

If you have any questions or comments on the dataset, please contact hatefulmemeschallenge@fb.com.