mediabiasgroup/mbib-base

Dataset Card for Media-Bias-Identification-Benchmark

Table of Contents

• —Dataset Card for Media-Bias-Identification-Benchmark
• —Table of Contents
• —Dataset Description
• —Dataset Summary
• —Tasks and Information
• —Baseline
• —Languages
• —Dataset Structure
• —Data Instances
• —cognitive-bias
• —Data Fields
• —Considerations for Using the Data
• —Social Impact of Dataset
• —Discussion of Biases
• —Other Known Limitations
• —Citation Information
• —Contributions

Dataset Description

• —Homepage: https://github.com/Media-Bias-Group/Media-Bias-Identification-Benchmark
• —Repository: https://github.com/Media-Bias-Group/Media-Bias-Identification-Benchmark
• —Paper: https://doi.org/10.1145/3539618.3591882
• —Point of Contact: Martin Wessel

Baseline

<table> <tr><td><b>Task</b></td><td><b>Model</b></td><td><b>Micro F1</b></td><td><b>Macro F1</b></td></tr>

<td>cognitive-bias</td> <td> ConvBERT/ConvBERT</td> <td>0.7126</td> <td> 0.7664</td></tr> <td>fake-news</td> <td>Bart/RoBERTa-T</td> <td>0.6811</td> <td> 0.7533</td> </tr> <td>gender-bias</td> <td> RoBERTa-T/ELECTRA</td> <td>0.8334</td> <td>0.8211</td> </tr> <td>hate-speech</td> <td>RoBERTA-T/Bart</td> <td>0.8897</td> <td> 0.7310</td> </tr> <td>linguistic-bias</td> <td> ConvBERT/Bart </td> <td> 0.7044 </td> <td> 0.4995 </td> </tr> <td>political-bias</td> <td> ConvBERT/ConvBERT </td> <td> 0.7041 </td> <td> 0.7110 </td> </tr> <td>racial-bias</td> <td> ConvBERT/ELECTRA </td> <td> 0.8772 </td> <td> 0.6170 </td> </tr> <td>text-leve-bias</td> <td> ConvBERT/ConvBERT </td> <td> 0.7697</td> <td> 0.7532 </td> </tr> </table>

Languages

All datasets are in English

Dataset Structure

Data Instances

cognitive-bias

An example of one training instance looks as follows.

{
  "text": "A defense bill includes language that would require military hospitals to provide abortions on demand",
  "label": 1
}

Data Fields

• —text: a sentence from various sources (eg., news articles, twitter, other social media).
• —label: binary indicator of bias (0 = unbiased, 1 = biased)

Considerations for Using the Data

Social Impact of Dataset

We believe that MBIB offers a new common ground for research in the domain, especially given the rising amount of (research) attention directed toward media bias

Citation Information

@inproceedings{
    title = {Introducing MBIB - the first Media Bias Identification Benchmark Task and Dataset Collection},
    author = {Wessel, Martin and Spinde, Timo and Horych, Tomáš and Ruas, Terry and Aizawa, Akiko and Gipp, Bela},
    year = {2023},
    note = {[in review]}
}