samirmsallem/wiki_definitions_de_multitask

Dataset Card for Wikipedia Definitions for Multitask (NER/Text Classification)

The Wikipedia Definitions for Multitask (NER/Text Classification) dataset is a dataset to train language models to recognize definition sentences and non-definition sentences. The dataset includes training and test data to recognize this discipline by Named Entity Recognition, but also by Sentence Classification.

Dataset Sources

• —Wikimedia/wikipedia Dataset: https://huggingface.co/datasets/wikimedia/wikipedia/viewer/20231101.de/train

Languages

The language supported is German.

Dataset Structure

Data Instances

An example from the dataset for a definitional sentence looks as follows:

{
  "text":"Liberty Basic ist ein Dialekt der Programmiersprache BASIC und eine kommerzielle integrierte Entwicklungsumgebung, mit der Programme f\u00fcr Windows erstellt werden k\u00f6nnen.",
  "tokens":["Liberty","Basic","ist","ein","Dialekt","der","Programmiersprache","BASIC","und","eine","kommerzielle","integrierte","Entwicklungsumgebung",",","mit","der","Programme","f\u00fcr","Windows","erstellt","werden","k\u00f6nnen","."],
  "ner_tags":[3,4,5,1,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2],
  "ner_labels":["B-DF","I-DF","B-VF","B-GF","I-GF","I-GF","I-GF","I-GF","I-GF","I-GF","I-GF","I-GF","I-GF","I-GF","I-GF","I-GF","I-GF","I-GF","I-GF","I-GF","I-GF","I-GF","I-GF"],
  "label":1
}

An example from the dataset for a non-definitional sentence looks as follows:

{
  "text":"Drools ist ein Regelbasiertes System und kann selbst auf unterschiedliche Ereignisse reagieren.",
  "tokens":["Drools","ist","ein","Regelbasiertes","System","und","kann","selbst","auf","unterschiedliche","Ereignisse","reagieren","."],
  "ner_tags":[3,5,0,3,4,0,0,0,0,0,0,0,0],
  "ner_labels":["B-DF","B-VF","O","B-DF","I-DF","O","O","O","O","O","O","O","O"],
  "label":0
}

Data Fields

• —text: full text of the sample
• —tokens: the tokens of the example text
• —ner_tags: the NER tags of each token
• —ner_labels: the NER labels (translated ner_tags) of each token
• —label: the Text Classification label for the sentence

NER Tags

Text Classification Tags

Data Splits

The data is split into train and test set. The split sizes are as follow: