taishi-i/awesome-japanese-nlp-multilabel-dataset

Dataset overview

This is a dataset for Japanese natural language processing with multi-label annotations of research field labels for GitHub repositories in the NLP domain. Please refer to this paper for the specific method of constructing the dataset. It is written in Japanese.

Input and Output

• —Input: Information from GitHub repositories (description, README text, PDF text, screenshot images)
• —Output: Multi-label classification of NLP research fields

Problem Setting of the Dataset:

• —Training Data: GitHub repositories from before 2022
• —Test Data: GitHub repositories from 2023
• —Objective: To predict multi-labels of research field labels in the NLP domain
• —We used the positive examples from awesome-japanese-nlp-classification-dataset.

Dataset Features:

• —A multimodal dataset including GitHub summaries, README files, PDF files, and screenshot images.
• —The annotation labels were assigned with reference to Exploring-NLP-Research.

If you need image data and PDF files, please submit a request to the community. This dataset uses text extracted from PDF files and does not include image data.

Based on GitHub's terms of service, please use this dataset for research purposes only.

How to use this dataset

Install the datasets library.

pip install datasets

How to load in Python.

from datasets import load_dataset

dataset = load_dataset("taishi-i/awesome-japanese-nlp-multilabel-dataset")

# To access the dataset sample
print(dataset["train"][0])

Here is a sample of the dataset.

{
  "is_exist": True,
  "url": "https://github.com/scriptin/jmdict-simplified",
  "created_at": "2016-02-07T16:34:32Z",
  "description": "JMdict and JMnedict in JSON format",
  "pdf_text": "scriptin / jmdict-simplified\nPublic\nBranches\nTags\n",
  "readme_text": "# jmdict-simplified\n\n**[JMdict][], [JMnedict][], [Kanjidic][], and [Kradfile/Radkfile][Kradfile] in JSON format**<br>\n",
  "nlp_taxonomy_classifier_labels": [
    "Multilinguality"
  ],
  "awesome_japanese_nlp_labels": [
    "Annotation and Dataset Development",
    "Vocabulary, Dictionary, and Language Input Method"
  ]
}

• —is_exist: A boolean value indicating whether the GitHub repository exists (True if it exists) at the time of retrieval.
• —url: The URL of the GitHub repository.
• —created_at: The timestamp indicating when the repository was created (in ISO 8601 format).
• —description: A short description of the repository provided on GitHub.
• —pdf_text: Extracted text from a PDF file that contains a saved version of the GitHub repository's top page.
• —readme_text: Extracted text from the repository's README file.
• —nlp_taxonomy_classifier_labels: Manually annotated multi-label annotations of NLP research field labels.
• —awesome_japanese_nlp_labels: Manually annotated multi-label annotations of research field labels specifically for Japanese natural language processing.

Details of the dataset.

DatasetDict({
    train: Dataset({
        features: ['is_exist', 'url', 'created_at', 'description', 'pdf_text', 'readme_text', 'nlp_taxonomy_classifier_labels', 'awesome_japanese_nlp_labels'],
        num_rows: 407
    })
    validation: Dataset({
        features: ['is_exist', 'url', 'created_at', 'description', 'pdf_text', 'readme_text', 'nlp_taxonomy_classifier_labels', 'awesome_japanese_nlp_labels'],
        num_rows: 17
    })
    test: Dataset({
        features: ['is_exist', 'url', 'created_at', 'description', 'pdf_text', 'readme_text', 'nlp_taxonomy_classifier_labels', 'awesome_japanese_nlp_labels'],
        num_rows: 60
    })
})

Baseline

The baseline model was used for TimSchopf/nlp_taxonomy_classifier.

The fine-tuned model was trained using this dataset to fine-tune the baseline model.

License

We collect and publish this dataset under GitHub Acceptable Use Policies - 7. Information Usage Restrictions and GitHub Terms of Service - H. API Terms for research purposes. This dataset should be used solely for research verification purposes. Adhering to GitHub's regulations is mandatory.