globis-university/aozorabunko-chats
Overview This dataset is of conversations extracted from Aozora Bunko (青空文庫), which collects public-domain books in Japan, using a simple heuristic approach. [For Japanese] 日本語での概要説明を Qiita に記載しました: https://qiita.com/akeyhero/items/b53eae1c0bc4d54e321f Method First, lines surrounded by quotation mark pairs (「」) are extracted as utterances from the text field of globis-university/aozorabunko-clean. Then, consecutive utterances are collected and grouped together.… See the full description on the dataset page: https://huggingface.co/datasets/globis-university/aozorabunko-chats.
Overview
This dataset is of conversations extracted from Aozora Bunko (青空文庫), which collects public-domain books in Japan, using a simple heuristic approach.
[For Japanese] 日本語での概要説明を Qiita に記載しました: https://qiita.com/akeyhero/items/b53eae1c0bc4d54e321f
Method
First, lines surrounded by quotation mark pairs (「」) are extracted as utterances from the text field of globis-university/aozorabunko-clean. Then, consecutive utterances are collected and grouped together.
The code to reproduce this dataset is made available on GitHub: globis-org/aozorabunko-exctractor.
Notice
As the conversations are extracted using a simple heuristic, a certain amount of the data may actually be monologues.
Tips
If you prefer to employ only modern Japanese, you can filter entries with: row["meta"]["文字遣い種別"] == "新字新仮名".
Example
>>> from datasets import load_dataset
>>> ds = load_dataset('globis-university/aozorabunko-chats')
>>> ds
DatasetDict({
train: Dataset({
features: ['chats', 'footnote', 'meta'],
num_rows: 5531
})
})
>>> ds = ds.filter(lambda row: row['meta']['文字遣い種別'] == '新字新仮名') # only modern Japanese
>>> ds
DatasetDict({
train: Dataset({
features: ['chats', 'footnote', 'meta'],
num_rows: 4139
})
})
>>> book = ds['train'][0] # one of the works
>>> book['meta']['作品名']
'スリーピー・ホローの伝説'
>>> chats = book['chats'] # list of the chats in the work; type: list[list[str]]
>>> len(chats)
1
>>> chat = chats[0] # one of the chats; type: list[str]
>>> for utterance in chat:
... print(utterance)
...
人生においては、たとえどんな場合でも必ず利点や愉快なことがあるはずです。もっともそれは、わたくしどもが冗談をすなおに受けとればのことですが
そこで、悪魔の騎士と競走することになった人は、とかくめちゃくちゃに走るのも当然です
したがって、田舎の学校の先生がオランダ人の世継ぎ娘に結婚を拒まれるということは、彼にとっては、世の中で栄進出世にいたるたしかな一歩だということになりますLicense
CC BY 4.0