Datasetpublic

multilingual-discourse-hub/disrpt

Disrpt is a multilingual, multi-framework unified discourse analysis benchmark. It unifies discourse relation classification tasks (.rels) and discourse segmentation (.connlu) for many languages. ⚠️ This repo only contains the disrpt dataset when the underlying data is permissively licensed. Some datasets rely on corpora like the PTB. To load these datasets, run the following: pip install disrpt-utils Then from disrpt_utils import load_dataset corpora_paths={ # ⚠️✍️ TODO… See the full description on the dataset page: https://huggingface.co/datasets/multilingual-discourse-hub/disrpt.

sourceHugging Faceapache-2.0updated 1y agoView on Hugging Face

3likes15kdownloads

Dataset Card

Disrpt is a multilingual, multi-framework unified discourse analysis benchmark.

It unifies discourse relation classification tasks (.rels) and discourse segmentation (.connlu) for many languages.

⚠️ This repo only contains the disrpt dataset when the underlying data is permissively licensed. Some datasets rely on corpora like the PTB. To load these datasets, run the following:

pip install disrpt-utils Then

python

from disrpt_utils import load_dataset

corpora_paths={
    # ⚠️✍️ TODO Input your own paths 
    "tur.pdtb.tdb": "/mnt/nfs_share_magnet2/dsileo/data/disrpt/TDB1.0/raw/",
    "eng.rst.rstdt": "/mnt/nfs_share_magnet2/dsileo/data/disrpt/RSTtrees-WSJ-main-1.0-20231128T125253Z-001/"
}

config_name="eng.rst.rstdt"
load_dataset(config_name,'conllu',corpora_paths=corpora_paths)

Otherwise, most datasets are directly accessible:

python

from datasets import load_dataset
load_dataset('multilingual-discourse-hub/disrpt','eng.dep.scidtb.rels')

Citations

@inproceedings{braud2024disrpt,
  title={{DISRPT}: A Multilingual, Multi-domain, Cross-framework Benchmark for Discourse Processing},
  author={Braud, Chloé and Zeldes, Amir and Rivière, Laura and Liu, Yang Janet and Muller, Philippe and Sileo, Damien and Aoyama, Tatsuya},
  booktitle={Proceedings of LREC-COLING 2024},
  year={2024}
}

Underline datasets:

https://huggingface.co/datasets/multilingual-discourse-hub/disrpt/raw/main/bibtex.json