Datasetpublic

airudit/UD_v2_17_POS_LEMMA

Dataset Information This dataset is designed for the Part-of-Speech (POS) tagging and Lemmatization tasks and is used to train the Airudit multitask model. Dataset Description The dataset combines Universal Dependencies v2.17 French corpora available atcommul/universal_dependencies. Included corpora: commul/universal_dependencies/fr_gsd commul/universal_dependencies/fr_sequoia commul/universal_dependencies/fr_partut commul/universal_dependencies/fr_parisstories… See the full description on the dataset page: https://huggingface.co/datasets/airudit/UD_v2_17_POS_LEMMA.

sourceHugging Faceupdated 4mo agoView on Hugging Face

0likes48downloads

Dataset Card

Dataset Information

This dataset is designed for the Part-of-Speech (POS) tagging and Lemmatization tasks and is used to train the Airudit multitask model.

Dataset Description

The dataset combines Universal Dependencies v2.17 French corpora available at commul/universal_dependencies.

Included corpora:

—commul/universaldependencies/frgsd
—commul/universaldependencies/frsequoia
—commul/universaldependencies/frpartut
—commul/universaldependencies/frparisstories
—commul/universaldependencies/frrhapsodie
—commul/universaldependencies/frpoitevindivital
—commul/universaldependencies/frpud

Dataset Structure

The dataset contains the following splits:

DatasetDict({
    train: Dataset({
        features: ['sent_id', 'tokens', 'upos', 'lemmas'],
        num_rows: 20159
    })
    dev: Dataset({
        features: ['sent_id', 'tokens', 'upos', 'lemmas'],
        num_rows: 3768
    })
    test_fr_gsd: Dataset({
        features: ['sent_id', 'tokens', 'upos', 'lemmas'],
        num_rows: 416
    })
    test_fr_sequoia: Dataset({
        features: ['sent_id', 'tokens', 'upos', 'lemmas'],
        num_rows: 456
    })
    test_fr_partut: Dataset({
        features: ['sent_id', 'tokens', 'upos', 'lemmas'],
        num_rows: 110
    })
    test_fr_parisstories: Dataset({
        features: ['sent_id', 'tokens', 'upos', 'lemmas'],
        num_rows: 697
    })
    test_fr_rhapsodie: Dataset({
        features: ['sent_id', 'tokens', 'upos', 'lemmas'],
        num_rows: 840
    })
    test_fr_pud: Dataset({
        features: ['sent_id', 'tokens', 'upos', 'lemmas'],
        num_rows: 1000
    })
    test_fr_poitevindivital: Dataset({
        features: ['sent_id', 'tokens', 'upos', 'lemmas'],
        num_rows: 239
    })
})

Labels

Labels are encoded in the dataset and can be retrieved as following :

python

from datasets import load_dataset
ds = load_dataset('airudit/UD_v2_17_POS')
ds["train"].features["upos"].feature.names
# ['NOUN', 'PUNCT', 'ADP', 'NUM', 'SYM', 'SCONJ', 'ADJ', 'PART', 'DET', 'CCONJ', 'PROPN', 'PRON', 'X', '_', 'ADV', 'INTJ', 'VERB', 'AUX']

🔍 Quick usage example:

python

from datasets import load_dataset
ds = load_dataset('airudit/UD_v2_17_POS')
ds["train"][0]
# {'sent_id': [...], 'tokens': [...], 'upos': [...], 'lemmas': [...]}

📦 Dataset Pre-Processing :

—Train and dev splits from all corpora are merged into unified train and dev splits.
—Test splits are preserved and renamed according to their source corpus.
—Only the relevant columns are kept: ["sent_id", "tokens", "upos", "lemmas"].
—In the Partut dataset, particles such as "ne" and "n'" are annotated with PART. To ensure consistency with other UD corpora, these labels are converted to ADV.
—The dataset was generated using the following script: multitask-nlp/src/multitasknlp/datasets/posdataset_preparation.py