commul/ud_genre

UD Genre Labels full-ud-v1.1.1-ud2.18

Derived sentence-level genre annotations for the universal-dependencies/universal_dependencies Universal Dependencies dataset. These labels are produced by the bootstrapping pipeline and are not authoritative gold annotations.

Dataset Description

• —Homepage: https://github.com/bot-zen/ud-genre-bootstrap
• —Repository: https://github.com/bot-zen/ud-genre-bootstrap
• —Source dataset: hf://universal-dependencies/universal_dependencies
• —Paper: https://universaldependencies.org/udw26/papers/41_Paper.pdf
• —Point of Contact: appliedlinguisticsdevs@eurac.edu

Dataset Summary

This dataset provides a sentence-level genre layer aligned to the universal-dependencies/universal_dependencies Parquet release. Each row contains one derived genre label for one UD sentence and can be joined back to the UD source data by (treebank, split, sent_id).

The export is a derived annotation layer, not a replacement for the UD treebanks and not a hand-validated gold genre dataset.

Label Coverage And Provenance

• —Total UD sentences in artifact: 2,221,815
• —Labeled sentences: 2,221,815 (100.0%)
• —Unlabeled sentences: 0
• —Genres exported: 18
• —Metadata-derived labels: 1,334,339 (60.1% of labeled sentences)
• —Clustering-derived labels: 887,476 (39.9% of labeled sentences)

For cluster-derived rows, confidence is the top-1 cluster-label similarity score. Direct metadata-derived rows use confidence 1.0.

• —Mean confidence: 0.9835
• —Median confidence: 1.0000

Genre Distribution

Loading

from datasets import load_dataset

genres = load_dataset(
    "commul/ud_genre",
    revision="2.18",
    split="train",
)

The train split is the single exported split containing all sentence-level genre labels for this artifact.

For immutable provenance, load the artifact tag:

genres = load_dataset(
    "commul/ud_genre",
    revision="artifact/full-ud-v1.1.1/ud2.18",
    split="train",
)

Joining With Universal Dependencies

from datasets import load_dataset

genres = load_dataset(
    "commul/ud_genre",
    revision="2.18",
    split="train",
)

ud = load_dataset(
    "universal-dependencies/universal_dependencies",
    "en_ewt",
    revision="2.18",
    split="train",
)

genre_by_key = {
    (row["treebank"], row["split"], row["sent_id"]): row["genre"]
    for row in genres
    if row["treebank"] == "en_ewt" and row["split"] == "train"
}

first = ud[0]
genre = genre_by_key.get(("en_ewt", "train", first["sent_id"]))

Release Identity

• —Train ID: full-ud-v1.1.1
• —Artifact key: full-ud-v1.1.1-ud2.18
• —Inventory status: default_hotfix
• —HF branches: 2.18
• —HF tag: artifact/full-ud-v1.1.1/ud2.18
• —HF default branch: main
• —HF repo: commul/ud_genre
• —UD version: 2.18
• —Scope: full
• —Label schema: ud
• —Artifact version: v1.1.1
• —Source repo: https://github.com/bot-zen/ud-genre-bootstrap
• —Source commit: 36188cb591a9d3122dd95b67258dc8e1fc7dcd4f
• —Source branch: release/full-ud-v1
• —Source tag: source/full-ud-v1.1.1
• —Config SHA-256: ba9d3e8ec173f90659422a07e536ad74de9db2f848c3636b6d00321327a56eb8
• —Canonical labels: academic, bible, blog, email, fiction, government, grammar-examples, learner-essays, legal, medical, news, nonfiction, poetry, reviews, social, spoken, web, wiki

Release Configuration

• —Config: 2.18-community-release
• —Run ID: full-ud-v1.1.1-ud2.18
• —UD source: hf://universal-dependencies/universal_dependencies
• —UD source revision: 2.18
• —Embeddings: intfloat/multilingual-e5-large / mean
• —Clustering: gmm
• —Reference weighting: sentence_count

Output Columns

• —treebank, split, sent_id: primary join key back to UD
• —genre: derived sentence label
• —confidence: top-1 similarity score for cluster-derived rows; 1.0 for direct metadata-derived rows
• —method: single-genre-treebank, virtual-split, or cluster-derived
• —ud_version, model, pooling, clustering_method, config_name, run_id: compact row-level provenance

Evaluation Framing

• —paper_parity is used only for comparison with the original GMM+L paper protocol.
• —End-user quality is tracked with sentence-level generalization metrics, which are stricter and more directly relevant for downstream annotation use.
• —Known limitation: some paper-era treebank genre inventories are not fully recoverable from current sentence-level metadata subsets.

Evaluation Summary

• —Baseline: UD v2.18 all_available generalization baseline
• —Scope: Locked evaluation baseline measured on UD 2.18.
• —Protocol: generalization
• —Treebank set: all_available
• —Config: configs/releases/full-ud-v1.1.0.yaml
• —Source log: output/logs/2.18-all_available-generalization-e5_large-k10-anchor_combined-baseline.log

• —Missing anchor genres in this baseline: email

Release Summary

• —Total sentences: 2,221,815
• —Labeled sentences: 2,221,815
• —Genres exported: 18
• —Methods exported: cluster-derived, single-genre-treebank, virtual-split

Source Mapping Files

• —configs/genre_mappings.json (sha256: a52bbd9d5be0ec267da69624930e5087f28359e0dea88d9f039534b0167cbe0e)
• —configs/metadata_patterns.json (sha256: 2e0898fe402e72e6050a5efffd8f963696e6c693763325313c7b290540696d5c)
• —configs/pud-patterns.json (sha256: 4b5ca652110604a9b567deea630115eda24a2b734359b4f48e71c0fee7b2034d)

Citation

Please cite the UD Workshop paper associated with this dataset: https://universaldependencies.org/udw26/papers/41_Paper.pdf

Contact

Point of Contact: appliedlinguisticsdevs@eurac.edu