EXOROBOURII/Stanza-Wikitext-2

1---2license: cc-by-sa-4.03task_categories:4- text-generation5- token-classification6- feature-extraction7language:8- en9size_categories:10- 1M<n<10M11pretty_name: 'Stanza-Wikitext-2: A Structurally Enriched Modernization of WikiText-2'12tags:13- dependency-parsing14- universal-dependencies15- nlp-dataset16- structural-linguistics17- named-entity-recognition18- wikipedia19---20 21# Dataset Card for Stanza-Wikitext-222 23## Dataset Description24 25Stanza-Wikitext-2 is a structurally pristine, mathematically verified NLP dataset designed for multi-task language modeling, custom tokenizer training, structural NLP research, and mechanistic interpretability work.26 27It is a rigorously modernized and annotated derivative of the `wikitext-2-raw-v1` corpus. Using the Stanford NLP `Stanza` neural pipeline, every token in the corpus has been explicitly mapped to its grammatical, syntactic, and semantic function across seven aligned annotation layers. Stanza-Wikitext-2 preserves document geometry, explicitly labeling Markdown headers to support structure-aware neural architectures.28 29- **Curated by:** Jonathan R. Belanger (Exorobourii LLC)30- **Language:** English (`en`)31- **License:** CC-BY-SA-4.032- **DOI:** 10.57967/hf/806033- **Total Sentences:** 101,455 (across all splits)34- **Total Tokens:** 2,469,91235 36---37 38## Corpus Statistics39 40| Split | Sentences | Tokens |41|-------|-----------|--------|42| Train | 82,760 | 2,021,438 |43| Validation | 8,622 | 210,732 |44| Test | 10,073 | 237,742 |45| **Total** | **101,455** | **2,469,912** |46 47Rows removed by Phase 4c integrity repair: **8** (train split only)48 49---50 51## Structural Characterization52 53Unlike standard text corpora, Stanza-Wikitext-2 ships with a full quantitative geometric characterization derived from its dependency structure. These figures are provided to assist researchers in assessing corpus suitability before use.54 55### Dependency Degree Distribution56 57Dependency degree (number of dependents per token) is strongly right-skewed with faster-than-power-law decay. A KS-based MLE scan (Clauset et al., 2009) found no well-supported power-law regime across the observable degree range. The corpus is heavily left-concentrated — the majority of tokens are leaves.58 59| Percentile | Degree |60|-----------|--------|61| 50th (median) | 0 |62| 90th | 3 |63| 99th | 6 |64| 99.9th | 9 |65| Maximum | 43 |66 67- Degree entropy: **1.839 bits**68- Effective degree vocabulary: degree 0–11 (values above 12 are sparse artifacts of list coordination)69 70### Token Depth Distribution71 72Token depth (distance from dependency root, measured upward) characterizes positional distribution within the tree.73 74| Metric | Value |75|--------|-------|76| Range | 0 – 25 |77| Mean | 2.745 |78| Std | 1.674 |79| Entropy | 2.679 bits |80 81Mean subtree height (measured downward from each node): **5.45 nodes**. Maximum subtree height: **26 nodes**. Root center of mass: **0.24**.82 83### Structural Grammar Matrix84 85The cross-product of UPOS tags and DepRel labels yields **451 unique UPOS×DepRel combinations** observed across the corpus. This matrix constitutes a compact geometric fingerprint of the corpus's syntactic behavior and is available as `structural_grammar_matrix.csv` in the associated reports.86 87### Geometric Motif Analysis88 89A dependency motif is defined as a parent node (UPOS×DepRel) paired with a sorted tuple of its children's (UPOS×DepRel) labels. The train split contains **106,057 unique motifs** with a strongly right-skewed frequency distribution.90 91| Coverage | Motifs Required | % of Total Motifs |92|----------|----------------|-------------------|93| 50% | 343 | 0.32% |94| 80% | 7,743 | 7.30% |95| 90% | 33,080 | 31.19% |96| 95% | 69,571 | 65.60% |97| 100% | 106,057 | 100% |98 99The top 343 motifs account for half of all motif occurrences. The distribution is heavily long-tailed: 95% coverage requires 65.6% of the full motif vocabulary, indicating a compact high-frequency structural core alongside a large population of rare configurations.100 101### Structural Rigidity by UPOS102 103Dependency degree varies substantially by part-of-speech, reflecting syntactic valency differences. VERB is the highest-degree head class; functional categories cluster near zero.104 105| UPOS | Mean Degree | Max Degree | Entropy (bits) |106|------|-------------|------------|----------------|107| VERB | 3.54 | 15 | 2.88 |108| NOUN | 2.22 | 36 | 2.61 |109| PROPN | 1.32 | 43 | 2.27 |110| ADJ | 0.56 | 17 | 1.32 |111| ADV | 0.25 | 9 | 0.89 |112| AUX | 0.02 | 8 | 0.11 |113| DET | 0.02 | 10 | 0.11 |114| PUNCT | 0.006 | 11 | 0.03 |115| PART | 0.005 | 6 | 0.03 |116 117### Structural Information Content118 119Normalized mutual information between structural measurements and linguistic labels:120 121| Pair | NMI |122|------|-----|123| Degree × UPOS | 0.223 |124| Degree × DepRel | 0.294 |125| Depth × UPOS | 0.054 |126| Depth × DepRel | 0.118 |127 128Degree carries substantially more linguistic signal than depth. Neither measurement is redundant with linguistic category — they capture geometrically distinct aspects of syntactic structure.129 130### Per-Sentence Structural Complexity131 132Per-sentence degree entropy has mean **1.555 bits** (std 0.275, max 1.954 bits). Structural complexity means are stable across all three splits, confirming that the canonical WikiText-2 split boundaries do not introduce distributional artifacts.133 134---135 136## Dataset Structure137 138Stanza-Wikitext-2 uses **Parallel Arrays**. Each row represents a single sentence. All linguistic features are stored in co-indexed, equal-length arrays guaranteeing 1:1 token-to-annotation alignment.139 140### Schema141 142| Column | Type | Description |143|--------|------|-------------|144| `chunk_id` | int64 | Positional ID of the text block within the document stream |145| `sentence_id` | int64 | Positional ID of the sentence within its chunk |146| `raw_text` | string | Cleaned, normalized sentence text |147| `is_header` | bool | `True` if the sentence is a structural document header |148| `section_level` | int64 | Markdown header depth (1–6); `0` if not a header |149| `tokens` | list[str] | Surface word forms |150| `lemmas` | list[str] | Morphological base forms |151| `upos` | list[str] | Universal POS tags (17-class UD tagset) |152| `xpos` | list[str] | Penn Treebank POS tags |153| `head` | list[int64] | 1-indexed syntactic head positions (0 = root anchor) |154| `deprel` | list[str] | Universal Dependencies relation labels |155| `ner` | list[str] | Named entity tags in BIOES format |156 157All array columns are co-indexed: `column[i]` refers to the same token across all columns for a given row.158 159---160 161## Methodology & Provenance162 163### Phase 1: Cryptographic Ingestion164 165To prevent silent upstream updates from compromising downstream reproducibility, this dataset was built from a cryptographically verified snapshot of the `ggml-org/ci` raw mirror.166 167- **Source Archive:** `wikitext-2-raw-v1.zip`168- **SHA-256 Checksum:** `ef7edb566e3e2b2d31b29c1fdb0c89a4cc683597484c3dc2517919c615435a11`169 170### Phase 2: Degradation Audit171 172WikiText-2 is distributed in two variants: a pre-tokenized `.tokens` format in which low-frequency terms are replaced with `<unk>` substitution tokens, and a `.raw` format retaining original surface forms. This pipeline operates on the `.raw` files exclusively. A precautionary contamination audit computed a penalized degradation score per text block:173 174```175D*(P) = (|unk| / N) · log₂(1 + √N)176```177 178The audit confirmed zero `<unk>` tokens across all 23,767 text blocks, returning a clean result. No filtering was applied or required. This validates the source file selection: by operating on `.raw` rather than `.tokens`, the pipeline inherits no vocabulary substitution artifacts, and downstream analyses reflect genuine surface token distributions.179 180### Phase 3: GPU-Accelerated Normalization181 182Text normalization was performed using NVIDIA RAPIDS cuDF on an L4 GPU. Four operations applied in sequence:183 1841. **Whitespace normalization:** leading/trailing whitespace stripped1852. **Hyphen modernization:** legacy `@-@` artifacts collapsed to standard hyphens (e.g. `Apollo @-@ Soyuz` → `Apollo-Soyuz`)1863. **Punctuation normalization:** floating punctuation corrected via CPU bypass using Python `re` with backreferences (e.g. `word ,` → `word,`)1874. **Header normalization:** `= Title =` through `====== Title ======` converted to Markdown H1–H6 in strict descending order to preserve document hierarchy188 189### Phase 4: Stanza NLP Enrichment190 191Stanza 1.11.1 initialized with `tokenize, pos, lemma, depparse, ner` on GPU. Output serialized to Parquet with ZSTD compression (level 3).192 193Following enrichment, all Parquet files were subjected to a microscopic integrity audit guaranteeing:194 1951. **Dimensional symmetry:** all parallel arrays within a row are equal length1962. **Root singularity:** every sentence has exactly one dependency root (`head == 0`)1973. **Graph bounds:** no head index points outside the sentence boundary198 1998 structurally invalid sentences were identified in the train split and removed via automated ledger repair. The Stanza-Wikitext-2 dataset is **100% structurally valid** across all splits.200 201### Phase 5: Structural Metadata Injection202 203`is_header` and `section_level` columns injected via vectorized Markdown header detection. Enables structure-aware models to condition on document position without reprocessing raw text.204 205---206 207## Usage208 209```python210import pandas as pd211 212# Load a split213df = pd.read_parquet("hf://datasets/EXOROBOURII/Stanza-Wikitext-2/wiki.train.enriched.parquet")214 215# Aligned token access216sentence = df.iloc[0]217for token, upos, deprel, head in zip(218    sentence['tokens'],219    sentence['upos'],220    sentence['deprel'],221    sentence['head']222):223    print(f"{token:<20} {upos:<8} {deprel:<16} head={head}")224 225# Filter to content sentences only (exclude headers)226content = df[~df['is_header']].reset_index(drop=True)227 228# Filter to a specific section level229h2_headers = df[df['section_level'] == 2]230 231# Reconstruct dependency tree for a sentence232from collections import defaultdict233 234def get_children(head_array):235    children = defaultdict(list)236    for i, h in enumerate(head_array):237        if h > 0:238            children[h - 1].append(i)  # convert to 0-indexed239    return children240 241row = df.iloc[10]242children = get_children(row['head'])243root_idx  = list(row['head']).index(0)244print(f"Root token: {row['tokens'][root_idx]} ({row['upos'][root_idx]})")245print(f"Root dependents: {[row['tokens'][c] for c in children[root_idx]]}")246```247 248---249 250## Reports and Analysis Artifacts251 252The following analytical reports are available in the dataset repository:253 254| File | Description |255|------|-------------|256| `structural_grammar_matrix.csv` | 451 UPOS×DepRel combinations with frequencies |257| `geometric_motifs_wiki.train.enriched.csv` | 106,057 unique dependency motifs |258| `entity_distribution.csv` | Named entity frequencies and types |259| `entity_cooccurrence.csv` | Sentence-level entity co-occurrence pairs |260| `motif_analytics_summary.txt` | Motif coverage analysis and valency statistics |261| `structural_rigidity_full.csv` | Per-UPOS weighted valency statistics |262| `degree_distribution.csv` | Full token degree frequency table |263| `depth_distribution.csv` | Full token depth frequency table |264| `mi_summary.csv` | NMI values for degree/depth × UPOS/DepRel |265| `sentence_structural_stats.csv` | Per-sentence degree and depth statistics |266 267---268 269## Citation270 271```bibtex272@dataset{belanger2025stanza2,273  author    = {Belanger, Jonathan R.},274  title     = {Stanza-Wikitext-2: A Structurally Enriched Modernization of WikiText-2},275  year      = {2026},276  publisher = {HuggingFace},277  url       = {https://huggingface.co/datasets/EXOROBOURII/Stanza-Wikitext-2},278  doi       = {10.57967/hf/8060}279}280```281 282---283 284## License285 286CC-BY-SA-4.0. Derivative of WikiText-2 (CC-BY-SA-4.0, Merity et al. 2016).