Datasetpublic

EXOROBOURII/Stanza-Wikitext-2

Dataset Card for Stanza-Wikitext-2 Dataset Description Stanza-Wikitext-2 is a structurally pristine, mathematically verified NLP dataset designed for multi-task language modeling, custom tokenizer training, structural NLP research, and mechanistic interpretability work. It is a rigorously modernized and annotated derivative of the wikitext-2-raw-v1 corpus. Using the Stanford NLP Stanza neural pipeline, every token in the corpus has been explicitly mapped to its… See the full description on the dataset page: https://huggingface.co/datasets/EXOROBOURII/Stanza-Wikitext-2.

sourceHugging Facecc-by-sa-4.0updated 6mo agoView on Hugging Face

0likes103downloads

rootdiagnostic_error_ledger.jsonl

diagnostic_error_ledger.jsonl9 linesDownload Raw Back to root

1{"split": "wiki.train.enriched.parquet", "chunk_id": 7168, "sentence_id": 6, "error_type": "ROOT_VIOLATION", "details": "Sentence has 0 roots (Expected exactly 1)."}2{"split": "wiki.train.enriched.parquet", "chunk_id": 9057, "sentence_id": 1, "error_type": "OUT_OF_BOUNDS_GRAPH", "details": "Found dependency heads pointing to non-existent tokens: [np.int64(7)] (Max valid: 6)"}3{"split": "wiki.train.enriched.parquet", "chunk_id": 9085, "sentence_id": 5, "error_type": "OUT_OF_BOUNDS_GRAPH", "details": "Found dependency heads pointing to non-existent tokens: [np.int64(35), np.int64(35)] (Max valid: 34)"}4{"split": "wiki.train.enriched.parquet", "chunk_id": 11599, "sentence_id": 5, "error_type": "ROOT_VIOLATION", "details": "Sentence has 0 roots (Expected exactly 1)."}5{"split": "wiki.train.enriched.parquet", "chunk_id": 17639, "sentence_id": 1, "error_type": "OUT_OF_BOUNDS_GRAPH", "details": "Found dependency heads pointing to non-existent tokens: [np.int64(34), np.int64(34), np.int64(34)] (Max valid: 33)"}6{"split": "wiki.train.enriched.parquet", "chunk_id": 28556, "sentence_id": 1, "error_type": "OUT_OF_BOUNDS_GRAPH", "details": "Found dependency heads pointing to non-existent tokens: [np.int64(4), np.int64(4)] (Max valid: 3)"}7{"split": "wiki.train.enriched.parquet", "chunk_id": 28896, "sentence_id": 3, "error_type": "OUT_OF_BOUNDS_GRAPH", "details": "Found dependency heads pointing to non-existent tokens: [np.int64(37)] (Max valid: 36)"}8{"split": "wiki.train.enriched.parquet", "chunk_id": 34567, "sentence_id": 2, "error_type": "OUT_OF_BOUNDS_GRAPH", "details": "Found dependency heads pointing to non-existent tokens: [np.int64(16), np.int64(16), np.int64(16)] (Max valid: 15)"}9