Datasetpublic

dragonscale-ai/kniv-corpus-en

kniv-corpus-en A multi-domain English NLP corpus with four annotation layers: Named Entity Recognition (18 types), POS tagging (17 UPOS tags), dependency parsing, and dialog act classification (9 types). All data is commercially licensed (CC BY-SA 4.0 compatible). Built for training kniv multi-task NLP models that power the uniko cognitive memory system. Quick Start from datasets import load_dataset # Load the full corpus via HuggingFace ds =… See the full description on the dataset page: https://huggingface.co/datasets/dragonscale-ai/kniv-corpus-en.

sourceHugging Facecc-by-sa-4.0updated 5mo agoView on Hugging Face

0likes147downloads

Dataset Card

kniv-corpus-en

A multi-domain English NLP corpus with four annotation layers: Named Entity Recognition (18 types), POS tagging (17 UPOS tags), dependency parsing, and dialog act classification (9 types). All data is commercially licensed (CC BY-SA 4.0 compatible).

Built for training kniv multi-task NLP models that power the uniko cognitive memory system.

Quick Start

python

from datasets import load_dataset

# Load the full corpus via HuggingFace
ds = load_dataset("dragonscale-ai/kniv-corpus-en")
print(ds["train"][0])

python

# Load gold-filtered corpus (recommended for training)
import pandas as pd
from huggingface_hub import hf_hub_download

path = hf_hub_download("dragonscale-ai/kniv-corpus-en", "corpus/gold/train.parquet", repo_type="dataset")
gold = pd.read_parquet(path)
print(f"{len(gold)} gold sentences")

Repository Structure

dragonscale-ai/kniv-corpus-en/
|
+-- data/                                   # Full corpus (HF auto-detected Parquet)
|   +-- train-00000-of-00001.parquet            525,850 sentences  (127.7 MB)
|   +-- dev-00000-of-00001.parquet               65,731 sentences  (13.8 MB)
|   +-- test-00000-of-00001.parquet              65,732 sentences  (14.0 MB)
|
+-- corpus/gold/                            # Gold-filtered corpus (recommended)
|   +-- train.parquet                           237,646 sentences  (56.8 MB)
|   +-- dev.parquet                              65,731 sentences  (13.8 MB)
|   +-- test.parquet                             65,732 sentences  (14.0 MB)
|   +-- progress.json                           Filtering metadata
|
+-- prepared/deberta-v3-large-nlp-en/       # Training-ready JSON splits
|   +-- ner_train.json                          45,000 examples (domain-balanced subsample)
|   +-- ner_dev.json                            65,731 examples
|   +-- ner_test.json                           65,732 examples
|   +-- ud_train.json                           12,544 examples (UD English EWT)
|   +-- ud_dev.json                              2,001 examples
|   +-- ud_test.json                             2,077 examples
|   +-- label_vocabs.json                       Label vocabularies for all tasks
|
+-- train.conllu                            # Full corpus in CoNLL-U format (547.7 MB)
+-- dev.conllu                              (67.9 MB)
+-- test.conllu                             (68.8 MB)
+-- metadata.json                           Split sizes and domain list
+-- README.md

How the Data Was Built

Collection

Text was collected from 17 open sources across 6 domains, chosen for domain diversity and commercial licensing:

Domain	Sentences	Sources
Business	197,891	SEC EDGAR 10-K filings (public domain), Enron email corpus (public domain), OpenStax business textbooks (CC BY-4.0), Odoo ERP documentation (CC BY-SA 3.0), Wikipedia business articles (CC BY-SA 3.0), CUAD contract passages (CC BY-4.0), OpenAlex abstracts (ODC-BY)
Technical	193,032	Wikipedia CS/engineering articles (CC BY-SA 3.0), Python official documentation (PSF License)
Conversation	89,934	Taskmaster 1/2/3 (CC BY-4.0), OpenAssistant OASST1 (Apache 2.0), MultiWOZ 2.2 (Apache 2.0), Glaive Function Calling (Apache 2.0), Discord Dialogues (Apache 2.0)
Encyclopedic	66,490	Wikipedia articles on science, history, geography, arts (CC BY-SA 3.0)
News	58,026	Wikinews articles (CC BY-SA 2.5), Wikipedia journalism/politics articles (CC BY-SA 3.0)
Narrative	51,940	Project Gutenberg classic fiction (public domain)

Preprocessing

—Sentence splitting (non-conversation): spaCy sentencizer splits documents into individual sentences
—Utterance extraction (conversation): each dialog turn kept as a single unit (no sentence splitting -- conversational utterances are natural dialog act boundaries)
—Conversational context: within each conversation, prev_text links each utterance to the preceding turn for context-dependent classification
—Deduplication: exact-match dedup by text content; conversation domain deduped by (conv_id, turn_idx) pairs
—Quality filtering: minimum length, encoding validation, boilerplate removal

Annotation

Four-layer annotation in a single pipeline pass:

1.NER + POS + Dependency: spaCy en_core_web_trf (transformer-based) annotates all three in one forward pass, producing CoNLL-U + JSONL output
2.Validation: GPT-5.4-nano reviews annotations in batches of 5 sentences, returning corrections for POS, NER, and dependency labels. Only valid UPOS corrections are accepted (invalid suggestions like "PART (acceptable)" are rejected). Orphan BIO I-tags (I- without preceding B-) are automatically repaired to B-.
3.Classification: GPT-5.4-nano assigns one of 9 dialog act labels per sentence in batches of 20. Conversation domain sentences include the previous utterance (prev_text) as context for labels like correction, agreement, and filler.

Gold Filtering

The full corpus (525,850 train sentences) was validated sentence-by-sentence using Qwen3-8B via vLLM on an A100 GPU with 64 concurrent requests:

Check	What	Rejection Rate
NER	Every entity span: "Is '{entity}' correctly tagged as {type}?"	9.3%
CLS	Dialog act label validated (with prev_text context for conversations)	4.0%
POS	Up to 8 POS tags per sentence validated against Universal POS tagset	40.9%

Sentences failing any check were removed. 237,646 sentences survived (45.2% of training data). The high POS rejection rate reflects the LLM checking 8 tokens per sentence -- even a ~5% per-token disagreement rate compounds to ~34% per-sentence rejection.

Training Data Preparation

The prepared/ directory contains training-ready JSON splits:

—NER data (45,000 examples): domain-balanced subsample from the 237K gold corpus, with targets: conversation 15K, business 10K, technical 7K, narrative 5K, news 5K, encyclopedic 3K
—UD data (12,544 examples): Universal Dependencies English EWT v2.14 (expert-annotated, not gold-filtered -- already gold standard)

Data Format

Parquet Columns

Column	Type	Description
`sent_id`	string	Unique ID: `{domain}-{index}`
`text`	string	Sentence text
`prev_text`	string?	Previous utterance (conversation domain, nullable)
`cls`	string	Dialog act label
`tokens`	list[str]	Tokenized words
`pos_tags`	list[str]	Universal POS tags
`heads`	list[int]	Dependency head indices (0 = root)
`deprels`	list[str]	Dependency relation labels
`ner_tags`	list[str]	BIO-format NER tags

CoNLL-U Format

Standard 10-column CoNLL-U with metadata comments:

# sent_id = conversation-004217
# text = Oh, could you please change it for outside?
# prev_text = It looks like the table will be for inside.
# cls = correction
1    Oh         oh         INTJ     UH     _    5    discourse    _    _
2    ,          ,          PUNCT    ,      _    5    punct        _    _
3    could      could      AUX      MD     _    5    aux          _    _
4    you        you        PRON     PRP    _    5    nsubj        _    _
5    please     please     VERB     VB     _    0    root         _    _
6    change     change     VERB     VB     _    5    xcomp        _    _
7    it         it         PRON     PRP    _    6    obj          _    _
8    for        for        ADP      IN     _    9    case         _    _
9    outside    outside    NOUN     NN     _    6    obl          _    _
10   ?          ?          PUNCT    .      _    5    punct        _    _

NER tags appear in the MISC column as NER=B-ORG, NER=I-ORG, etc.

Annotation Details

NER Entity Types (18)

Type	Example
PERSON	Dr. Smith, Caroline
ORG	Apple, United Nations
GPE	France, New York
LOC	Mount Everest, Pacific Ocean
DATE	January 2024, last quarter
TIME	3pm, two hours
MONEY	$394B, 50 million euros
PERCENT	15.3%, a third
QUANTITY	100 kilometers, 5 pounds
ORDINAL	first, 3rd
CARDINAL	three, 42
NORP	British, Republican
FAC	the White House, Highway 101
PRODUCT	iPhone, Boeing 747
EVENT	World War II, Olympics
WORKOFART	Hamlet, The Starry Night
LAW	the First Amendment
LANGUAGE	English, Mandarin

Dialog Act Labels (9)

These labels are designed for uniko's cognitive memory system:

Label	Description	Uniko Action	Gold Distribution
`inform`	States a fact or observation	Extract observation	73.0%
`question`	Asks for information	Record knowledge gap	6.3%
`plan_commit`	Commits to action, offers, suggests	Link to Goal/Task	5.3%
`request`	Asks someone to do something	Create action node	4.5%
`filler`	Fragments, headers, structural text	Skip entirely	4.4%
`correction`	Corrects or contradicts prior statement	Flag for update	3.2%
`social`	Greeting, goodbye, thanks, apology	Skip extraction	1.3%
`agreement`	Agrees with or confirms something	Reinforce observation	1.2%
`feedback`	Acknowledges without adding info	Skip extraction	0.8%

Context-dependent labels (correction, agreement, feedback) use prev_text for classification in the conversation domain. For non-conversational text, these labels are assigned based on sentence content alone.

Dependency Parsing (dep2label)

Dependencies are encoded as token-level labels using the rel-pos scheme (Strzyz et al., 2019):

+1@nsubj@VERB   ->  "1st VERB to the right is the head, relation = nsubj"
-2@amod@NOUN    ->  "2nd NOUN to the left is the head, relation = amod"
 0@root@ROOT    ->  "this token is the root"

This reformulation enables dependency parsing as standard token classification (~1,411 unique labels). Decoding back to a tree is O(n) per sentence.

Full vs Gold: Which Should I Use?

Use Case	Dataset
Model training	`corpus/gold/` or `prepared/`
Corpus linguistics / analysis	`data/` (full corpus)
Building custom subsamples	`corpus/gold/`
Evaluation	`prepared/*/ner_dev.json` etc.

The gold corpus is a strict subset of the full corpus. Every sentence in gold passed three independent LLM validation checks (NER, CLS, POS). The prepared data is a domain-balanced subsample of the gold corpus, ready for model training with label vocabularies.

Training

bash

git clone https://github.com/rustic-ai/kniv-nlp-models
cd kniv-nlp-models

# Prepare training data from gold corpus
python models/deberta-v3-large-nlp-en/prepare_data.py \
    --gold corpus/output/final-gold

# Train teacher model (DeBERTa-v3-large, 435M params)
python models/deberta-v3-large-nlp-en/train.py

# Quick test (1 epoch, 100 samples)
python models/deberta-v3-large-nlp-en/train.py --quick-test

Pipeline

collect  ->  preprocess  ->  annotate   ->  validate   ->  classify   ->  export  ->  gold filter
(17 src)     (dedup,         (spaCy         (GPT-5.4       (GPT-5.4       (CoNLL-U    (Qwen3-8B
              split,          trf)           nano)          nano)           Parquet)    vLLM)
              prev_text)
657K sentences -----------------------------------------------------------> 237K gold

Source

—Code: rustic-ai/kniv-nlp-models
—Models: dragonscale-ai on HuggingFace

Citation

bibtex

@misc{kniv-corpus-en-2026,
  title={kniv-corpus-en: Multi-domain English NLP Corpus},
  author={Dragonscale AI},
  year={2026},
  publisher={Hugging Face},
  url={https://huggingface.co/datasets/dragonscale-ai/kniv-corpus-en}
}

License

CC BY-SA 4.0. All source datasets are commercially licensed (CC BY, CC BY-SA, Apache 2.0, MIT, PSF, public domain, or ODC-BY).