NbAiLab/NPSC_orto_morphcoded_clean

NPSC Ortho Morphcoded Clean

This is a consensus-filtered derivative of NbAiLab/NPSC_orto_morphcoded at revision 'c0a5864ffde32ca05a683652b54282ee785ca16d'. It retains the original id, source, and target schema and the original source text.

Cleaning method

A T5Gemma 2 1B model fine-tuned on the source dataset generated one prediction for every row. Exact model/target agreements were retained. Every disagreement was shown to two isolated language-model reviewers with the source plus randomized candidates A and B; neither reviewer saw candidate provenance or the other review. Each reviewer could select only A, B, or None.

A disagreement was retained only when both reviewers independently selected the same non-None candidate after candidate randomization was decoded. Every other disagreement was dropped. A unanimous candidate was also dropped if it failed AltMorph syntax or exact source-recoverability checks. Selected targets were canonicalized to case-insensitive alphabetical alternative order.

The automated reviewers are a quality-control heuristic, not human linguistic annotation. Agreement between the generator, model, or reviewers does not prove that every retained alternative is authoritative or context-preserving.

Published splits

train is the cleaned original training split. eval and test preserve all original evaluation rows and labels for direct comparability. evalclean and testclean contain only rows retained by the consensus-cleaning procedure, with the selected target.

Split overlap

Split IDs are disjoint, but the upstream random split contains repeated rows. The following counts are measured against train after cleaning. Exact repeated source/target pairs can make evaluation optimistic and must not be interpreted as independent generalization.

Fields

Provenance

• —Generated: 2026-08-29T00:00:23.289993+00:00
• —Source dataset: 'NbAiLab/NPSCortomorphcoded'
• —Source revision: 'c0a5864ffde32ca05a683652b54282ee785ca16d'
• —Cleaning model: google/t5gemma-2-1b-1b, fine-tuned run t5gemma2-1b-altmorph-v2-alpha
• —Reviewers: two independently isolated automated language-model agents
• —Decision choices: A, B, or None
• —Retention rule for disagreements: unanimous A or unanimous B only

The source dataset card documents NPSC and AltMorph generation in detail, including HumIT, Ordbank, North-T5, source licensing, and attribution. This derivative uses CC BY-SA 3.0 conservatively and preserves those attribution requirements.

Please credit:

• —the National Library of Norway and NPSC dataset maintainers;
• —the University of Oslo HumIT team and Humit-Oslo/humit-tagger-large;
• —the University of Bergen Ordbank service;
• —the North-T5 authors and north/t5baseNCC;
• —the AltMorph authors; and
• —Google and the T5Gemma authors for the cleaning model.

The upstream parliamentary transcriptions are described as CC0, while the Hugging Face curation and this derivative use CC BY-SA 3.0.

Limitations

The dataset remains automatically generated silver data. In-sample predictions on the training split can reflect memorization, and shared model/target errors survive automatic agreement filtering. Agent judgments can also be correlated or linguistically wrong. Use eval/test for comparison with the original silver labels and evalclean/testclean for the stricter retained subset. Fresh human or linguist evaluation is recommended for consequential use.