Datasetpublic

VytautoDidziojoUniversitetas/NUS-LT-PII-corpus

NUS Lithuanian PII Corpus Description Lithuanian text annotated for personal information (PII), spanning three subject domains — administrative, scientific, and media — plus a stratified validation set. The corpus covers 24 entity types: 16 general categories (PER, LOC, ORG, …) and 8 GDPR special-category "sensitive" entities (REL, POL, SEX, GENDER, MAR, FAM, ETH, HEALTH). Dataset Summary Subsets: 4 (3 training categories + 1 validation set) Total… See the full description on the dataset page: https://huggingface.co/datasets/VytautoDidziojoUniversitetas/NUS-LT-PII-corpus.

sourceHugging Faceopenrailupdated 5mo agoView on Hugging Face

0likes97downloads

Dataset Card

NUS Lithuanian PII Corpus

Description

Lithuanian text annotated for personal information (PII), spanning three subject domains — administrative, scientific, and media — plus a stratified validation set. The corpus covers 24 entity types: 16 general categories (PER, LOC, ORG, …) and 8 GDPR special-category "sensitive" entities (REL, POL, SEX, GENDER, MAR, FAM, ETH, HEALTH).

Dataset Summary

—Subsets: 4 (3 training categories + 1 validation set)
—Total records: 41,312
—Total sentences: 520,904
—Total words: 13,762,478
—Total entity annotations: 1,487,589

Subset	Records	Sentences	Words	Entities
mokslinis	270	39,343	1,108,602	63,649
ziniasklaida	2,451	55,394	1,107,917	106,912
administraciniai	38,468	423,559	11,487,490	1,309,705
validacija	123	2,608	58,469	7,323
Total	41,312	520,904	13,762,478	1,487,589

Words = re.findall(r"\w+", text). Sentences = ./!/? followed by whitespace + uppercase, ignoring Lithuanian abbreviations (dr., prof., pvz., …).

Main Columns

json

{
  "id": "corpus identifier (e.g. LT-mokslinis-0001)",
  "category": "domain subset the record belongs to: mokslinis, ziniasklaida, or administraciniai",
  "text": "full document text",
  "label": [
    {
      "start": "character offset where the entity begins (int)",
      "end": "character offset where the entity ends, exclusive (int)",
      "label": "entity type (one of the 24 listed below)"
    }
  ],
  "meta": {
    "original_id": "identifier from the upstream source",
    "url": "original publication URL",
    "title": "title of the source document",
    "date": "publication date (ISO 8601 or year)",
    "source_name": "publication, database, or domain name"
  }
}

Output Formats

Each subset is provided in three NER-friendly formats:

—`data.jsonl` — primary format. One record per line with span annotations (character offsets). Used by the YAML configs block above for load_dataset.
—`data.conll` — CoNLL-2003 BIO format. Whitespace-tokenised tokens with B-LABEL / I-LABEL / O tags, blank line between documents. Standard for seqeval-style evaluation and many NER toolkits.
—`data.xml` — TEI P5 with inline <rs type="LABEL"> markup wrapping each entity span inside <TEI><text><body>…</body></text></TEI>.

Additionally per subset:

—metadata.xml — Dublin Core metadata
—stats.json — per-subset statistics (records, words, sentences, entities by label, sources)

Usage

python

from datasets import load_dataset

# Full corpus (train + test splits, where test = validacija)
ds = load_dataset("VytautoDidziojoUniversitetas/NUS-LT-PII-corpus")

# Per-domain subsets
ds = load_dataset("VytautoDidziojoUniversitetas/NUS-LT-PII-corpus", "mokslinis")
ds = load_dataset("VytautoDidziojoUniversitetas/NUS-LT-PII-corpus", "ziniasklaida")
ds = load_dataset("VytautoDidziojoUniversitetas/NUS-LT-PII-corpus", "administraciniai")

# Validation set only
ds = load_dataset("VytautoDidziojoUniversitetas/NUS-LT-PII-corpus", "validacija")

Entity Distribution

24 entity types in two groups: general (16, ≥95% of annotations) and sensitive / GDPR special-category (8, 2–5% of annotations).

Label	Meaning
`PER`	Person names
`LOC`	Locations and addresses
`ORG`	Organizations
`ID_PER`	Personal ID codes (Lithuanian 11-digit national `asmens kodas`)
`ID_MISC`	Other identifiers (passport, document numbers, etc.)
`NUM_PHONE`	Phone numbers
`NUM_CAR`	Vehicle license plates
`MISC`	Email addresses and miscellaneous identifiers
`OCC`	Occupations
`EDU`	Education
`AGE`	Age
`DATE`	Dates
`TIME`	Times
`DURATION`	Durations
`VALUE`	Numeric amounts and quantities
`NAT`	Nationality
`HEALTH`	Health information (diagnoses, conditions, treatments)
`REL`	Religion
`POL`	Political views
`ETH`	Ethnic origin
`MAR`	Marital status (single, married, divorced, etc.)
`FAM`	Family relations (mentions of relatives)
`GENDER`	Gender
`SEX`	Sexual orientation

Counts of general vs sensitive entities per subset:

Subset	General	Sensitive	Sensitive %	Total
mokslinis	59,882	3,767	5.92	63,649
ziniasklaida	103,677	3,235	3.03	106,912
administraciniai	1,274,645	35,060	2.68	1,309,705
validacija	7,172	151	2.06	7,323
Total	1,445,376	42,213	2.84	1,487,589

Per-label counts are reported in each subset's stats.json.

Sources

All texts are in Lithuanian. The corpus is divided into three subject-matter categories plus a stratified validation set.

Category	Description	Sources	Source files	Date range	Synthetic
mokslinis	Scientific publications	5	VDU CRIS (Lituanistika repository), open-access Vilnius University Press journals	2000–2025	–
ziniasklaida	News and media articles	1	LRT.lt (2019–2020 archive and 2024–2025 portal scrape)	2019–2025	–
administraciniai	Administrative and government documents	111	Lithuanian municipality and government websites	2001–2025	1,480 (3.8%) Gemma 2 27B
validacija	Mixed-domain held-out validation set	7	Filtered subset of the above	mixed	–

All texts are originally human-written, except for the synthetic administraciniai records flagged above (identifiable by original_id prefix translated_synthetic_admin_texts_*).

Licensing

This dataset is distributed under the NewGenLTU OpenRAIL-D licence.

This licence was specifically created by a team of researchers at Vytautas Magnus University for the publication of datasets produced in the course of research and development projects funded by the New Generation Lithuania plan. The licence aims to support open and responsible downstream use while ensuring compliance with applicable requirements related to copyright, personal data protection, and the EU AI Act.

Please see the accompanying LICENSE.txt file in this repository for the full licence text.

Intended Use

The corpus is intended for:

—Training Lithuanian NER models for personal-information detection
—Automatic de-identification of GDPR-relevant data in Lithuanian text
—Evaluation of cross-domain NER performance (admin / scientific / media)
—Research on rare-entity detection (sensitive GDPR categories)

Restrictions

Under the NewGenLTU OpenRAIL-D licence, users must not use the dataset for:

—discrimination,
—the development of weapons or military applications,
—automated decision-making affecting people,
—disinformation,
—privacy infringement,
—medical or health advice,
—creation of malware,
—harassment,
—dishonest research,
—collection, extraction, or reconstruction of personal data.

Limitations and Bias

—Subset proportions: ~84% of words come from administraciniai; the other categories are smaller and from fewer sources. Training on the full corpus produces a model tilted toward administrative language.
—Cross-convention evaluation: the corpus uses contextual intent to resolve cases that other NER conventions handle differently — e.g. Kauno savivaldybė is labeled ORG when it acts as the institution and LOC when it refers to the place; "2024 metais" is DATE when it marks a point in time and DURATION when it marks a span. When comparing against conventions that draw these lines differently, consider scoring related labels jointly.
—Class imbalance: the 8 sensitive entities (REL, POL, SEX, GENDER, MAR, FAM, ETH, HEALTH) intentionally make up 2–5% of annotations to reflect their expected real-world prevalence in GDPR-relevant text. Two general labels are also rare in the source domains: NUM_CAR (130 instances) and ID_PER (236 instances). Per-class evaluation on small classes has high variance.

Citation

Please cite the dataset as:

Vytautas Magnus University and Kaunas University of Technology. 2026. NUS Lithuanian PII Corpus. Hugging Face. https://huggingface.co/datasets/VytautoDidziojoUniversitetas/NUS-LT-PII-corpus

BibTeX:

bibtex

@dataset{nus_lt_pii_2026,
  author    = {{Vytautas Magnus University} and {Kaunas University of Technology}},
  title     = {NUS Lithuanian PII Corpus},
  year      = {2026},
  publisher = {Hugging Face},
  url       = {https://huggingface.co/datasets/VytautoDidziojoUniversitetas/NUS-LT-PII-corpus},
  note      = {Developed under the New Generation Lithuania plan, project 02-100-K-0001.}
}

Acknowledgement

Funded by the European Union — NextGenerationEU. Naujos kartos Lietuva.

This project is funded by the Recovery and Resilience Facility under the Lithuanian "Naujos kartos Lietuva" plan and by the State Budget of the Republic of Lithuania.

Project: Nuasmeninimo tekstyno sukūrimas (NUS) — Project No. 02-100-K-0001.