tlabdev/ner-irish-dataset

NER IRISH Dataset

A manually annotated Indonesian news corpus for Named Entity Recognition (NER), created using Label Studio. This dataset provides raw, character-level annotations to ensure maximum flexibility across different tokenization strategies and preprocessing pipelines.

Dataset Summary

• —Language: Indonesian
• —Task: Named Entity Recognition (NER)
• —Annotation Type: Character-level spans (start/end offsets)
• —Format: Raw text with entity annotations
• —Total Samples: 1909
• —Annotation Tool: Label Studio

Quick Start

from datasets import load_dataset

# Load the dataset
dataset = load_dataset("tlabdev/ner-irish-dataset")

# View dataset structure
print(dataset)

# Access splits
train_data = dataset['train']
val_data = dataset['validation']
test_data = dataset['test']

Entity Types

The dataset includes 9 entity types commonly found in news articles:

Dataset Structure

Data Fields

Each sample contains:

{
    "text": str,          # Full article text
    "entities": [         # List of entity annotations
        {
            "text": str,  # Entity mention
            "label": str, # Entity type (PER, ORG, etc.)
            "start": int, # Character start position
            "end": int    # Character end position
        }

Data Splits

The dataset is split at the article level to prevent data leakage:

Each split maintains article boundaries—no article is split across multiple sets.

Example

{
    "text": "Presiden Joko Widodo meresmikan Bandara Internasional Yogyakarta di Kulon Progo.",
    "entities": [
        {"text": "Joko Widodo", "label": "PER", "start": 9, "end": 20},
        {"text": "Bandara Internasional Yogyakarta", "label": "FAC", "start": 33, "end": 66},
        {"text": "Kulon Progo", "label": "GPE", "start": 70, "end": 81}
    ]
}

Data Processing

The dataset has undergone careful cleaning while preserving annotation integrity:

Applied Processing

• —Deduplication by URL
• —Deduplication by (title, publishedAt) pairs
• —Content-based deduplication using hashing
• —Removal of duplicate entity spans within articles
• —Safe character normalization that preserves text length and span offsets

Intentionally NOT Applied

This dataset is provided in raw form and does not include:

• —Tokenization or token IDs
• —Subword segmentation (WordPiece, BPE, etc.)
• —BIO/IOB tagging scheme
• —Sliding window chunking
• —Class balancing or oversampling
• —Gazetteer augmentation
• —Label smoothing

These preprocessing steps should be implemented downstream based on your specific model architecture and training requirements.

Use Cases

Suitable for:

• —Fine-tuning transformer models (BERT, XLM-RoBERTa, etc.) for Indonesian NER
• —Benchmarking multilingual NER systems
• —Entity-based information extraction pipelines
• —Gazetteer and knowledge base construction
• —Studying class imbalance in NER tasks
• —Cross-lingual NER transfer learning

Limitations:

• —Focused on formal news text (may not generalize to informal/social media text)
• —Entity types limited to 9 common categories
• —Some entity types may have class imbalance

Important Notes

Character Offset Preservation

Entity annotations use character-level spans (start/end positions). Any text modifications that change string length will invalidate these offsets.

Safe operations:

• —Case normalization (if done consistently)
• —Whitespace trimming at document boundaries

Unsafe operations (will break offsets):

• —Removing characters
• —Inserting characters
• —Unicode normalization that changes byte length
• —Aggressive text cleaning

If you need to modify the text, ensure you update entity offsets accordingly or perform modifications before entity extraction.

Citation

If you use this dataset in your research, please cite:

@dataset{indonesian_news_ner,
  author = {TLab Developer and Muhammad Faiz Khoiri},
  title = {NER IRISH Dataset},
  year = {2025},
  publisher = {Hugging Face},
  url = {https://huggingface.co/datasets/tlabdev/ner-irish-dataset}
}

Acknowledgments

• —Annotated using Label Studio
• —News articles sourced from various Indonesian news portals