tlabdev/ner-irish-dataset

1# NER IRISH Dataset2 3A manually annotated Indonesian news corpus for Named Entity Recognition (NER), created using Label Studio. This dataset provides **raw, character-level annotations** to ensure maximum flexibility across different tokenization strategies and preprocessing pipelines.4 5## Dataset Summary6 7- **Language:** Indonesian8- **Task:** Named Entity Recognition (NER)9- **Annotation Type:** Character-level spans (start/end offsets)10- **Format:** Raw text with entity annotations11- **Total Samples:** 190912- **Annotation Tool:** Label Studio13 14## Quick Start15```python16from datasets import load_dataset17 18# Load the dataset19dataset = load_dataset("tlabdev/ner-irish-dataset")20 21# View dataset structure22print(dataset)23 24# Access splits25train_data = dataset['train']26val_data = dataset['validation']27test_data = dataset['test']28```29 30## Entity Types31 32The dataset includes 9 entity types commonly found in news articles:33 34| Label | Description | Examples |35|-------|-------------|----------|36| **PER** | Person names | Joko Widodo, Megawati |37| **ORG** | Organizations, institutions, companies | Bank Indonesia, UNESCO |38| **GPE** | Geopolitical entities (administrative regions) | Jakarta, Indonesia, Jawa Barat |39| **LOC** | Natural locations (non-administrative) | Sungai Citarum, Gunung Merapi |40| **FAC** | Facilities and infrastructure | Bandara Soekarno-Hatta, Monas |41| **LAW** | Laws, regulations, legislative acts | UU No. 20 Tahun 2003 |42| **EVENT** | Named events | Pilkada 2024, Asian Games |43| **MONEY** | Monetary values | Rp 10 miliar, $500 |44| **DATE** | Dates and time periods | 17 Agustus 1945, Januari 2024 |45 46## Dataset Structure47 48### Data Fields49 50Each sample contains:51```python52{53    "text": str,          # Full article text54    "entities": [         # List of entity annotations55        {56            "text": str,  # Entity mention57            "label": str, # Entity type (PER, ORG, etc.)58            "start": int, # Character start position59            "end": int    # Character end position60        }61```62 63### Data Splits64 65The dataset is split at the **article level** to prevent data leakage:66 67| Split | Percentage | Description |68|-------|------------|-------------|69| Train | 80% | Training set |70| Validation | 10% | Development/validation set |71| Test | 10% | Held-out test set |72 73Each split maintains article boundaries—no article is split across multiple sets.74 75### Example76```python77{78    "text": "Presiden Joko Widodo meresmikan Bandara Internasional Yogyakarta di Kulon Progo.",79    "entities": [80        {"text": "Joko Widodo", "label": "PER", "start": 9, "end": 20},81        {"text": "Bandara Internasional Yogyakarta", "label": "FAC", "start": 33, "end": 66},82        {"text": "Kulon Progo", "label": "GPE", "start": 70, "end": 81}83    ]84}85```86 87## Data Processing88 89The dataset has undergone careful cleaning while preserving annotation integrity:90 91### Applied Processing92- Deduplication by URL93- Deduplication by (title, publishedAt) pairs94- Content-based deduplication using hashing95- Removal of duplicate entity spans within articles96- Safe character normalization that **preserves text length** and span offsets97 98### Intentionally NOT Applied99This dataset is provided in raw form and does **not** include:100- Tokenization or token IDs101- Subword segmentation (WordPiece, BPE, etc.)102- BIO/IOB tagging scheme103- Sliding window chunking104- Class balancing or oversampling105- Gazetteer augmentation106- Label smoothing107 108These preprocessing steps should be implemented downstream based on your specific model architecture and training requirements.109 110## Use Cases111 112**Suitable for:**113- Fine-tuning transformer models (BERT, XLM-RoBERTa, etc.) for Indonesian NER114- Benchmarking multilingual NER systems115- Entity-based information extraction pipelines116- Gazetteer and knowledge base construction117- Studying class imbalance in NER tasks118- Cross-lingual NER transfer learning119 120**Limitations:**121- Focused on formal news text (may not generalize to informal/social media text)122- Entity types limited to 9 common categories123- Some entity types may have class imbalance124 125## Important Notes126 127### Character Offset Preservation128 129Entity annotations use **character-level spans** (start/end positions). Any text modifications that change string length will invalidate these offsets.130 131**Safe operations:**132- Case normalization (if done consistently)133- Whitespace trimming at document boundaries134 135**Unsafe operations (will break offsets):**136- Removing characters137- Inserting characters138- Unicode normalization that changes byte length139- Aggressive text cleaning140 141If you need to modify the text, ensure you update entity offsets accordingly or perform modifications before entity extraction.142 143## Citation144 145If you use this dataset in your research, please cite:146```bibtex147@dataset{indonesian_news_ner,148  author = {TLab Developer and Muhammad Faiz Khoiri},149  title = {NER IRISH Dataset},150  year = {2025},151  publisher = {Hugging Face},152  url = {https://huggingface.co/datasets/tlabdev/ner-irish-dataset}153}154```155 156## Acknowledgments157 158- Annotated using [Label Studio](https://labelstud.io/)159- News articles sourced from various Indonesian news portals