Balogvn/swahili-ner-dataset

swahili-ner-dataset

Dataset Card

Dataset Name: swahili-ner-dataset Language: sw (Swahili) Number of Samples: 3 Model Used for Annotation: dslim/bert-base-NER Files Processed: 1 Texts Processed: 3 Processing Time: 4.01 seconds Generated: 2025-10-13 09:13:19 UTC

Description

This is an automatically annotated dataset for Swahili Named Entity Recognition (NER). The dataset was processed using the February AI Pipeline, which recursively discovers and processes JSON, JSONL, and TXT files from the data directory with performance optimizations for large datasets.

All dataset files are in JSONL format for maximum HuggingFace compatibility.

Processing Limits Applied

This dataset was processed with the following limits for performance optimization:

• —Max Files: 10
• —Max Texts per File: 50
• —Max Total Texts: 500
• —Max Text Length: 5000 characters
• —Batch Size: 20
• —Output Format: JSONL only (HuggingFace optimized)

Dataset Structure

The dataset contains annotations in standardized JSONL format with the following entity types:

• —PERSON - Names of people
• —ORGANIZATION - Companies, institutions, organizations
• —LOCATION - Places, cities, countries, geographical locations
• —MISCELLANEOUS - Other named entities

Processing Pipeline

1. 1.File Discovery: Recursively scans data/ directory for .json, .jsonl, and .txt files
2. 2.Smart Limiting: Applies configurable limits for files, texts, and processing time
3. 3.Batch Processing: Processes texts in batches to manage memory efficiently
4. 4.Text Extraction: Handles various formats with consistent JSONL output
5. 5.Auto-Annotation: Uses dslim/bert-base-NER for entity recognition
6. 6.Format Standardization: Converts all outputs to HuggingFace-compatible JSONL
7. 7.Validation: Validates all JSONL files for format correctness

Files

• —dataset.jsonl - Full dataset in JSONL format (HuggingFace compatible)
• —train.jsonl - Training split (80% of data)
• —test.jsonl - Test split (20% of data)
• —dataset_info.json - HuggingFace dataset configuration
• —label_studio_annotations.json - Original Label Studio format (reference only)
• —README.md - This documentation

Dataset Format

Each record in the JSONL files contains:

{
  "id": 0,
  "text": "Rais wa Tanzania Samia Suluhu Hassan amekutana na wabunge.",
  "tokens": ["Rais", "wa", "Tanzania", "Samia", "Suluhu", "Hassan", "amekutana", "na", "wabunge."],
  "labels": ["O", "O", "B-LOCATION", "B-PERSON", "I-PERSON", "I-PERSON", "O", "O", "O"],
  "ner_tags": ["O", "O", "B-LOCATION", "B-PERSON", "I-PERSON", "I-PERSON", "O", "O", "O"],
  "entities": [
    {
      "start": 11,
      "end": 19,
      "text": "Tanzania",
      "label": "LOCATION"
    },
    {
      "start": 20,
      "end": 38,
      "text": "Samia Suluhu Hassan", 
      "label": "PERSON"
    }
  ]
}

Label Format

• —BIO Tagging: B- (Beginning), I- (Inside), O (Outside)
• —Entity Types: PERSON, ORGANIZATION, LOCATION, MISCELLANEOUS

Usage with HuggingFace

from datasets import load_dataset

# Load the dataset
dataset = load_dataset('json', data_files={
    'train': 'train.jsonl',
    'test': 'test.jsonl'
})

# Or load from HuggingFace Hub (after upload)
# dataset = load_dataset('your-username/swahili-ner-dataset')

# Access the data
print(dataset['train'][0])

Manual Usage

import json

# Load JSONL file
dataset = []
with open('dataset.jsonl', 'r', encoding='utf-8') as f:
    for line in f:
        dataset.append(json.loads(line))

# Process records
for record in dataset:
    print(f"Text: {record['text']}")
    print(f"Tokens: {record['tokens']}")  
    print(f"Labels: {record['labels']}")
    print("---")

Citation

If you use this dataset in your research, please cite:

@dataset{swahili_ner_dataset_20251013_091319,
  title={swahili-ner-dataset},
  author={February AI Pipeline},
  year={2025},
  language={sw},
  samples={3},
  annotation_model={dslim/bert-base-NER},
  processing_time={4.01},
  format={JSONL}
}

License

This dataset is provided as-is for research and educational purposes.

Generated automatically by February AI Pipeline on 2025-10-13 09:13:19 UTC Processing completed in 4.01 seconds All files in JSONL format for HuggingFace compatibility