MWirelabs/nagamesebert

NagameseBERT

  

A Foundational BERT model for Nagamese Creole - A compact, efficient language model for a low resource Northeast Indian language.

Overview

NagameseBERT is a 7M parameter RoBERTa-style BERT model pre-trained on 42,552 Nagamese sentences. Despite being 15× smaller than multilingual models like mBERT (110M) and XLM-RoBERTa (125M), it achieves competitive performance on downstream NLP tasks while offering significant efficiency advantages.

Key Features:

• —Compact: 6.9M parameters (15× smaller than mBERT)
• —Efficient: Pre-trained in 35 minutes on single A40 GPU
• —Custom tokenizer: 8K BPE vocabulary optimized for Nagamese
• —Rigorous evaluation: Multi-seed testing (n=3) with reproducible results
• —Open: Model, code, and data splits publicly available

Performance

Multi-seed evaluation results (mean ± std, n=3):

Trade-off: 6-7 percentage points lower accuracy with 15× parameter reduction, enabling resource-constrained deployment.

Model Details

Architecture

• —Type: RoBERTa-style BERT (no token type embeddings)
• —Hidden size: 256
• —Layers: 6 transformer blocks
• —Attention heads: 4 per layer
• —Intermediate size: 1,024
• —Max sequence length: 64 tokens
• —Total parameters: 6,878,528

Tokenizer

• —Type: Byte-Pair Encoding (BPE)
• —Vocabulary size: 8,000 tokens
• —Special tokens: [PAD], [UNK], [CLS], [SEP], [MASK]
• —Normalization: NFD Unicode + accent stripping
• —Case: Preserved (for proper nouns and code-switched English)

Training Data

• —Corpus size: 42,552 Nagamese sentences
• —Average length: 11.82 tokens/sentence
• —Split: 90% train (38,296) / 10% validation (4,256)
• —Sources: Web, social media, community contributions (deduplicated)

Pre-training

• —Objective: Masked Language Modeling (15% masking)
• —Optimizer: AdamW (lr=5e-4, weight_decay=0.01)
• —Batch size: 64
• —Epochs: 50
• —Training time: ~35 minutes
• —Hardware: NVIDIA A40 (48GB)
• —Final validation loss: 2.79

Usage

Load Model and Tokenizer

from transformers import AutoTokenizer, AutoModel

model_name = "MWirelabs/nagamesebert"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name)

# Example usage
text = "Toi moi laga sathi hobo pare?"
inputs = tokenizer(text, return_tensors="pt")
outputs = model(**inputs)

Fine-tuning for Token Classification

from transformers import AutoModelForTokenClassification, TrainingArguments, Trainer

# Load model with classification head
model = AutoModelForTokenClassification.from_pretrained(
    "MWirelabs/nagamesebert",
    num_labels=num_labels
)

# Training arguments
training_args = TrainingArguments(
    output_dir="./results",
    num_train_epochs=100,
    per_device_train_batch_size=8,
    learning_rate=3e-5,
    weight_decay=0.01
)

# Train
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset
)
trainer.train()

Evaluation

Dataset

• —Source: NagaNLP Annotated Corpus
• —Total: 214 sentences
• —Split (seed=42): 171 train / 21 dev / 22 test (80/10/10)
• —POS tags: 13 Universal Dependencies tags
• —NER tags: 4 entity types (PER, LOC, ORG, MISC) in IOB2 format

Experimental Setup

• —Seeds: 42, 123, 456 (n=3 for variance estimation)
• —Batch size: 32
• —Learning rate: 3e-5
• —Epochs: 100
• —Optimization: AdamW with 100 warmup steps
• —Hardware: NVIDIA A40
• —Metrics: Token-level accuracy and macro-averaged F1

Data Leakage Statement: All splits created with fixed seed (42) with no sentence overlap between train/dev/test sets.

Limitations

• —Corpus size: 42K sentences is modest; expansion to 100K+ could improve performance
• —Evaluation scale: Small test set (22 sentences) limits statistical power
• —Task scope: Only evaluated on token classification; needs broader task assessment
• —Efficiency metrics: No quantitative inference benchmarks (latency, memory) yet provided
• —Data documentation: Complete data provenance and licenses to be formalized

Citation

If you use NagameseBERT in your research, please cite:

@misc{nagamesebert2025,
  title={Bootstrapping BERT for Nagamese: A Low-Resource Creole Language},
  author={MWire Labs},
  year={2025},
  url={https://huggingface.co/MWirelabs/nagamesebert}
}

Contact

MWire Labs Shillong, Meghalaya, India Website: MWire Labs

License

This model is released under Creative Commons Attribution 4.0 International (CC BY 4.0).

You are free to:

• —Share — copy and redistribute the material
• —Adapt — remix, transform, and build upon the material

Under the following terms:

• —Attribution — You must give appropriate credit to MWire Labs

Acknowledgments

We thank the Nagamese-speaking community for their contributions to corpus development and validation.