goodpixelltd/distilbert-mlm-multilingual-coreml

DistilBERT MLM Multilingual - CoreML (Fast)

This is a CoreML conversion of distilbert-base-multilingual-cased with the Masked Language Model (MLM) head, optimized for iOS deployment.

Why DistilBERT?

DistilBERT is a smaller, faster version of BERT that retains 97% of BERT's language understanding while being:

• —40% smaller (~258MB vs ~340MB)
• —60% faster inference
• —Same 104 language support

Model Description

• —Base Model: distilbert-base-multilingual-cased
• —Task: Masked Language Modeling (MLM)
• —Languages: 104 languages including English, Chinese, Cantonese
• —Format: CoreML (.mlmodelc)
• —Size: ~258MB compiled

Intended Use

Grammar correction that preserves code-switching (mixed language text). Ideal for mobile keyboards where speed is important.

Comparison with BERT

Model Files

• —vocab.txt - WordPiece vocabulary (119,547 tokens)
• —distilbert_mlm.mlmodelc/ - Compiled CoreML model for iOS

Technical Details

• —Architecture: DistilBERT (6 layers, 768 hidden, 12 attention heads)
• —Parameters: ~66M (vs BERT's 110M)
• —Max Sequence Length: 128 tokens
• —Compute Units: CPU (for iOS background app compatibility)
• —Minimum iOS: 15.0

Usage in iOS

import CoreML

// Load model
let config = MLModelConfiguration()
config.computeUnits = .cpuOnly
let model = try MLModel(contentsOf: modelURL, configuration: config)

// Prepare inputs (DistilBERT doesn't use token_type_ids)
let inputIds: MLMultiArray = // tokenized input with [MASK] tokens
let attentionMask: MLMultiArray = // attention mask

// Run inference
let input = try MLDictionaryFeatureProvider(dictionary: [
    "input_ids": MLFeatureValue(multiArray: inputIds),
    "attention_mask": MLFeatureValue(multiArray: attentionMask)
])
let output = try model.prediction(from: input)
let logits = output.featureValue(for: "logits")?.multiArrayValue

License

This model is released under the Apache 2.0 License.

Attribution

• —Original Model: distilbert-base-multilingual-cased
• —DistilBERT: Created by Hugging Face
• —Conversion: CoreML conversion using coremltools
• —Converted by: goodpixelltd

Citation

@article{sanh2019distilbert,
  title={DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter},
  author={Sanh, Victor and Debut, Lysandre and Chaumond, Julien and Wolf, Thomas},
  journal={arXiv preprint arXiv:1910.01108},
  year={2019}
}