LocalDoc/language_detection

1---2language:3- ar4- az5- bg6- de7- el8- en9- es10- fr11- hi12- it13- ja14- nl15- pl16- pt17- ru18- sw19- th20- tr21- ur22- vi23- zh24license: cc-by-nc-4.025tags:26- language detect27pipeline_tag: text-classification28widget:29- text: "Əlqasım oğulları vorzakondu"30---31 32# Multilingual Language Detection Model33 34## Model Description35This repository contains a multilingual language detection model based on the XLM-RoBERTa base architecture. The model is capable of distinguishing between 21 different languages including Arabic, Azerbaijani, Bulgarian, German, Greek, English, Spanish, French, Hindi, Italian, Japanese, Dutch, Polish, Portuguese, Russian, Swahili, Thai, Turkish, Urdu, Vietnamese, and Chinese.36 37## How to Use38You can use this model directly with a pipeline for text classification, or you can use it with the `transformers` library for more custom usage, as shown in the example below.39 40### Quick Start41First, install the transformers library if you haven't already:42```bash43pip install transformers44```45 46```python47from transformers import AutoModelForSequenceClassification, XLMRobertaTokenizer48import torch49 50# Load tokenizer and model51tokenizer = XLMRobertaTokenizer.from_pretrained("LocalDoc/language_detection")52model = AutoModelForSequenceClassification.from_pretrained("LocalDoc/language_detection")53 54# Prepare text55text = "Əlqasım oğulları vorzakondu"56encoded_input = tokenizer(text, return_tensors='pt', truncation=True, max_length=512)57 58# Prediction59model.eval()60with torch.no_grad():61    outputs = model(**encoded_input)62 63# Process the outputs64logits = outputs.logits65probabilities = torch.nn.functional.softmax(logits, dim=-1)66predicted_class_index = probabilities.argmax().item()67labels = ["az", "ar", "bg", "de", "el", "en", "es", "fr", "hi", "it", "ja", "nl", "pl", "pt", "ru", "sw", "th", "tr", "ur", "vi", "zh"]68predicted_label = labels[predicted_class_index]69print(f"Predicted Language: {predicted_label}")70```71 72## Language Label Information73 74The model outputs a label for each prediction, corresponding to one of the languages listed below. Each label is associated with a specific language code as detailed in the following table:75 76| Label | Language Code | Language Name |77|-------|---------------|---------------|78| LABEL_0     | az            | Azerbaijani   |79| LABEL_1     | ar            | Arabic        |80| LABEL_2     | bg            | Bulgarian     |81| LABEL_3     | de            | German        |82| LABEL_4     | el            | Greek         |83| LABEL_5     | en            | English       |84| LABEL_6     | es            | Spanish       |85| LABEL_7     | fr            | French        |86| LABEL_8     | hi            | Hindi         |87| LABEL_9     | it            | Italian       |88| LABEL_10    | ja            | Japanese      |89| LABEL_11    | nl            | Dutch         |90| LABEL_12    | pl            | Polish        |91| LABEL_13    | pt            | Portuguese    |92| LABEL_14    | ru            | Russian       |93| LABEL_15    | sw            | Swahili       |94| LABEL_16    | th            | Thai          |95| LABEL_17    | tr            | Turkish       |96| LABEL_18    | ur            | Urdu          |97| LABEL_19    | vi            | Vietnamese    |98| LABEL_20    | zh            | Chinese       |99 100This mapping is utilized to decode the model's predictions into understandable language names, facilitating the interpretation of results for further processing or analysis.101 102 103Training Performance104 105The model was trained over three epochs, showing consistent improvement in accuracy and loss:106 107    Epoch 1: Training Loss: 0.0127, Validation Loss: 0.0174, Accuracy: 0.9966, F1 Score: 0.9966108    Epoch 2: Training Loss: 0.0149, Validation Loss: 0.0141, Accuracy: 0.9973, F1 Score: 0.9973109    Epoch 3: Training Loss: 0.0001, Validation Loss: 0.0109, Accuracy: 0.9984, F1 Score: 0.9984110 111Test Results112 113The model achieved the following results on the test set:114 115    Loss: 0.0133116    Accuracy: 0.9975117    F1 Score: 0.9975118    Precision: 0.9975119    Recall: 0.9975120    Evaluation Time: 17.5 seconds121    Samples per Second: 599.685122    Steps per Second: 9.424123 124 125License126 127The dataset is licensed under the Creative Commons Attribution-NonCommercial 4.0 International  license. This license allows you to freely share and redistribute the dataset with attribution to the source but prohibits commercial use and the creation of derivative works.128 129 130 131Contact information132 133If you have any questions or suggestions, please contact us at [v.resad.89@gmail.com].