CoolFace
Apppublic

arikda/smart-hebrew-translator-pos-tagging

sourceHugging Facemitupdated 1y agoView on Hugging Face
0likes
App README

Smart Multilingual Translator

An intelligent translation application powered by Facebook's mBART-50 model, translating Hebrew text to three languages: English, Spanish, and French.

Key Features

  • —Translation: mBART-50 powered translation from Hebrew to English, Spanish, and French
  • —POS Tagging: Part-of-speech analysis using spaCy Hebrew model
  • —Dependency Parsing: Syntactic dependency tree visualization
  • —Lemmatization: Root form identification for Hebrew words
  • —Automatic Language Detection: Input language verification using langdetect
  • —Interactive Interface: User-friendly Gradio interface with RTL support
  • —Real-time Analysis: Simultaneous translation and linguistic analysis
  • —Built-in Examples: Sample texts for demonstration and testing

Models and Libraries Used

Translation Model: mBART-50

  • —Model: facebook/mbart-large-50-many-to-many-mmt
  • —Capabilities: 50 languages, many-to-many translation
  • —Advantages: High quality, direct translation without intermediate language

Linguistic Analysis: spaCy

  • —Model: hecorenews_sm (Hebrew)
  • —Features: POS tagging, dependency parsing, lemmatization
  • —Language Support: Hebrew with full RTL support

Language Detection: langdetect

  • —Purpose: Input language verification
  • —Accuracy: High accuracy for Hebrew text detection

Comparison with Other Models

MarianMT (Helsinki-NLP)
  • —Advantages: Fast, easy to use
  • —Disadvantages: Separate model for each language pair, limited to paired translation
Google Translate API
  • —Advantages: Excellent quality, wide support
  • —Disadvantages: Requires paid API key, dependent on internet connection
NLLB (No Language Left Behind)
  • —Advantages: Support for 200+ languages
  • —Disadvantages: Requires more computational resources

Local Installation and Running

System Requirements

Python 3.8+
CUDA (optional, for better performance)

Installing Dependencies

pip install -r requirements.txt
python -m spacy download he_core_news_sm

Running the Application

python app.py

The application will open at: http://localhost:7860

Project Structure

smart-translator/
├── app.py                 # Main Gradio application
├── requirements.txt       # Python dependencies
├── map_model_lang.json   # Language and model mapping
├── README.md             # This documentation
└── assets/
    └── screenshot.png    # Application screenshot

Technical Configuration

Language Codes in mBART Model

  • —Hebrew: he_IL
  • —English: en_XX
  • —Spanish: es_XX
  • —French: fr_XX

Translation Parameters

  • —Max Length: 512 tokens
  • —Num Beams: 5 (beam search)
  • —Early Stopping: True

Usage

  1. 1.Enter Hebrew text in the input text box
  2. 2.Click "Translate & Analyze" to process the text
  3. 3.View translation results in English, Spanish, and French
  4. 4.Analyze linguistic features:
  5. 5.POS Table: Detailed part-of-speech tags for each token
  6. 6.Dependency Tree: Visual representation of syntactic relationships
  7. 7.POS Statistics: Distribution of grammatical categories
  8. 8.Check status to ensure processing succeeded

Features Explanation

POS (Part-of-Speech) Tags
  • —Token: Original word from the text
  • —Lemma: Root form of the word
  • —POS: Grammatical category (NOUN, VERB, ADJ, etc.)
  • —Tag: Detailed morphological tag
  • —Description: Human-readable explanation
Dependency Parsing
  • —Visual tree showing grammatical relationships between words
  • —Arrows indicate syntactic dependencies
  • —Labels show relationship types (subject, object, modifier, etc.)

Advanced Usage Examples

  • —Simple sentence: "אני אוהב ספרים" (I love books)
  • —Complex sentence: "הילדים משחקים בגינה עם הכלב החדש שלהם" (The children are playing in the garden with their new dog)
  • —Questions: "איך אתה מרגיש היום?" (How are you feeling today?)

Usage Examples

  • —"שלום, איך אתה מרגיש היום?" (Hello, how are you feeling today?)
  • —"הטכנולוגיה משנה את העולם בצורה מהירה" (Technology is changing the world rapidly)
  • —"אני אוהב לקרוא ספרים ולצפות בסרטים" (I love reading books and watching movies)

Performance

  • —Translation Time: 3-8 seconds (depending on text length)
  • —POS Analysis Time: 1-2 seconds (depending on text complexity)
  • —Memory Required: approximately 3GB RAM (increased due to spaCy model)
  • —GPU Support: Yes (accelerates translation, POS analysis runs on CPU)

Troubleshooting Common Issues

"Model not loaded" Error Message

# Ensure all dependencies are installed
pip install -r requirements.txt
python -m spacy download he_core_news_sm

# Try downloading the models in advance
python -c "from transformers import pipeline; pipeline('translation', model='facebook/mbart-large-50-many-to-many-mmt')"
python -c "import spacy; spacy.load('he_core_news_sm')"

"spaCy Hebrew model not loaded"

# Install Hebrew model explicitly
python -m spacy download he_core_news_sm

# Verify installation
python -c "import spacy; nlp = spacy.load('he_core_news_sm'); print('Hebrew model loaded successfully!')"

Slow Performance

  • —Ensure GPU is available for translation
  • —Use CPU for spaCy analysis (recommended)
  • —Reduce max_length parameter for translation
  • —Use torch.float16 instead of torch.float32

Contributing to the Project

  1. 1.Fork the project
  2. 2.Create a new branch (git checkout -b feature/amazing-feature)
  3. 3.Commit changes (git commit -m 'Add amazing feature')
  4. 4.Push to branch (git push origin feature/amazing-feature)
  5. 5.Open Pull Request

License

This project is distributed under the MIT License. See LICENSE for more details.

Acknowledgments

  • —Hugging Face for the Transformers library and models
  • —Facebook AI for the mBART-50 model
  • —Gradio for the excellent user interface

Smart Multilingual Translator | Built with mBART-50