RyhanSunny/AI_Multi-lingual_Document_File_Summarizer_Demo

Text Summarization App by TEAM 2 ================================

This application provides automated text summarization and translation functionalities, allowing users to extract and summarize content from PDF, DOCX, and text files. It supports translations between English and French, utilizing state-of-the-art models. Team Members:

Sajib (Ryhan) Suny Sharmi Das Amaka Genevieve Jane Awa Matthew Kolawole Oluwadamilare  Noah Nkalubo Nsimbe

Install requirements --------------------

To set up the application, install the necessary Python libraries using the following command:

pip install -r requirements.txt

Run the app -----------

Start the application by running:

python app.py

Components and Libraries ------------------------

• —Gradio: Used to create the web interface for the application.
• —Transformers: Provides the models for summarization and translation.
• —Torch, torchvision, torchaudio: PyTorch libraries used by the Transformers for neural network operations.
• —PyPDF: Used for extracting text from PDF files.
• —python-docx: Used for extracting text from DOCX files.
• —Langdetect: Detects the language of the input text.
• —SentencePiece, Sacremoses: Used by Transformers for text preprocessing and tokenization.

Major Classes and Functions ---------------------------

• —TextExtractor: This class handles the extraction of text from different file types (PDF, DOCX, TXT). It preprocesses text to remove URLs and image tags and normalizes whitespace.
• —Summarization and Translation Pipelines: Utilizes models from the Transformers library to perform text summarization and translation between English and French.

Process Flow ------------

• —Text Extraction: The TextExtractor class takes a document file, identifies its format, and extracts clean text.
• —Summarization: Summarization is performed using a predefined model (facebook/bart-large-cnn). If the initial summarization fails due to length constraints, manual tokenization and summarization are performed.
• —Translation: If requested, the summary is translated into either French or English using translation models (Helsinki-NLP/opus-mt-en-fr and Helsinki-NLP/opus-mt-fr-en).
• —Gradio Interface: Provides an interface where users can upload documents and choose the output language for the summary. It displays the summarized and potentially translated text.

Deployment ----------

The application is wrapped with Gradio's Interface class, which creates an easy-to-use web interface for the summarization functionality. The application supports a variety of document formats including PDF, DOCX, and TXT.

UML Components --------------

• —Classes: TextExtractor, possibly a class for handling translation and summarization logic.
• —Interfaces: Gradio interface components (inputs and outputs).
• —Processes: Text extraction, text summarization, text translation, error handling.

facebook/bart-large-cnn for text summarization. Helsinki-NLP/opus-mt-en-fr for translation from English to French. Helsinki-NLP/opus-mt-fr-en for translation from French to English.