Apppublic
OniiOniiChan/LogisticsDataExtractionValidationSystem

sourceHugging Faceupdated 7mo agoView on Hugging Face
0likes
README.md292 linesDownload Raw Back to root
1---2title: Logistics Data Extraction3emoji: 📦4colorFrom: blue5colorTo: indigo6sdk: docker7app_file: app.py8pinned: false9---10 11# Logistics Data Extraction & Validation System12 13A comprehensive web application for intelligent extraction and validation of logistics data from Vietnamese customs documents. Features advanced AI-powered OCR, cross-document validation, database storage, and professional reporting capabilities.14 15## Features16 17- **AI-Powered Data Extraction**: Uses Google Gemini AI for intelligent document analysis18- **Multi-Format Support**: PDF, JPG, PNG, GIF, WEBP19- **Advanced Validation**: Cross-document consistency checking with severity-based flagging (CRITICAL, ERROR, WARNING, INFO)20- **Database Storage**: Supabase PostgreSQL for session history and validation records21- **Professional Reporting**: Session tracking with comparison tables and quality metrics22- **Docker Support**: Ready for deployment with Docker and Docker Compose23 24## Quick Start25 26### 1. Environment Setup27 281. **Create Supabase Project**:29   - Go to [supabase.com](https://supabase.com) and create a new project30   - Note your Project URL and anon/public key31 322. **Set Environment Variables**:33   Create a `.env` file with your credentials:34   ```35   SUPABASE_URL=https://your-project.supabase.co36   SUPABASE_KEY=your-anon-key-here37   GEMINI_API_KEY=your-google-gemini-api-key38   ```39 403. **Create Database Tables**:41   - Open Supabase Dashboard → SQL Editor42   - Run the SQL from `backend/database/schema.sql`43 44### 2. Installation45 46```bash47# Create virtual environment48python -m venv venv49 50# Activate virtual environment51venv\Scripts\activate  # Windows52# or53source venv/bin/activate  # Linux/Mac54 55# Install dependencies56pip install -r requirements.txt57```58 59### 3. Run Application60 61```bash62python run.py63```64 65Open browser to: http://127.0.0.1:500066 67## Architecture Overview68 69```70┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐71│   Web Frontend  │    │   Flask Backend  │    │   Supabase DB   │72│   (HTML/JS)     │◄──►│   API Routes      │◄──►│   PostgreSQL    │73└─────────────────┘    └──────────────────┘    └─────────────────┘74                              │75                              ▼76                       ┌─────────────────┐77                       │   AI Services   │78                       │ • Google Gemini │79                       │ • OCR Processing│80                       │ • Validation    │81                       └─────────────────┘82```83 84## Core Components85 86### Data Extraction Pipeline87 881. **Document Upload**: Multi-format support (PDF, images)892. **OCR Processing**: EasyOCR for Vietnamese/English text extraction903. **AI Analysis**: Google Gemini extracts structured logistics data914. **Validation Layer**: Cross-document consistency checks925. **Database Storage**: Session history and validation records93 94### Validation Features95 96- **Severity Levels**: CRITICAL, ERROR, WARNING, INFO97- **Cross-Document Checks**: BL vs Invoice, weight consistency, container validation98- **Smart Flagging**: Automatic issue detection with recommendations99- **Audit Trail**: Complete session history with timestamps100 101### Supported Document Types102 103- Customs Declarations (Tờ khai hải quan)104- Bills of Lading (Vận đơn)105- Commercial Invoices (Hóa đơn thương mại)106- Packing Lists (Danh sách đóng gói)107 108### Extracted Fields109 110| Field | Description |111|-------|-------------|112| `doc_type` | Document type (Invoice/PL/BL/Customs) |113| `bl_no` | Bill of Lading number |114| `invoice_no` | Invoice number |115| `shipper` | Exporter name |116| `consignee` | Importer name |117| `vessel` | Vessel name |118| `containers` | List of containers |119| `total_weight` | Total weight (KG) |120| `total_packages` | Total packages |121| `hs_code` | HS code |122| `hs_code_suggestions` | Suggested HS codes |123 124## API Endpoints125 126| Method | Endpoint | Description |127|--------|----------|-------------|128| POST | `/api/extract` | Extract data from single document |129| POST | `/api/extract-all` | Extract and validate multiple documents |130| GET | `/api/session/{session_id}` | Get session history |131| GET | `/api/sessions` | List recent sessions (paginated) |132| GET | `/api/statistics` | System statistics |133| GET | `/api/fields` | List extraction fields |134| GET | `/api/document-types` | Available document types |135| GET | `/api/config` | Current configuration |136| GET | `/api/save-comparison` | Save user comparison data |137| GET | `/api/qc-report/{session_id}` | Generate QC report |138 139### POST /api/extract140 141Extract data from a single document.142 143```bash144curl -X POST http://localhost:5000/api/extract \145  -F "file=@document.pdf"146```147 148### POST /api/extract-all149 150Extract and validate multiple documents simultaneously.151 152```bash153curl -X POST http://localhost:5000/api/extract-all \154  -F "file=@invoice.pdf" \155  -F "file=@packing_list.pdf" \156  -F "file=@bill_of_lading.pdf"157```158 159Response:160```json161{162  "success": true,163  "session_id": "...",164  "extracted_documents": [...],165  "validation": {166    "issues": [...],167    "summary": {168      "total_issues": 0,169      "critical": 0,170      "errors": 0,171      "warnings": 0,172      "info": 0173    }174  }175}176```177 178## Project Structure179 180```181KeywordExtractionFromPDF/182├── README.md                      # This file183├── API_DOCUMENTATION.md           # Detailed API reference184├── CHANGELOG.md                   # Version history185├── requirements.txt               # Python dependencies186├── run.py                         # Application entry point187├── Procfile                       # Heroku deployment188├── Dockerfile                     # Docker configuration189├── runtime.txt                    # Python version190├── backend/191│   ├── __init__.py192│   ├── app.py                     # Flask application193│   ├── config.py                  # Configuration settings194│   ├── database/195│   │   ├── manager.py             # Supabase database operations196│   │   └── schema.sql             # Database schema197│   ├── extractor/198│   │   ├── __init__.py199│   │   ├── ai_extractor.py        # Google Gemini integration200│   │   ├── base.py                # BaseExtractor class201│   │   ├── customs_extractor.py   # Legacy customs extractor202│   │   ├── ocr_processor.py       # OCR processing203│   │   ├── pdf_processor.py       # PDF text extraction204│   │   └── text_extractor.py      # Text extraction205│   ├── routes/206│   │   ├── __init__.py207│   │   ├── api.py                 # API endpoints208│   │   └── web.py                 # Web routes209│   ├── services/210│   │   ├── __init__.py211│   │   ├── advanced_validator.py   # Cross-document validation212│   │   ├── report_generator.py    # Report generation213│   │   └── validator.py           # Legacy validator214│   └── utils/215│       ├── __init__.py216│       └── helpers.py217├── frontend/218│   ├── index.html                 # Main HTML219│   ├── css/220│   │   └── styles.css            # Styles221│   └── js/222│       ├── config.js              # Configuration223│       ├── api.js                 # API functions224│       ├── ui.js                  # UI rendering225│       └── app.js                 # Main entry226├── pdf_examples/                   # Sample documents227└── venv/                           # Virtual environment228```229 230## Configuration231 232### Environment Variables233 234| Variable | Description | Required |235|----------|-------------|----------|236| `SUPABASE_URL` | Supabase project URL | Yes |237| `SUPABASE_KEY` | Supabase anon key | Yes |238| `GEMINI_API_KEY` | Google Gemini API key | Yes |239| `FLASK_ENV` | Flask environment | No |240| `FLASK_DEBUG` | Enable debug mode | No |241 242## Troubleshooting243 244### Common Issues245 2461. **Supabase Connection Error**:247   - Verify `SUPABASE_URL` and `SUPABASE_KEY` in `.env`248   - Check Supabase project is active249 2502. **AI Extraction Fails**:251   - Verify `GEMINI_API_KEY` is valid252   - Check API rate limits253 2543. **PDF Processing Issues**:255   - Ensure PDF is not password protected256   - Check file is not corrupted257 258### Logs259 260Application logs are available in the terminal when running. Set `FLASK_DEBUG=true` for detailed logging.261 262## Docker Deployment263 264```bash265# Build Docker image266docker build -t logistics-extraction .267 268# Run container269docker run -p 5000:5000 --env-file .env logistics-extraction270```271 272## Technology Stack273 274- **Backend**: Flask 3.0.0 (Python)275- **AI**: Google Gemini API276- **OCR**: EasyOCR277- **PDF Processing**: PyMuPDF278- **Database**: Supabase PostgreSQL279- **Frontend**: HTML, CSS, Vanilla JavaScript280 281## Contributing282 2831. Fork the repository2842. Create a feature branch2853. Make changes with proper validation2864. Test with sample documents2875. Submit pull request288 289## License290 291MIT License292