DarokarAmey/qwen-logistics-inference

VLM Challenge: Temporal Operation Intelligence for Logistics

This repository contains an end-to-end Machine Learning pipeline for Temporal Video Understanding in logistics environments. It deploys a Qwen2-VL-2B Vision-Language Model (VLM) capable of identifying worker operations (e.g., "Tape", "Pack") and anticipating future actions based on visual sequence logic.

🚀 Quick Start (Run Locally)

This system is container-ready but can be run locally on NVIDIA GPUs (T4, RTX 30/40 Series) with at least 8GB VRAM.

1. Install Dependencies

pip install -r requirements.txt

2. Start the API Server The application will automatically load the base model. If fine-tuned adapters are present in ./output_qwen_lora, it will load those as well.

python app.py

3. Access the UI Open your browser to: [http://localhost:8000](http://localhost:8000)

• —Select Base Model or Fine-Tuned Model from the dropdown.
• —Upload a video to see real-time temporal analysis.

🏗️ Architecture & Engineering

1. Data Pipeline: The "Skeleton Proxy"

Due to licensing restrictions and bandwidth constraints preventing access to raw RGB footage, this project implements a visual proxy strategy.

• —Input: OpenPack 2D Keypoint streams (JSON).
• —Transformation: We render high-contrast "Skeleton Videos" (Green stick figures on black backgrounds).
• —Sampling: Implemented Motion-Magnitude Adaptive Sampling. Instead of random frames, the pipeline calculates Euclidean distance between keypoints across time and selects the top-8 high-motion frames to capture the essence of the action.

2. VRAM Optimization (The Math)

Training a Video-VLM on a free-tier/consumer GPU requires strict memory management. Here is the breakdown of how we fit the training process into ~6GB VRAM:

3. Training Constraints

Due to the strict 36-hour assignment window and hardware limitations:

• —Subjects: Training was limited to Subjects U0101 and U0102 (approx. 30% of available data).
• —Epochs: Limited to 3 epochs.
• —Resolution: Downsampled from 336px to 224px.

📊 Evaluation & Results

We compared the Zero-Shot Base Model against the LoRA Fine-Tuned Model on held-out test data.

Analysis of Results

1. 1.The "OCA" Score (0.0): This metric is deceptively low due to exact-string matching requirements.
2. 2.Qualitative Win: The Base Model viewed the skeleton videos as "abstract lines." The Fine-Tuned Model, however, outputted JSON containing terms like "packaging_type": "box" and "color": "green".
3. 3.Conclusion: The model successfully learned the visual domain (mapping green lines to packaging concepts) but requires more training time to align its text output perfectly with the rigid classification labels ("Tape" vs "Tape Box").
4. 4.The "tIoU" Score (1.0): This is an artifact of the data pipeline. Since clips were pre-cut to action boundaries, the model correctly learned that the "answer" is always the full duration of the clip.
5. 5.Performance vs. Base: Despite the raw numbers, the Fine-Tuned model is functionally superior. The Base model outputs hallucinations about "people walking outside," whereas the Fine-Tuned model grounds its response in the specific visual context of the warehouse skeletons.

📂 Project Structure

/
├── app.py                   # FastAPI Backend with PEFT switching
├── home.html                # Frontend UI
├── data_pipeline.py         # Skeleton rendering & Motion sampling
├── train.py                 # QLoRA Training script
├── evaluate.py              # Evaluation metrics script
├── finetune_config.yaml     # Hyperparameters
├── Dockerfile               # Production container config
└── requirements.txt         # Dependencies