sonson0910/engineering-drawing-detection

Engineering Drawing Object Detection and OCR System

Technical Assessment — Sotatek Computer Vision / AI Engineer

An end-to-end pipeline for detecting and extracting structured information from engineering drawings using a commercially-licensed detection model, advanced post-processing, and a full OCR stack.

🔗 Live Demo: huggingface.co/spaces/sonson0910/engineering-drawing-detection 📦 Model Weights: Download best_model.pth (Google Drive)

Table of Contents

• —Overview
• —Architecture
• —Key Techniques
• —Setup and Installation
• —Training
• —Inference
• —Web Demo
• —Technical Report
• —License

Overview

The system detects three classes of objects in engineering drawings:

Architecture

Input Image
  → [TTA] Multi-scale + flip augmentation (6 views)
  → Faster R-CNN ResNet-50 FPN v2  (inference on each view)
  → Weighted Boxes Fusion (WBF)    (merge all view predictions)
  → Per-class NMS + size filtering
  → Content-aware box tightening
  → [Overlap Resolution] Priority-based arbitration
      Table > Note > PartDrawing
      Each overlap pixel classified by visual content:
        grid lines → Table | dense text → Note | sparse geometry → PartDrawing
  → Crop detected regions
      ├── PartDrawing → save crop image
      ├── Note        → PaddleOCR text extraction
      └── Table       → PaddleOCR PP-Structure (HTML table output)
  → JSON output + visualization

Framework Choice

⚠️ YOLO was explicitly excluded due to AGPL-3.0 licensing restrictions on commercial use.

Key Techniques

1. Test-Time Augmentation (TTA) with WBF

At inference time, the model runs on 6 augmented views of each input image:

• —3 scales: 0.75×, 1.0×, 1.25×
• —Each scale: original + horizontal flip

All 6 sets of predictions are fused via Weighted Boxes Fusion (WBF) — a state-of-the-art ensemble method that averages box coordinates weighted by confidence scores, rather than simply suppressing low-confidence duplicates like standard NMS.

This delivers ~5–10% higher recall with no retraining, at ~6× inference cost.

2. Overlap Resolution

Per the assessment requirement:

"There may be cases where bounding boxes overlap. You will need to properly separate these overlapping regions and determine which specific bounding box each area belongs to."

The overlap_resolver.py module implements a 3-tier arbitration system:

1. 1.Class priority rule: Table > Note > PartDrawing Engineering drawings are structured such that when two objects overlap:
2. 2.A Table almost always takes precedence (it has defined borders)
3. 3.A Note annotation takes precedence over a general drawing area
4. 4.Confidence score tiebreaker: higher confidence wins when priority is equal
5. 5.Visual content analysis: each overlap pixel-patch is analyzed:
6. 6.Grid/line structure → Table
7. 7.Dense horizontal text rows → Note
8. 8.Sparse geometric content → PartDrawing

The resolution runs iteratively until zero overlapping pairs remain. Every resolved overlap is logged in the JSON output under overlap_resolution.details[] for full transparency.

3. Training Enhancements

4. Post-Processing Pipeline

Raw model predictions
  → Min threshold filter (class-specific)
  → Per-class NMS (standard for PartDrawing/Note; prefer-larger variant for Table)
  → Size filter (min_w × min_h × area per class)
  → Content-aware box tightening (morphological trimming of white margins)
  → Iterative overlap resolution (see above)

Setup and Installation

Prerequisites

• —Python 3.10+
• —PyTorch 2.0+ (CUDA-capable GPU recommended, 8GB+ VRAM)

Installation

git clone https://github.com/<your-username>/cv-assessment.git
cd cv-assessment

python -m venv venv
# Windows:
venv\Scripts\activate
# Linux/macOS:
source venv/bin/activate

pip install -r requirements.txt

Dataset

COCO format, 58 images, 412 annotations across 3 classes:

datasets/
└── BOM-Folder- BOM-Dataset.coco/
    └── train/
        ├── _annotations.coco.json
        └── *.png

Training

python src/detection/train.py

Key configuration (config/train_config.yaml):

Achieved Results

Inference

Single image

python src/detection/inference.py \
    --model models/best_map_model_backup.pth \
    --input path/to/image.jpg \
    --output outputs/

Batch directory

python src/pipeline/pipeline.py \
    --model models/best_map_model_backup.pth \
    --input path/to/images/ \
    --output outputs/

Output structure

outputs/
├── crops/
│   ├── PartDrawing/
│   ├── Note/
│   └── Table/
├── visualizations/
└── json/

JSON schema

{
  "image": "drawing_001.jpg",
  "objects": [
    {
      "id": 1,
      "class": "Table",
      "confidence": 0.97,
      "bbox": { "x1": 120, "y1": 340, "x2": 680, "y2": 520 },
      "ocr_content": {
        "type": "table",
        "rows": [["Header1", "Header2"], ["Data1", "Data2"]],
        "html": "<table>...</table>",
        "raw_text": "Header1 | Header2\nData1 | Data2"
      }
    }
  ],
  "overlap_resolution": {
    "total_overlaps_detected": 1,
    "details": [
      {
        "box_a": { "class": "PartDrawing", "score": 0.91 },
        "box_b": { "class": "Table", "score": 0.88 },
        "overlap_iou": 0.23,
        "winner_class": "Table",
        "method": "class_priority"
      }
    ]
  },
  "pipeline_info": {
    "tta_enabled": true,
    "tta_views": "3 scales × 2 flips → WBF",
    "model": "Faster R-CNN ResNet-50 FPN v2 (Apache 2.0)",
    "postprocess": "per-class NMS + priority-based overlap arbitration"
  }
}

Web Demo

python src/web/app.py
# Access at http://localhost:7860

Features:

• —Upload drawing → instant detection visualization
• —🔬 TTA toggle — enable/disable multi-scale inference on the fly
• —Confidence threshold slider
• —📋 JSON Output tab — full structured result
• —📝 OCR Results tab — extracted text from Notes and Tables
• —🔀 Overlap Resolution tab — resolution log for each detected overlap

Technical Report

Methodology

Detection strategy:

• —Faster R-CNN (two-stage) was chosen for its precision on complex engineering drawings. One-stage detectors (SSD, FCOS) tend to miss small annotations in dense layouts.
• —Per-class confidence thresholds account for the severe class imbalance (41 Note annotations vs. 268 PartDrawing).
• —Custom prefer-larger NMS for the Table class ensures footer title-blocks (low confidence, large area) are not suppressed by partial fragment detections (high confidence, small area).

OCR strategy:

• —PaddleOCR with angle classification handles rotated text in drawings.
• —PP-Structure produces HTML table output, preserving row/column alignment for BOM tables.

Experiments conducted

1. 1.Model comparison: Faster R-CNN vs. RetinaNet vs. SSD. Faster R-CNN achieved highest recall for small Notes.
2. 2.Augmentation ablation: Adding CLAHE + Perspective reduced validation loss by ~15%.
3. 3.Oversampling: 4× Note oversampling improved Note AP from ~0.61 → ~0.87.
4. 4.Copy-Paste: ~+4pp Note AP improvement by generating synthetic training varieties.
5. 5.TTA: +5–10% recall on held-out test images, particularly for small Notes near drawing edges.

Direction for future improvements

• —Instance Segmentation (Mask R-CNN): Non-rectangular drawing boundaries for cleaner OCR crops
• —Domain-specific fine-tuning of PP-Structure: Improve column alignment on faded engineering tables
• —Confidence calibration (Temperature Scaling): Calibrate raw model logits for better-calibrated confidence scores
• —RT-DETR ensemble: Combine Transformer-based detector predictions as additional TTA view

License

Detection components: torchvision BSD-3 — commercially compatible. OCR components: PaddleOCR Apache 2.0 — commercially compatible. No AGPL/GPL components used.