JVal123/DMDBench

DMDBench

DMDBench is a benchmark of 1,000 real-world images of digital measurement devices (DMDs). It is designed to evaluate visual question answering (VQA) and optical character recognition (OCR) under practical capture conditions, including variations in scale, viewpoint, illumination, background clutter, reflections, and motion blur.

The benchmark accompanies CAD2DMD-SET: Synthetic Dataset Generator of Digital Measurement Devices.

Device categories

DMDBench contains images from six device categories:

Repository contents

The repository contains:

• —the 1,000 DMDBench images;
• —VQA question–answer annotations generated using CAD2DMD-SET;
• —OCR annotations containing quadrilateral text regions and their transcriptions; and
• —this README describing the annotation and label-generation procedures.

The VQA dataset table contains the following fields:

Annotation protocol

All 1,000 images were annotated by one author. For each image, the author recorded the structured ground-truth information required to describe the device display. Depending on the device and its operating configuration, this information included:

• —the device type;
• —the operating mode or measurement type;
• —the displayed numerical value or values; and
• —the corresponding measurement unit or units.

Numerical readings were transcribed exactly as represented by the device. The following elements were preserved:

• —negative signs;
• —leading zeros;
• —decimal separators; and
• —the displayed number of decimal places.

When a display contained multiple relevant elements, each measurement, unit, or operating-mode indicator was recorded separately.

For images captured under visually challenging conditions, such as motion blur, reflections, or difficult illumination, the transcription was verified against the known device reading recorded during image acquisition. This reference-based check was used to confirm the ground-truth annotation.

Because the annotations were prepared by one author, annotation quality was assessed through reference-based verification rather than an inter-annotator agreement statistic.

VQA annotation generation

The VQA question–answer annotations were generated automatically using the label-generation functionality of CAD2DMD-SET.

The manually recorded structured ground-truth information for each image was provided to CAD2DMD-SET, which generated the corresponding questions and answers following the same procedure used to create VQA labels for the synthetic CAD2DMD-SET training data.

Accordingly, the semantic ground truth was derived from the manually recorded device information, while the construction and formatting of the VQA question–answer pairs were automated. This workflow ensured consistent question and answer structures throughout the benchmark.

Questions may address one or more aspects of the visible device, including:

• —device identity;
• —operating mode or measurement type;
• —displayed numerical readings; and
• —measurement units.

OCR annotations

The OCR annotations are distributed in gt_detections. The archive contains one text file for each image:

gt_detections/
├── img1.txt
├── img2.txt
├── ...
└── img1000.txt

Each imgN.txt file corresponds to the image with index N. Across the 1,000 images, the files contain 2,754 annotated text regions.

Each line represents one OCR annotation and follows this comma-separated format:

x1,y1,x2,y2,x3,y3,x4,y4,transcription

The first eight values are the image-pixel coordinates of the four vertices defining the quadrilateral text region. The final field is the ground-truth transcription associated with that region.

Example:

564,743,674,730,674,806,570,818,54
449,751,452,777,519,771,518,744,TEMPO

In this example, the image contains two annotated regions: the displayed value 54 and the operating-mode label TEMPO.

The OCR annotations include visible:

• —numerical measurements;
• —measurement units; and
• —operating-mode or configuration indicators.

This unified format provides both the spatial information required for text detection and the corresponding transcription required for text recognition.

Coordinate convention

Coordinates are expressed in pixels relative to the original image dimensions. The four coordinate pairs are stored in the order produced by the annotation export and should be treated as the vertices of a quadrilateral rather than as an axis-aligned bounding box.

Intended use

DMDBench is intended for research on:

• —VQA for digital measurement devices;
• —scene-text detection and recognition;
• —OCR and multimodal-model benchmarking;
• —robustness evaluation under realistic image degradation; and
• —numerical and unit extraction from digital displays.

Data considerations

The images contain digital measurement devices collected for this study. No personally identifying information is included.

For medical, electrical, industrial, or other safety-sensitive applications, model predictions should be confirmed through suitable human or system-level verification procedures.