Benchmark Dataset for VLM Symbol Recognition in Engineering Blueprints

The BlueprintSymVL benchmark dataset for evaluating Vision Language Model (VLM) performance on symbol recognition in engineering drawings, which is published as part the paper "BlueprintSymVL: A Discriminative Benchmark for VLM Symbol Recognition in Engineering Blueprints".The dataset was derived from the Digitize-PID dataset originally published by Shubham Paliwal, Arushi Jain, Monika Sharma, and Lovekesh Vig, as part of the paper "Digitize-PID: Automatic Digitization of Piping and Instrumentation Diagrams".The original dataset was modified for the purposes of evaluating the visual in-context learning capabilities of VLMs for symbol recognition in engineering diagrams. The changes include tiling, selection of appropriate examples, and the creation of new annotations.The dataset is organized as follows:

`Regular/`: 100 clean blueprint regions for the primary evaluation.

`Noisy/`: 100 occluded versions of the regions in `Regular/` to test robustness.

`Example Symbols/`: 5 standard visual examples (2 annotated instances each).

`Alternate Example Symbols/`: 5 alternate visual examples (1 annotated instance each).

`Symbol crops/`: 5 cropped symbol images for an ablation study.

The `Regular/` and `Noisy/` directories are organized by challenge scenario (e.g., `dense_regions`) and symbol class (e.g., `gate_valve`). Each folder contains 5 evaluation cases, each consisting of three files:

`image.jpg`: An 860x860 pixel image of the blueprint region.

`image_q.txt`: A text file with the question for the VLM (e.g., "How many gate symbols are in this image?").

 `image_a.txt`: The ground truth answer.

*Line 1*: The correct symbol count.

*Subsequent Lines*: The text labels of each symbol instance.

For questions, please contact Vasil Shteriyanov at [e-mail].

This dataset is available under the Creative Commons Attribution 4.0 International (CC BY 4.0) license.