RefSite3D: A Multi-Phase and Multi-Scenario 3D Point Cloud Dataset from the Reference Construction Site Aachen for Target Extraction, Scan-to-BIM Verification and Construction Monitoring

RefSite3D is a multi-phase and multi-scenario construction progress monitoring dataset created within the TARGET-X project and further processed within the Cluster of Excellence CARE and the CARE Future Technologies Experimental Construction Site. The dataset documents the assembly, disassembly, transport, and reassembly of the modular, multi-material TARGET-X ReStage demonstrator on the Reference Construction Site in Aachen and at the bauma 2025 trade fair. It contains: Planned 3D building models in OBJ format Synthetic point clouds generated from the planning models Real-world point cloud scans from multiple sensing technologies Target-extracted subsets Object verification labels Semantic segmentation labels Metadata for each scene Utility scripts and documentation The dataset is designed to support two sequential use cases: Target Extraction Construction Progress Monitoring via Object Verification The TARGET-X ReStage demonstrator, including the planning models, construction activities, and data acquisition, was developed within TARGET-X and funded by the Smart Networks and Services Joint Undertaking (SNS JU) under the European Union’s Horizon Europe research and innovation programme (Grant Agreement No. 101096614). The transformation of the acquired and simulated data into a structured benchmark dataset, including annotation generation, utility script development, and dataset documentation, was carried out within the Cluster of Excellence CARE. CARE - Climate-Neutral and Resource-Efficient Construction is funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany’s Excellence Strategy – EXC 3115 – 533767731. 1. TARGET-X ReStage Demonstrator The TARGET-X ReStage demonstrator was developed as part of the research project TARGET-X. The demonstrator represents a modular building system composed of: Precast concrete elements Timber elements Steel elements Additional modular units The assembly process was documented across multiple construction phases. During each phase, high-resolution 3D point cloud scans were captured using terrestrial laser scanning, photogrammetry, and NeRF-based reconstruction. The resulting data reflects realistic construction scenarios and enables the development and benchmarking of algorithms for digital construction and spatial AI. 1.1 Construction Phases Phase 1: Initial Assembly at the Reference Construction Site Aachen In Phase 1, the first building elements were assembled at the Reference Construction Site in Aachen. This phase contains the initial construction state of the demonstrator. Phase 2: bauma 2025 Extension Module In Phase 2, an additional modular unit was fabricated for exhibition at the bauma 2025 trade fair. The module was transported and assembled independently. Phase 3: Reuse Scenario In Phase 3, the bauma 2025 module was deconstructed, transported back to Aachen, and reconstructed as extension to the original demonstrator. This phase demonstrates: Reuse of building components Modular construction principles Circular construction workflows 1.2 Progress Monitoring States Several construction progress monitoring states were recorded. These states represent incremental construction stages. Progress State Description PM1 Concrete elements only PM2 Concrete + timber elements PM3 Concrete + timber + steel elements PM4 Standalone bauma 2025 extension module PM5 Complete demonstrator including bauma 2025 module 2. Benchmark Tasks Benchmark Task 1: Target Extraction The first task is to identify and extract the Object of Interest (OOI) from a full scene point cloud. The Object of Interest may represent: The TARGET-X ReStage demonstrator A machine An assembly of steel or timber components A storage area containing multiple construction components Accordingly, the dataset includes both construction progress monitoring scenes and additional target extraction benchmark scenes. These additional scenes were acquired between the progress monitoring states, when TARGET-X ReStage components were temporarily stored on the Reference Construction Site Aachen. They also include standalone scenes of steel assemblies, timber assemblies, and industrial machinery. Scan Acquisition Strategy All target extraction scenes were acquired using an object-oriented scanning strategy. In this approach, the scanner is moved closely around the predefined region of interest to obtain high point density and geometric completeness. This strategy ensures that all scenes are captured under consistent conditions and are directly comparable. Label Definition The target extraction task uses a three-class semantic labeling scheme: Label ID Class Name Description 0 Environment All points outside the region of interest and its supporting ground, such as surrounding structures, equipment, and vegetation 1 Ground The ground surface directly underlying the region of interest 2 Target All points belonging to the Object of Interest The target class may consist of a single object, multiple related objects, or a spatially distributed storage area. Only the relevant objects are labeled as target, while the floor or terrain beneath them is always labeled as ground. Applications Semantic segmentation Target extraction Object detection Construction object localization Building extraction from complex scenes Extraction of storage areas and component assemblies Benchmark Task 2: Progress Monitoring via Object Verification After extracting the target object, the extracted point cloud is registered with the corresponding planning model. Each planned building component is then compared to the scanned data to determine whether sufficient evidence exists to confirm its presence. Label Definition The object verification task uses a binary labeling scheme: Label ID Class Name Description 0 Object not verified The component could not be confirmed in the scan data 1 Object verified The component was successfully identified in the scan data These labels are stored in JSON files in the annotations/progress_tracking/ directories. Applications Automated construction progress monitoring Scan-to-BIM comparison Scan-vs-BIM verification Object-level installation status assessment Digital twin updating 3. Processing Workflow The dataset supports a two-stage workflow for automated construction progress monitoring. In the first stage, semantic segmentation is used to extract the relevant target structure from the full scene point cloud. In the second stage, the extracted target point cloud is aligned with the corresponding planning model and analyzed at the component level to determine which planned elements are present in the scan data. Full Scene Point Cloud ↓Semantic Target Extraction ↓Target Point Cloud ↓Registration with Planning Model ↓Component-Level Verification ↓Construction Progress Assessment Synthetic point clouds generated from the planning models serve as idealized reference geometries for the component-wise comparison. The additional benchmark scenes included in the dataset are intended exclusively for evaluating the semantic target extraction task. They provide annotated point clouds for diverse object categories, but do not include planning models or object verification labels. This separation allows the dataset to be used both as: a general benchmark for semantic target extraction, and a complete benchmark for Scan-to-BIM-based construction progress monitoring. 4. Planning Models and Synthetic Point Clouds The planning models were originally created in IFC4 format. They were converted into OBJ meshes for each construction phase. For every mesh, synthetic point clouds were generated. These synthetic point clouds provide idealized, noise-free representations of each building element and serve as references for object verification. Advantages Known geometry No sensor noise Complete coverage Direct component correspondence 5. Real Scan Data The dataset includes real-world point clouds generated using multiple sensing technologies. Acquisition Methods Terrestrial laser scanning (RIEGL VZ-400i, Leica RTC360 3D-Laserscanner) Image-based photogrammetry using Pix4D with images captured by an Apple iPhone X NeRF-based reconstruction using Luma AI with video captured by an Apple iPhone X For each scene The dataset contains: Full scene point cloud Building-extracted point cloud Scene metadata 6. Annotation Data The dataset contains annotation files for two benchmark tasks: Semantic target extraction Object-level verification for construction progress monitoring Semantic Segmentation Labels Semantic segmentation labels are stored as plain text files in the annotations/scene_segmentation/ directory. Each file contains one semantic class label per line. Each semantic segmentation label file corresponds to the full scene point cloud stored in the `scans/` directory (e.g., `scene_001_pm1_..._scene_... .ply`). The i-th line in the label file contains the semantic class label of the i-th point in the corresponding full scene point cloud. The labels are therefore defined for the complete scene, including the target structure, the supporting ground surface, and all surrounding objects. The labels are not associated with the building-extracted point cloud. Label Definition Label ID Class Name Description 0 Environment All points outside the region of interest and its supporting ground, such as surrounding structures, equipment, vegetation, and temporary objects 1 Ground The ground surface directly underlying the region of interest 2 Target All points belonging to the Object of Interest Example TargetTargetTargetGroundGroundEnvironmentEnvironmentEnvironment This means: Points 1 to 3 belong to the target structure. Points 4 to 5 belong to the ground. Remaining points belong to the environment. Object Verification Labels Object verification annotations are stored as JSON files in the annotations/progress_tracking/ directory. Each JSON file contains a list of planned components and their binary verification labels. Each entry consists of: source_file: path to the scanned building point cloud. reference_file: path to the synthetic point cloud of a planned component. ground_truth_class_object_verification: binary ground-truth label. Label Definition Label ID Class Name Description 0 Object not verified The component could not be confirmed in the scan data 1 Object verified The component was successfully identified in the scan data Example Structure { "scene_007": [ { "source_file": "phases/phase_03/scans/scene_007/...building_extracted...ply", "reference_file": "phases/phase_03/synthetic_pointclouds/components/...IfcBeam...ply", "ground_truth_class_object_verification": 1 } ]} The uploaded example scene_007_object_verification_labels.json follows exactly this structure. Label Map The file label_map.json defines the mapping between numeric label IDs and class names. This file ensures consistent label interpretation across all phases and scenes. Annotation Workflow The annotation process follows two consecutive steps: Semantic Segmentation Separate the target structure from the complete scene. Assign each point to environment, ground, or target. Object Verification Compare each planned component with the registered scan data. Determine whether the component is verified (1) or not verified (0). This two-stage workflow mirrors the practical inspection process on construction sites: first isolate the building of interest, then inspect each planned component individually. 7. Dataset Contents The dataset contains the following data modalities. Data Type Format Description Planning models OBJ Phase-specific building meshes Synthetic point clouds PLY Noise-free point clouds from meshes Real point clouds PLY Captured scene and target object point clouds Metadata JSON Acquisition and processing information Object verification labels JSON Component status labels Semantic segmentation labels TXT Per-point labels Utility scripts Python Visualization and validation tools Documentation Markdown Technical documentation 8. Dataset Distribution and Folder Structure The dataset is distributed as a compressed ZIP archive named: RefSite3D_v1.0.0.zip After extraction, the archive contains the complete dataset/ directory shown below. dataset/├── README.md├── LICENSE├── label_map.json├── folder_structure.md├── dataset_description.json├── CITATION.cff├── utils/│ ├── outputs/├── target_extraction_benchmark/│ ├── 01_timber_components/│ ├── 02_steel_components/│ └── 03_construction_machines/├── phases/│ ├── phase_01/│ ├── phase_02/│ └── phase_03/└── docs/ phases/ This directory contains the construction progress monitoring benchmark. Each phase includes: models/Phase-specific planning models in OBJ format. synthetic_pointclouds/Synthetic point clouds generated from the planning models. scans/Real-world point clouds, including full-scene and target-extracted scans. annotations/Semantic segmentation labels and object verification labels. target_extraction_benchmark/ This directory contains additional benchmark scenes for evaluating semantic target extraction independently of the construction progress monitoring workflow. Each scenario includes: scans/Full scene point clouds acquired using the object-oriented scanning strategy. annotations/Semantic segmentation labels using the classes environment, ground, and target. utils/ Python scripts for preprocessing, visualization, and annotation quality control. All outputs generated by these scripts are written to: utils/outputs/ docs/ Supplementary documentation, including annotation guidelines, coordinate conventions, and file format specifications. File Description coordinate_system.md Coordinate conventions file_format_specification.md File format details annotation_guidelines.md Labeling instructions changelog.md Dataset version history Important Note The dataset structure and file names should not be modified. Several utility scripts automatically derive the locations of related files, such as annotations and planning models, from the predefined folder hierarchy. Renaming or moving files may therefore prevent the scripts from functioning correctly. This structure separates the generic target extraction benchmark from the full construction progress monitoring benchmark while maintaining a consistent organization across all scenarios. 9. Utility Scripts The utils/ directory contains standalone Python GUI applications for visualization, preprocessing, and quality control of the dataset annotations. Current scripts include: ply_semantic_segmentation_class_splitter_and_colorizer_gui.py ply_object_knn_extractor_gui.py ply_object_verification_class_colorizer_gui.py These tools support: Semantic target extraction and annotation validation Object-level point cloud extraction Visualization of object verification labels All generated files are written to the utils/outputs/ directory. Each script creates its own output subdirectory, typically named after the processed scene or input file. The original dataset files are never modified. Important Folder Structure Requirement The original dataset folder structure and file names should not be modified. The utility scripts automatically infer the locations of related files, such as annotations, planning models, and synthetic point clouds, from the predefined dataset hierarchy. Renaming or moving files may therefore prevent the scripts from functioning correctly. Typical Workflow The utility scripts support the following processing sequence: Full Scene Point Cloud↓Semantic Segmentation Splitter↓Target Point Cloud↓Registration with Planning Model↓Component-Level Verification↓Construction Progress Assessment Together, these tools support annotation validation, debugging, and reproducible benchmark preparation. Companion GitHub Repository The latest version of the utility scripts, including installation instructions, detailed input and output descriptions, and usage examples, is available at: Construction3D-utils 10. Potential Research Applications RefSite3D can be used for: Target extraction from construction scenes Scan-to-BIM registration Construction progress monitoring Semantic segmentation Change detection Digital twin updating Circular construction research Sensor fusion benchmarking Domain adaptation between synthetic and real data