Experiment 4: Best A3 Transfer to Dataset 3 — Expert Annotations, Refined Training Data, Model Checkpoints, and Evaluation Results

# Experiment 4: Best A3 Transfer to Dataset 3 ## 1. Overview This Zenodo repository provides research materials associated with **Experiment 4: Transfer of the Best A3 Multi-Source Annotation-Refinement Strategy to Dataset 3**, conducted as part of the study: **Distribution-Aware Dataset Design and Multi-Source Pseudo-Label Refinement for Tree Species Detection in Dense Forest Aerial Imagery** The experiment investigates the transferability of the selected **Best A3 (A3-native, native)** annotation-refinement pathway to a distribution-aware, image-disjoint Dataset 3 constructed using a 60/20/20 training, validation, and test partitioning strategy based on Mixed-Integer Linear Programming (MILP). The central objective is to determine whether multi-source pseudo-label refinement improves four-class tree-species detection in dense forest aerial RGB imagery and whether incorporating additional pseudo-labeled images into the final detector-training set provides further benefits. The experimental framework employs **Faster R-CNN with a ResNet-50 Feature Pyramid Network (FPN)** as the downstream detection architecture. ## 2. Dataset Description The dataset is derived from high-resolution SWISSIMAGE aerial RGB imagery of Swiss forests, with an approximate ground sampling distance of 10 cm. The labeled image collection contains 4,438 aerial image patches and 10,128 expert-provided bounding-box annotations covering four tree species: - **C1:** Norway spruce (*Picea abies*)- **C2:** Silver fir (*Abies alba*)- **C3:** Scots pine (*Pinus sylvestris*)- **C4:** European beech (*Fagus sylvatica*) Dataset 3 follows a 60/20/20 image-disjoint partitioning scheme: | Subset | Number of Images ||---|---:|| Training | 2,663 || Validation | 888 || Test | 887 || **Total** | **4,438** | The experiment uses expert-provided annotations as the reference labels. Accepted model-generated pseudo-labels are incorporated into the training data, while the validation and test annotations remain unchanged. An additional collection of 7,059 unlabeled aerial RGB image patches was available for the upstream semi-supervised annotation-refinement process. ## 3. Best A3 Multi-Source Annotation Refinement The Best A3 strategy was selected from the multi-source annotation-refinement configurations investigated in the preceding experiments. It combines predictions from three complementary annotation sources: 1. **RetinaNet**, used to generate additional candidate tree bounding boxes.2. **Supervised EfficientTree**, used as a complementary tree-detection and annotation source.3. **SSL-enhanced EfficientTree**, refined through a warm-start Teacher–Student semi-supervised learning framework using additional unlabeled imagery. The selected pathway uses the native-confidence annotation-selection strategy. During refinement, accepted model predictions are filtered to reduce redundancy and overlap with existing expert annotations. Expert-provided bounding boxes are preserved throughout the process. The resulting expanded annotation sets are used to train independently initialized Faster R-CNN models. ## 4. Experiment 4 Configurations Experiment 4 investigates two training-data configurations. ### Configuration A: Without Final Unlabeled-Image Inclusion The final Faster R-CNN training set consists of the original expert-annotated training images supplemented with accepted pseudo-labels produced through the Best A3 refinement pathway. Although additional unlabeled imagery is used during upstream Teacher–Student semi-supervised learning, pseudo-labeled images from this separate collection are not included in the final Faster R-CNN training set. The reported training composition is: - 2,663 training images- 6,078 original expert-provided bounding boxes- 7,450 additional accepted pseudo-labels- **13,528 total training bounding boxes** ### Configuration B: With Final Unlabeled-Image Inclusion This configuration uses the same upstream Best A3 refinement strategy but additionally incorporates accepted pseudo-labels generated from the separate unlabeled-image collection into final Faster R-CNN training. The reported training composition is: - 4,209 training images- 13,528 bounding boxes from the refined labeled training subset- 3,213 additional pseudo-labels from the unlabeled-image collection- **16,741 total training bounding boxes** Both configurations preserve the same expert-only validation and test partitions. The experiment therefore evaluates the effect of including additional pseudo-labeled images in final detector training while retaining the same downstream detection architecture and evaluation subsets. ## 5. Description of Uploaded Archives The repository contains the following archives. ### 5.1. Expert Annotation Archive **File:** `expert_60_20_20.rar` This archive is associated with the expert-annotation data for the Dataset 3 60/20/20 partitioning scheme. The expert annotations provide the reference labels for the experimental dataset and support the separation of training, validation, and test data. ### 5.2. Warm-Start Semi-Supervised Learning Checkpoint **File:** `checkpoint_D3_A0_ssl_warmstart_burn0_ep20.rar` This archive contains materials associated with a saved model checkpoint used in the Dataset 3 warm-start semi-supervised learning workflow. The checkpoint is associated with the initialization and refinement process employed in the Teacher–Student framework. ### 5.3. Supplementary Best A3 Files **File:** `best_a3_EXP4.rar` This supplementary archive is associated with the Best A3 configuration used in Experiment 4. It provides supporting materials related to the selected annotation-refinement configuration. The exact included files should be identified from the archive contents. ### 5.4. Complete Experiment 4 Archive **Files:** - `0000_best_A3-exp4.part1.rar`- `0000_best_A3-exp4.part2.rar`- `0000_best_A3-exp4.part3.rar`- `0000_best_A3-exp4.part4.rar`- `0000_best_A3-exp4.part5.rar`- `0000_best_A3-exp4.part6.rar` These six files form a multipart compressed archive associated with the Experiment 4 experimental directory and evaluation outputs. The experimental directory is organized around four evaluation configurations: - `exp4_dataset3_with_unlables-threshold=0.5`- `exp4_dataset3_with_unlables-threshold=0.25`- `exp4_dataset3_without_unlables-threshold=0.5`- `exp4_dataset3_without_unlables-threshold=0.25` Each configuration is organized into five independent experimental runs: - `run_01`- `run_02`- `run_03`- `run_04`- `run_05` The experiment directories also include aggregate evaluation files, such as: **`all_runs_summary.csv`** Provides a summary of the performance metrics recorded across the independent experimental runs. **`all_runs_mean_std.csv`** Provides aggregate performance statistics, including mean values and standard deviations across five runs. ### 5.5. Annotation CSV Files The Experiment 4 materials also include annotation files associated with the following roles: **`train_A3_native_native_without_unlabeled.csv`** Contains the refined training annotations generated by the transferred Best A3 pathway without including additional pseudo-labeled images in final detector training. **`train_A3_native_native_WITH_unlabeled.csv`** Contains the expanded training annotations for the configuration that additionally includes pseudo-labels generated from the separate unlabeled-image collection. **`val_expert_only.csv`** Contains expert-provided bounding-box annotations for the fixed validation subset. **`test_expert_only.csv`** Contains expert-provided bounding-box annotations for the fixed test subset. These files support examination of the training-data composition and the reference annotations used during evaluation. ## 6. Experimental Evaluation Protocol The downstream detector is Faster R-CNN, configured for four-class tree-species detection. Performance is evaluated using an object-detection matching Intersection over Union (IoU) threshold of 0.50. Two prediction confidence-score thresholds are examined: - **0.25**- **0.50** The evaluation metrics include: - Precision- Recall- Macro F1-score- Accuracy Five independent experimental runs are conducted for each configuration, and aggregate performance is reported using the mean and sample standard deviation. The validation and test sets remain fixed across the compared Experiment 4 configurations and contain only expert-provided annotations. ## 7. Summary of Reported Experimental Results The following results are reported in the associated manuscript for Experiment 4. ### Evaluation at Confidence Threshold 0.50 | Training Configuration | Precision | Recall | Macro F1-score | Accuracy ||---|---:|---:|---:|---:|| Expert-only baseline | 0.975 | 0.594 | 0.726 | 0.577 || Best A3 without final unlabeled inclusion | 0.959 | 0.720 | 0.812 | 0.675 || Best A3 with final unlabeled inclusion | 0.967 | 0.726 | 0.820 | 0.683 | ### Evaluation at Confidence Threshold 0.25 | Training Configuration | Precision | Recall | Macro F1-score | Accuracy ||---|---:|---:|---:|---:|| Best A3 without final unlabeled inclusion | 0.949 | 0.795 | 0.861 | 0.769 || Best A3 with final unlabeled inclusion | 0.955 | 0.786 | 0.858 | 0.760 | The values represent five-run mean performance. The corresponding standard deviations and individual-run results are documented in the associated experimental records. Both transferred Best A3 configurations improve performance relative to the expert-only baseline at the common evaluation threshold of 0.50. At threshold 0.25, excluding additional pseudo-labeled images from the final training set produces a slightly higher mean F1-score, whereas including them produces slightly higher Precision. The additional benefit of final unlabeled-image inclusion is small and depends on the evaluation threshold. The paired analyses reported in the manuscript do not establish a consistently positive advantage for final inclusion. ## 8. Reproducibility and Data Integrity These materials are provided to support reproducibility, independent evaluation, and verification of the Experiment 4 annotation-refinement and detection results. Several methodological principles are maintained throughout the experiment: - Expert-provided annotations are preserved during training-data refinement.- Pseudo-labels are incorporated only into the applicable training configurations.- Validation and test annotations remain expert-only.- The compared configurations use fixed validation and test partitions.- The same Faster R-CNN architecture is used for downstream evaluation.- Multiple independent runs are reported to characterize performance variability. Because the original expert-provided labels are incomplete, evaluation metrics should be interpreted relative to the available expert annotations rather than as exhaustive measures of all visible trees. The results are specific to the evaluated Swiss aerial RGB imagery and do not independently establish geographic generalization to other forest regions. ## 9. Instructions for Extracting the Multipart Archive The `0000_best_A3-exp4` experiment archive is distributed across six compressed parts. To extract the archive: 1. Download all six files (`part1` through `part6`).2. Place all parts in the same directory.3. Use compatible archive-extraction software, such as WinRAR or 7-Zip.4. Initiate extraction from `0000_best_A3-exp4.part1.rar`. All six archive parts are required for successful extraction. ## 10. Intended Use The dataset files, model checkpoints, and experimental results are intended to support: - Reproducibility of Experiment 4.- Analysis of multi-source pseudo-label refinement.- Evaluation of alternative final training-data configurations.- Verification of annotation expansion and preservation of expert labels.- Further research into semi-supervised tree-species detection using high-resolution aerial imagery. **Associated study:** *Distribution-Aware Dataset Design and Multi-Source Pseudo-Label Refinement for Tree Species Detection in Dense Forest Aerial Imagery.* **Primary imagery source:** SWISSIMAGE, Swiss Federal Office of Topography (swisstopo). **Original annotated dataset:** Beloiu et al. (2023), https://doi.org/10.5281/zenodo.7528566