Benchmark Data for Tracking Progress in General-Purpose VLMs and Agents for Earth Observation

Public mirror of the IEEE DataPort dataset doi:10.21227/7zqe-3b78. The files are identical to the IEEE DataPort version (see SHA256SUMS.txt). This dataset contains the complete test set used in the paper "Tracking Progress in General-Purpose VLMs and Agents for Earth Observation" (S. D. Liu and S. Wang, submitted to IEEE Transactions on Geoscience and Remote Sensing): 13 Earth observation tasks with 7,273 test samples, which we used to evaluate 48 general-purpose vision-language models and OpenAI Agents API workflows. Each sample is stored exactly as it was sent to the models: the input image(s), the zero-shot prompt and system message, and the ground-truth answer with the source annotations. One ZIP archive per task contains annotations.jsonl, an images/ folder and a README. 00_README.txt documents all fields. Tasks (samples): Landmarks Recognition (61); DIOR-RSVG Object Classification (1,000); PatternNet Scene Recognition (1,000); RarePlanes Aircraft Counting (1,000); COWC Car Counting (1,000); xView2 Pre-disaster Building Counting and xView2 Change Detection (933 shared image pairs); NEON Tree Counting (194); Aerial Animal Counting (112); FPCD Farm Pond Change Detection (76); DIOR-RSVG Localization (1,000); DOTA Localization (445); FAIR1M Localization (452). Important: third-party data. None of the images or annotations in this dataset were collected or created by us. All of them come from existing public datasets made by other authors, mostly through the VLEO-Bench and GEOBench-VLM benchmarks. Our contribution is only the selection of the 7,273 test samples, the prompts and the packaging used in our paper. Copyright and licenses stay with the original authors listed below. Please cite the original papers and follow each dataset's own license or terms of use; where a dataset has its own license, that license governs its data here (for example, xView2/xBD is CC BY-NC-SA 4.0, RarePlanes is CC BY-SA 4.0, and DOTA is for academic use only). VLEO-Bench (task collection): C. Zhang and S. Wang, CVPRW 2024, doi:10.1109/CVPRW63382.2024.00780 GEOBench-VLM (task collection, DOTA/FAIR1M Ref-Det): M. S. Danish et al., ICCV 2025, doi:10.1109/ICCV51701.2025.00670 DIOR-RSVG: Y. Zhan, Z. Xiong and Y. Yuan, IEEE TGRS 61, 2023, doi:10.1109/TGRS.2023.3250471 DIOR (images of DIOR-RSVG): K. Li et al., ISPRS J. Photogramm. Remote Sens., 2020, doi:10.1016/j.isprsjprs.2019.11.023 PatternNet: W. Zhou, S. Newsam, C. Li and Z. Shao, ISPRS J. Photogramm. Remote Sens. 145, 2018, doi:10.1016/j.isprsjprs.2018.01.004 RarePlanes (CC BY-SA 4.0): J. Shermeyer et al., WACV 2021, doi:10.1109/WACV48630.2021.00025 COWC: T. N. Mundhenk, G. Konjevod, W. A. Sakla and K. Boakye, ECCV 2016, doi:10.1007/978-3-319-46487-9_48 NEON Tree: B. G. Weinstein et al., PLoS Comput. Biol. 17, 2021, doi:10.1371/journal.pcbi.1009180 Aerial Animal: J. A. J. Eikelboom et al., Methods Ecol. Evol. 10, 2019, doi:10.1111/2041-210X.13277 FPCD: C. Tundia, R. Kumar, O. Damani and G. Sivakumar, VISIGRAPP 2023, doi:10.5220/0011797600003417 DOTA (academic use only): G.-S. Xia et al., CVPR 2018, doi:10.1109/CVPR.2018.00418 FAIR1M: X. Sun et al., ISPRS J. Photogramm. Remote Sens. 184, 2022, doi:10.1016/j.isprsjprs.2021.12.004 xView2 / xBD: R. Gupta et al., arXiv:1911.09296, 2019, https://xview2.org (CC BY-NC-SA 4.0) Code and scoring scripts: github.com/Skyler-D-Liu/VLMs_progress_agents