Cultural ecosystem service labels for photos from Flickr and Twitter using artificial intelligence models

Description: Dataset of photos downloaded from Flickr (241,582 photos) and Twitter-X (1,035,488 photos) labeled by different artificial intelligence models and validated by labels assigned by human experts. The entire dataset was labeled using different AI models. First, we applied a Large Language Model (GPT-4.1 from OPENAI) and Llava 1.6 (on a subset of the data) to extract semantic labels from the image content based on prompts fine-tuned using prompt engineering. In parallel, we used the base version of DINO (a self-supervised vision transformation model), fine-tuned with a subset of human expert-labeled images from our own dataset, to generate inferences for the entire image collection. We also incorporated labels derived from expert vision models pre-trained on established datasets such as ImageNet, COCO, Places365, and Nature, which provided complementary classification information. The labels used correspond to two categories (Table 1): Table 1. Categories used in social media photo tagging: Stoten, based on the scientific framework proposed by Moreno-Llorca et al. (2020). Level 3, a hierarchical tagging system developed by our team to provide greater thematic detail, especially suited for the identification of Cultural Ecosystem Services. Stoten Level3 Cultural Accommodation Fauna/Flora Air activities Gastronomy Animals Nature & Landscape Breakwater Not relevant Bridge Recreational Commerce facilities Religious Cities Rural tourism Clouds Sports Dam Sun and beach Dock Urban Fungus Heritage and culture Knowledge Landscapes Lighthouse Not relevant Other abiotic features Plants Roads Shelter Skies Spiritual, symbolic and related connotations Terrestrial activities Towns and villages Tracks and trails Vegetation and habitats Vehicle Water activities Wind farm Winter activities Table 2. Table of contents of the dataset Folder format Description AI models DINO model .pt and pth Model fine-tuned with a subset of expert-labeled images Expert models CES_label_tree .csv Equivalence table used to assign labels generated by expert models to our categories of interest (Stoten and Level3) LLMs GPT and Llava prompts GPT_Label_local_files .py Python script used for labeling photos using OPENAI models (in our case we used the GPT 4.1 model) Level3_GPT_LLava_7_prompts_used .txt Seven prompts used for photo tagging using GPT 4.1 and Llava 1.6 Stoten_GPT_LLava_7_prompts_used .txt Seven prompts used for photo tagging with Stoten using GPT 4.1 and Llava 1.6 Stoten_Level3_categories .csv Seven prompts used for photo tagging with level 3 using GPT 4.1 and Llava 1.6 Flickr AI based labels DINO Flickr_DINO_all .csv Inferences for all Flickr photos from the DINO model trained with the ground truth Expert models Flickr_expert_models_all .csv Labels generated by expert models for the entire database GPT Flickr_GPT_all .csv Database of Flickr photos tagged with CES using OPENAI's GPT-4.1 model. Flickr_GPT_7_prompts_8192 .csv Subset of the Flickr photo database with CES-related tags assigned by the GPT 4.1 model where 7 prompts are tested for Stoten and Level 3. Llava 1.6 Flickr_Llava_1-6 .csv Subset of the Flickr photo database with CES-related tags assigned by the Llava 1.6 model where 7 prompts are tested for Stoten and Level 3. Ground truth Ground Truth labels Flickr_Database_Labeled_1082 .csv Contain labels assigned by human experts and after rounds of review and consensus, for both Stoten and Level 3, from 1082 Flickr photos Flickr_Database_Labeled_7110 .csv Ground Truth, an archive containing labels assigned by human experts and after rounds of review and consensus, for both Stoten and Level 3, from 7110 Flickr photos Flickr_Database_Labeled_8192 .csv Union of the two databases labeled above Ground Truth photos 1082 .jpg/png Photos labeled by human experts, these photos were selected to be representative of different parks, with different levels of protection and representative of different CES 7110 .jpg/png Photos labeled by human experts, these photos were selected to be representative of different parks, with different levels of protection and representative of different CES Human labels Flickr_DataBase_Labeled_1082_expert1_AS .csv File containing tags assigned by expert 1 for both Stoten and Level 3, from 1082 Flickr photos Flickr_DataBase_Labeled_1082_expert2_FG .csv File containing tags assigned by expert 2 for both Stoten and Level 3, from 1082 Flickr photos Flickr_DataBase_Labeled_7110_expert1_CN .csv File containing tags assigned by expert 1 for both Stoten and Level 3, from 7110 Flickr photos Twitter AI based labels DINO Twitter_DINO_all .csv Inferences for all Twitter photos from the DINO model trained with the ground truth Expert models Twitter_expert_models_all .csv Labels generated by expert models for the entire database GPT Twitter_GPT_all .csv Database of Twitter photos tagged with CES using OPENAI's GPT-4.1 model. Twitter_GPT_7_prompts_150 .csv Subset of the Twitter photo database with CES-related tags assigned by the GPT 4.1 model where 7 prompts are tested for Stoten and Level 3. Llava 1.6 Twitter_Llava_1-6 .csv Subset of the Twitter photo database with CES-related tags assigned by the Llava 1.6 model where 7 prompts are tested for Stoten and Level 3. Ground truth Ground Truth labels Twitter_Database_Labeled_150 .csv Contain labels assigned by human experts and after rounds of review and consensus, for both Stoten and Level 3, from 150 Twitter photos Twitter_Database_Labeled_6804 .csv Contain labels assigned by human experts and after rounds of review and consensus, for both Stoten and Level 3, from 6804 Twitter photos Ground Truth photos 150 .jpg/png Photos labeled by human experts, these photos were selected to be representative of different parks, with different levels of protection and representative of different CES 6804 .jpg/png Photos labeled by human experts, these photos were selected to be representative of different parks, with different levels of protection and representative of different CES Human labels Flickr_DataBase_Labeled_150_7experts .csv File containing tags assigned by 7 experts for both Stoten and Level 3, from 150 Twitter photos Flickr_DataBase_Labeled_6804_expert1_FG .csv File containing tags assigned by expert 2 for both Stoten and Level 3, from 6804 Twitter photos References: Moreno-Llorca, R., Méndez, P. F., Ros-Candeira, A., Alcaraz-Segura, D., Santamaría, L., Ramos-Ridao, Á. F., ... & Vaz, A. S. (2020). Evaluating tourist profiles and nature-based experiences in Biosphere Reserves using Flickr: Matches and mismatches between online social surveys and photo content analysis. Science of the Total Environment, 737, 140067. https://doi.org/10.1016/j.scitotenv.2020.140067