Cultural ecosystem service labels for photos from Flickr and Twitter using artificial intelligence models

Description:

Dataset of photos downloaded from Flickr (241,582 photos) and Twitter-X (1,035,488 photos) labeled by different artificial intelligence models and validated by labels assigned by human experts. 

The entire dataset was labeled using different AI models. First, we applied a Large Language Model (GPT-4.1 from OPENAI) and Llava 1.6 (on a subset of the data) to extract semantic labels from the image content based on prompts fine-tuned using prompt engineering.

In parallel, we used the base version of DINO (a self-supervised vision transformation model), fine-tuned with a subset of human expert-labeled images from our own dataset, to generate inferences for the entire image collection.

We also incorporated labels derived from expert vision models pre-trained on established datasets such as ImageNet, COCO, Places365, and Nature, which provided complementary classification information.

The labels used correspond to two categories (Table 1):

Table 1. Categories used in social media photo tagging: Stoten, based on the scientific framework proposed by Moreno-Llorca et al. (2020). Level 3, a hierarchical tagging system developed by our team to provide greater thematic detail, especially suited for the identification of Cultural Ecosystem Services.

Stoten

Level3

Cultural

Accommodation

Fauna/Flora

Air activities

Gastronomy

Animals

Nature & Landscape

Breakwater

Not relevant

Bridge

Recreational

Commerce facilities

Religious

Cities

Rural tourism

Clouds

Sports

Dam

Sun and beach

Dock

Urban

Fungus

Heritage and culture

Knowledge

Landscapes

Lighthouse

Not relevant

Other abiotic features

Plants

Roads

Shelter

Skies

Spiritual, symbolic and related connotations

Terrestrial activities

Towns and villages

Tracks and trails

Vegetation and habitats

Vehicle

Water activities

Wind farm

Winter activities

Table 2. Table of contents of the dataset

Folder

format

Description

AI models

DINO

model

.pt and pth

Model fine-tuned with a subset of expert-labeled images

Expert models

CES_label_tree

.csv

Equivalence table used to assign labels generated by expert models to our categories of interest (Stoten and Level3)

LLMs GPT and Llava prompts

GPT_Label_local_files

.py

Python script used for labeling photos using OPENAI models (in our case we used the GPT 4.1 model)

Level3_GPT_LLava_7_prompts_used

.txt

Seven prompts used for photo tagging using GPT 4.1 and Llava 1.6

Stoten_GPT_LLava_7_prompts_used

.txt

Seven prompts used for photo tagging with Stoten using GPT 4.1 and Llava 1.6

Stoten_Level3_categories

.csv

Seven prompts used for photo tagging with level 3 using GPT 4.1 and Llava 1.6

Flickr

AI based labels

DINO

Flickr_DINO_all

.csv

Inferences for all Flickr photos from the DINO model trained with the ground truth

Expert models

Flickr_expert_models_all

.csv

Labels generated by expert models for the entire database

GPT

Flickr_GPT_all

.csv

Database of Flickr photos tagged with CES using OPENAI's GPT-4.1 model.

Flickr_GPT_7_prompts_8192

.csv

Subset of the Flickr photo database with CES-related tags assigned by the GPT 4.1 model where 7 prompts are tested for Stoten and Level 3.

Llava 1.6

Flickr_Llava_1-6

.csv

Subset of the Flickr photo database with CES-related tags assigned by the Llava 1.6 model where 7 prompts are tested for Stoten and Level 3.

Ground truth

Ground Truth labels

Flickr_Database_Labeled_1082

.csv

Contain labels assigned by human experts and after rounds of review and consensus, for both Stoten and Level 3, from 1082 Flickr photos

Flickr_Database_Labeled_7110

.csv

Ground Truth, an archive containing labels assigned by human experts and after rounds of review and consensus, for both Stoten and Level 3, from 7110 Flickr photos

Flickr_Database_Labeled_8192

.csv

Union of the two databases labeled above

Ground Truth photos

1082

.jpg/png

Photos labeled by human experts, these photos were selected to be representative of different parks, with different levels of protection and representative of different CES

7110

.jpg/png

Photos labeled by human experts, these photos were selected to be representative of different parks, with different levels of protection and representative of different CES

Human labels

Flickr_DataBase_Labeled_1082_expert1_AS

.csv

File containing tags assigned by expert 1 for both Stoten and Level 3, from 1082 Flickr photos

Flickr_DataBase_Labeled_1082_expert2_FG

.csv

File containing tags assigned by expert 2 for both Stoten and Level 3, from 1082 Flickr photos

Flickr_DataBase_Labeled_7110_expert1_CN

.csv

File containing tags assigned by expert 1 for both Stoten and Level 3, from 7110 Flickr photos

Twitter

AI based labels

DINO

Twitter_DINO_all

.csv

Inferences for all Twitter photos from the DINO model trained with the ground truth

Expert models

Twitter_expert_models_all

.csv

Labels generated by expert models for the entire database

GPT

Twitter_GPT_all

.csv

Database of Twitter photos tagged with CES using OPENAI's GPT-4.1 model.

Twitter_GPT_7_prompts_150

.csv

Subset of the Twitter photo database with CES-related tags assigned by the GPT 4.1 model where 7 prompts are tested for Stoten and Level 3.

Llava 1.6

Twitter_Llava_1-6

.csv

Subset of the Twitter photo database with CES-related tags assigned by the Llava 1.6 model where 7 prompts are tested for Stoten and Level 3.

Ground truth

Ground Truth labels

Twitter_Database_Labeled_150

.csv

Contain labels assigned by human experts and after rounds of review and consensus, for both Stoten and Level 3, from 150 Twitter photos

Twitter_Database_Labeled_6804

.csv

Contain labels assigned by human experts and after rounds of review and consensus, for both Stoten and Level 3, from 6804 Twitter photos

Ground Truth photos

150

.jpg/png

Photos labeled by human experts, these photos were selected to be representative of different parks, with different levels of protection and representative of different CES

6804

.jpg/png

Photos labeled by human experts, these photos were selected to be representative of different parks, with different levels of protection and representative of different CES

Human labels

Flickr_DataBase_Labeled_150_7experts

.csv

File containing tags assigned by 7 experts  for both Stoten and Level 3, from 150 Twitter photos

Flickr_DataBase_Labeled_6804_expert1_FG

.csv

File containing tags assigned by expert 2 for both Stoten and Level 3, from 6804 Twitter photos

References: 

Moreno-Llorca, R., Méndez, P. F., Ros-Candeira, A., Alcaraz-Segura, D., Santamaría, L., Ramos-Ridao, Á. F., ... & Vaz, A. S. (2020). Evaluating tourist profiles and nature-based experiences in Biosphere Reserves using Flickr: Matches and mismatches between online social surveys and photo content analysis. Science of the Total Environment, 737, 140067. https://doi.org/10.1016/j.scitotenv.2020.140067