The Effects of the Machine-driven Cross-modal Semantic on Soundscape

This repository contains the processed data, metadata, and supplementary materials supporting the study “Scalable Assessment of Perceived Soundscape Restorativeness in Urban Context Using Cross-Modal Audiovisual Semantics”. The study examined 130 urban sites across nine cities using one-minute binaural recordings, 360° panoramic images, and 3,993 valid evaluations provided by 506 participants. We extracted auditory semantic evidence using PANNs and organized it into Nature, Human activity, and Technology domains. Visual semantic composition was derived using DeepLabV3+, while audiovisual representational similarity was estimated in ImageBind’s shared embedding space. The deposit provides site-level perceived soundscape restorativeness scores, auditory and visual semantic variables, audiovisual similarity scores, semantic mapping and pooling rules, model-validation outputs, and source data for the reported figures and tables. It also includes scripts for descriptive analysis, correlation analysis, repeated nested cross-validation, explainable boosting machine interpretation, and visualization. A variable dictionary documents each variable's definition, unit, transformation, aggregation procedure, and provenance. English and Chinese questionnaire materials, scoring procedures, data-quality criteria, and supplementary tables are included. Participant-level responses and unredacted audiovisual materials are not included in the public dataset because they require separate assessment of consent, privacy, and third-party rights. Third-party training datasets and pretrained model weights are not redistributed; their sources, versions, licenses, and retrieval information are documented in the repository. These materials support reproducible statistical analysis of machine-derived audiovisual semantics and perceived soundscape restorativeness, and may facilitate further research on multisensory urban assessment, soundscape modelling, and interpretable machine learning. The audiovisual source materials used in this study were obtained from the Urban Soundscapes of the World dataset, version 1.0.0 (De Coensel et al., 2023), available at https://doi.org/10.5281/zenodo.10106181 under the Creative Commons Attribution 4.0 International licence. This deposit constitutes a derived research dataset and adds questionnaire documentation, site-level perceptual measures, machine-derived semantic features, model outputs, and analysis code. Any redistributed audiovisual stimuli are identified at the file level, and all transformations from the source materials are documented. No endorsement by the original dataset creators is implied. De Coensel, B., Botteldooren, D., Kang, S., & Van Renterghem, T. (2023). Urban Soundscapes of the World (Version 1.0.0) [Data set]. Ghent University. https://doi.org/10.5281/zenodo.10106181