Image and sound data from film Fantasia produced by Walt Disney

This repository contains the data used in the article Convolutional neural networks and transfer learning applied to automatic composition of descriptive music published in the 15th International Conference on Distributed Computing and Artificial Intelligence (DCAI). Data structure is explained in detail in the article. This proposal is the continuation of an earlier work whose data are available in a GitHub repository. Abstract Visual and musical arts has been strongly interconnected throughout history. The aim of this work is to compose music on the basis of the visual characteristics of a video. For this purpose, descriptive music is used as a link between image and sound and a video fragment of film Fantasia is deeply analyzed. Specially, convolutional neural networks in combination with transfer learning are applied in the process of extracting image descriptors. In order to establish a relationship between the visual and musical information, Naive Bayes, Support Vector Machine and Random Forest classifiers are applied. The obtained model is subsequently employed to compose descriptive music from a new video. The results of this proposal are compared with those of an antecedent work in order to evaluate the performance of the classifiers and the quality of the descriptive musical composition. DATAtrain_data.arff: Image descriptors and the most important sound of each frame from the fragment "The Nutcracker Suite" in film Fantasia obtained by means of CNNs. Data stored into ARFF format.test_data.arff: Image descriptors of each frame from the fragment "The Firebird" in film Fantasia 2000 obtained by means of CNNs. Data stored into ARFF format.midi.csv: Frame number of the fragment "The Firebird" in film Fantasia 2000 and the sound predicted by the system encoded in MIDI. Data stored into CSV format.firebird_prediction.mp3: Audio file with the synthesizing of the prediction data for the fragment "The Firebird" of film Fantasia 2000.LICENSEData is available under MIT License. To make use of the data the article must be cited.本仓库包含发表于第15届分布式计算与人工智能国际会议（DCAI）的论文《卷积神经网络与迁移学习应用于描述性音乐自动创作》中所使用的数据集。该数据集的结构已在上述论文中详细说明。本研究是此前一项工作的延续，该工作的数据集已公开于GitHub代码仓库。 摘要：视觉艺术与音乐艺术在人类历史进程中始终保持着深度互联的关联。本研究旨在基于视频的视觉特征创作音乐。为此，本研究采用描述性音乐作为图像与声音之间的桥梁，并对电影《幻想曲》（Fantasia）的一段视频片段进行了深度分析。具体而言，本研究将卷积神经网络（Convolutional Neural Networks, CNN）与迁移学习（Transfer Learning）相结合，用于提取图像描述符。为建立视觉信息与音乐信息之间的关联，本研究应用了朴素贝叶斯（Naive Bayes）、支持向量机（Support Vector Machine）与随机森林（Random Forest）分类器。所得模型后续被用于从新的视频片段中创作描述性音乐。为评估分类器性能与描述性音乐创作质量，本研究将本次方法的结果与此前一项工作的结果进行了对比。 train_data.arff：通过卷积神经网络提取自电影《幻想曲》中《胡桃夹子组曲》片段的各帧图像描述符，以及对应帧的关键音频特征，数据以ARFF格式存储。 test_data.arff：通过卷积神经网络提取自电影《幻想曲2000》（Fantasia 2000）中《火鸟》片段的各帧图像描述符，数据以ARFF格式存储。 midi.csv：电影《幻想曲2000》中《火鸟》片段的帧编号，以及系统预测的、以MIDI格式编码的音频数据，数据以CSV格式存储。 firebird_prediction.mp3：基于电影《幻想曲2000》中《火鸟》片段的预测数据合成的音频文件。 本数据集采用MIT许可证进行开源授权。使用本数据集时需引用上述发表论文。