NewHaven01

The performance of music involves the physical expression of musical material in a complex and multimodal process. Furthermore, musical performance involves a sense of 'flow', or immersion in the creative act, that can be better understood through a careful and holistic examination of data captured from this complex physical activity. Flow is especially relevant in improvisatory performance contexts where musicians must make real-time decisions about content and its expression. The project detailed in this paper involves the design and creation of a low-cost protocol for collecting simultaneous streams of data from improvising human musicians that are performing from a common score. The protocol records and synchronises audio recording with body-, facial- and physiological response tracking with a ground-truth annotation through the reported flow of a performer. This association yields a robust dataset that serves to capture the complex and multi-model process of making music 'in the flow'. This dataset can be a useful tool for a range of applications, such as creative AI practices, music generation in game engines, music information retrieval and humanisation of static systems—e.g. MIDI file playback and sound processing parameters.

音乐表演是一种复杂且多模态的过程，涉及音乐素材的具身表达。此外，音乐表演蕴含‘心流（flow）’状态——即沉浸于创作行为之中，通过对这一复杂身体活动所采集的数据进行细致且整体的审视，我们能更深入地理解这一心流状态。心流状态在即兴表演场景中尤为关键，此时音乐家需针对演奏内容及其表达形式做出实时决策。本文详述的研究项目旨在设计并构建一套低成本采集协议，用于采集基于共同乐谱进行即兴演奏的人类音乐家的同步数据流。该采集协议可实现音频录制与身体、面部及生理反应追踪数据的同步，并以演奏者自述的心流状态作为真实标注（ground-truth annotation）。这种数据关联方式可生成一套稳健的高质量数据集，能够完整捕捉‘处于心流状态’的音乐创作这一复杂多模态过程。该数据集可应用于诸多场景，例如创作式人工智能实践、游戏引擎中的音乐生成、音乐信息检索（Music Information Retrieval），以及静态系统的人性化改造——比如MIDI文件播放与声音处理参数调整。