UK03

<b>Abstract</b>The performance of music involves the physical expression of musical material in a complex and multimodal process. Furthermore, musical performance involves a sense of 'flow', or immersion in the creative act, that can be better understood through a careful and holistic examination of data captured from this complex physical activity. Flow is especially relevant in improvisatory performance contexts where musicians must make real-time decisions about content and its expression. The project detailed in this paper involves the design and creation of a low-cost protocol for collecting simultaneous streams of data from improvising human musicians that are performing from a common score. The protocol records and synchronises audio recording with body-, facial- and physiological response tracking with a ground-truth annotation through the reported flow of a performer. This association yields a robust dataset that serves to capture the complex and multi-model process of making music 'in the flow'. This dataset can be a useful tool for a range of applications, such as creative AI practices, music generation in game engines, music information retrieval and humanisation of static systems—e.g. MIDI file playback and sound processing parameters.

<b>摘要</b> 音乐表演是一种复杂且多模态的过程，涉及音乐素材的物理性表达。此外，音乐表演还蕴含心流（flow）体验，即沉浸于创作行为之中，通过对该复杂物理活动采集的数据开展细致且整体性的分析，能够更透彻地理解这类体验。心流体验在即兴表演场景中尤为关键，此时音乐家需针对演奏内容及其表达形式做出实时决策。本论文所阐述的项目，旨在设计并搭建一套低成本采集协议，用于从基于共同乐谱进行即兴演奏的人类演奏者群体中同步采集多流数据。该协议可将音频录制与身体、面部及生理反应追踪数据进行同步整合，并结合演奏者自述的心流体验完成真值标注。由此构建的鲁棒数据集能够完整捕捉“沉浸于心流状态”中的音乐创作复杂多模态过程。该数据集可广泛应用于多个场景，例如创意人工智能创作实践、游戏引擎中的音乐生成、音乐信息检索，以及静态系统的人性化适配——如MIDI文件播放与音效处理参数优化。