NewHaven04

The performance of music involves the physical expression of musical material in a complex and multimodal process. Furthermore, musical performance involves a sense of 'flow', or immersion in the creative act, that can be better understood through a careful and holistic examination of data captured from this complex physical activity. Flow is especially relevant in improvisatory performance contexts where musicians must make real-time decisions about content and its expression. The project detailed in this paper involves the design and creation of a low-cost protocol for collecting simultaneous streams of data from improvising human musicians that are performing from a common score. The protocol records and synchronises audio recording with body-, facial- and physiological response tracking with a ground-truth annotation through the reported flow of a performer. This association yields a robust dataset that serves to capture the complex and multi-model process of making music 'in the flow'. This dataset can be a useful tool for a range of applications, such as creative AI practices, music generation in game engines, music information retrieval and humanisation of static systems—e.g. MIDI file playback and sound processing parameters.

音乐表现力涉及音乐材料的物理表达，这是一个复杂且多模态的过程。此外，音乐表演还包含一种‘流畅感’或沉浸于创造性活动中的感觉，这可以通过对从这一复杂物理活动中捕获的数据的细致而全面的审查来更好地理解。在即兴表演的情境中，‘流畅感’尤为重要，因为音乐家必须实时做出关于内容和其表达方式的决定。本文详细描述的项目涉及设计并创建一种低成本协议，用于收集即兴演奏者从共同乐谱演奏时的同时数据流。该协议记录并同步音频录音与身体、面部和生理反应跟踪，并通过表演者报告的‘流畅感’进行地面实况标注。这种关联产生了一个稳健的数据集，用于捕捉音乐‘在流畅感中’的复杂和多模型制作过程。该数据集可以成为多种应用的实用工具，例如创意AI实践、游戏引擎中的音乐生成、音乐信息检索以及静态系统的拟人化——例如MIDI文件播放和声音处理参数。