Supplementary Material for "Gesture Desk -- An Integrated Multi-Modal Interface for Interactive Sonification"

This paper presents the gesture desk, a new platform for a human-computer interface at a regular computer workplace. It extends classical input devices like keyboard and mouse by arm and hand gestures, without the need to use any inconvenient accessories like data gloves or markers. A central element is a "gesture box" containing two infrared cameras and a color camera which is positioned under a glass desk. Arm and hand motions are tracked in three dimensions. A synchronizer board has been developed to provide an active glare-free IR-illumination for robust body and hand tracking. As a first application, we demonstrate interactive real-time browsing and querying of auditory self-organizing maps (AuSOMs). An AuSOM is a combined visual and auditory presentation of high-dimensional data sets. Moving the hand above the desk surface allows to select neurons on the map and to manipulate how they contribute to data sonification. Each neuron is associated with a prototype vector in high-dimensional space, so that a set of 2D-topologically ordered feature maps is queried simultaneously. The level of detail is selected by hand altitude over the table surface, allowing to emphasize or deemphasize neurons on the map.

本研究提出了手势工作台（gesture desk），一种适配常规计算机办公场景的人机交互界面新平台。该平台通过手臂与手部手势拓展了键盘、鼠标等经典输入设备的交互能力，无需佩戴数据手套、标记点等繁琐外设。其核心组件为置于玻璃工作台下方的“手势盒（gesture box）”，内置两台红外相机与一台彩色相机，可实现手臂与手部动作的三维追踪。研究团队开发了同步控制板，用于提供主动无眩光红外照明，以实现稳定可靠的身体与手部追踪。作为首个应用案例，本研究展示了针对听觉自组织映射（auditory self-organizing maps, AuSOMs）的交互式实时浏览与查询功能。听觉自组织映射是一种对高维数据集进行可视化与听觉化结合呈现的技术。用户可在工作台表面上方移动手部，以选中映射中的神经元，并调整其对数据声化过程的贡献权重。每个神经元均对应高维空间中的一个原型向量，因此可同时查询一组具备二维拓扑有序性的特征映射。用户可通过调整手部在工作台上方的高度来调整细节展示层级，从而强化或弱化映射中特定神经元的权重。