MIDI event.

Virtual Reality (VR) technology uses computers to simulate the real world comprehensively. VR has been widely used in college teaching and has a huge application prospect. To better apply computer-aided instruction technology in music teaching, a music teaching system based on VR technology is proposed. First, a virtual piano is developed using the HTC Vive kit and the Leap Motion sensor fixed on the helmet as the hardware platform, and using Unity3D, related SteamVR plug-ins, and Leap Motion plug-ins as software platforms. Then, a gesture recognition algorithm is proposed and implemented. Specifically, the Dual Channel Convolutional Neural Network (DCCNN) is adopted to collect the user’s gesture command data. The dual-size convolution kernel is applied to extract the feature information in the image and the gesture command in the video, and then the DCCNN recognizes it. After the spatial and temporal information is extracted, Red-Green-Blue (RGB) color pattern images and optical flow images are input into the DCCNN. The prediction results are merged to obtain the final recognition result. The experimental results reveal that the recognition accuracy of DCCNN for the Curwen gesture is as high as 96%, and the recognition accuracy varies with different convolution kernels. By comparison, it is found that the recognition effect of DCCNN is affected by the size of the convolution kernel. Combining convolution kernels of size 5×5 and 7×7 can improve the recognition accuracy to 98%. The research results of this study can be used for music teaching piano and other VR products, with extensive popularization and application value.

虚拟现实（Virtual Reality，VR）技术依托计算机对现实世界进行全方位模拟。目前，虚拟现实技术已在高校教学领域得到广泛应用，拥有广阔的应用前景。为更好地将计算机辅助教学技术应用于音乐教学，本研究提出了一种基于虚拟现实技术的音乐教学系统。首先，本研究以固定于头盔的HTC Vive开发套件与Leap Motion传感器作为硬件平台，依托Unity3D、配套SteamVR插件及Leap Motion插件搭建软件平台，开发出一款虚拟钢琴；随后，本研究提出并实现了一套手势识别算法。具体而言，本研究采用双通道卷积神经网络（Dual Channel Convolutional Neural Network，DCCNN）采集用户的手势指令数据，通过双尺寸卷积核提取图像中的特征信息与视频中的手势指令，再由双通道卷积神经网络完成识别。在提取时空信息后，将红绿蓝（Red-Green-Blue，RGB）色彩模式图像与光流图像输入至双通道卷积神经网络，对各预测结果进行融合即可得到最终的识别结果。实验结果表明，双通道卷积神经网络对柯尔文手势（Curwen gesture）的识别准确率高达96%，且识别准确率随卷积核尺寸的不同而变化；对比实验显示，其识别效果受卷积核尺寸的影响，将5×5与7×7尺寸的卷积核进行组合，可将识别准确率提升至98%。本研究的成果可应用于音乐教学虚拟钢琴及其他虚拟现实产品，具备广泛的推广与应用价值。