多模态人类意图理解MOSI数据集
收藏资源简介：
MOSI数据集的视频来源主要集中在视频博客（vlog）上。许多视频网站用户通过更新的视频表达自己对不同主题的看法，这些视频通常会用#vlog标签进行索引。视频保留了原始的上传分辨率，并以MP4格式录制，视频长度通常在2到5分钟之间。所有视频片段都经过人工转录，以提取口语词汇以及每个口语话语的开始时间。转录过程分为四个阶段。首先，由一名专家转录员手动转录所有视频；然后，第二名转录员审查并纠正转录内容。转录方案包括对停顿填充词（如"umm"、"uhh"等）、重音和语音停顿的详细标注。在第三阶段，使用名为P2FA4的强制对齐工具，将文本在单词和音素级别上与音频进行对齐。最后，手动检查比对结果，必要时使用PRAAT进行校正。音频和视觉特征从MPEG文件中自动提取，音频的帧率为1000，视频的帧率为30。视觉特征包括16个面部动作单元、68个面部标志、头部姿势和方向、6种基本情绪以及眼睛注视方向。音频特征则通过COVAREP提取，包括超过32个特征，如音调、能量、NAQ（归一化幅度商）、MFCC（梅尔频率倒谱系数）、峰值斜率和能量斜率等。
The video sources of the MOSI dataset are primarily from video blogs (vlogs). Many video platform users express their viewpoints on various topics through uploaded videos, which are typically indexed with the #vlog hashtag. The videos retain their original upload resolution, are recorded in MP4 format, and generally last between 2 and 5 minutes. All video clips undergo manual transcription to extract spoken vocabulary and the start time of each spoken utterance. The transcription process consists of four stages. First, all videos are manually transcribed by a professional transcriber; subsequently, a second transcriber reviews and revises the transcribed content. The transcription scheme involves detailed annotations for filled pauses (e.g., "umm", "uhh"), prosodic stress, and speech pauses. In the third stage, the forced alignment tool named P2FA4 is utilized to align the transcribed text with the audio at both word and phoneme levels. Finally, the alignment results are manually inspected, with corrections made using PRAAT when necessary. Audio and visual features are automatically extracted from the MPEG files, with an audio frame rate of 1000 and a video frame rate of 30. Visual features encompass 16 facial action units, 68 facial landmarks, head pose and orientation, 6 basic emotions, and eye gaze direction. Audio features are extracted via COVAREP, including more than 32 features such as pitch, energy, NAQ (Normalized Amplitude Quotient), MFCC (Mel-Frequency Cepstral Coefficients), peak slope, energy slope, and others.
