RealVAD: A Real-world Dataset for Voice Activity Detection

RealVAD: A Real-world Dataset for Voice Activity Detection The task of automatically detecting “Who is Speaking and When” is broadly named as Voice Activity Detection (VAD). Automatic VAD is a very important task and also the foundation of several domains, e.g., human-human, human-computer/ robot/ virtual-agent interaction analyses, and industrial applications. RealVAD dataset is constructed from a YouTube video composed of a panel discussion lasting approx. 83 minutes. The audio is available from a single channel. There is one static camera capturing all panelists, the moderator and audiences. Particular aspects of RealVAD dataset are: It is composed of panelists with different nationalities (British, Dutch, French, German, Italian, American, Mexican, Columbian, Thai). This aspect allows studying the effect of ethnic origin variety to the automatic VAD. There is a gender balance such that there are four female and five male panelists. The panelists are sitting in two rows and they can be gazing audience, other panelists, their laptop, the moderator or anywhere in the room while speaking or not-speaking. Therefore, they were captured not only from frontal-view but also from side-view varying based on their instant posture and head orientation. The panelists are moving freely and are doing various spontaneous actions (e.g., drinking water, checking their cell phone, using their laptop, etc.), resulting in different postures. The panelists’ body parts are sometimes partially occluded by their/other's body part or belongings (e.g., laptop). There are also natural changes of illumination and shadow rising on the wall behind the panelists in the back row. Especially, for the panelists sitting in the front row, there is sometimes background motion occurring when the person(s) behind them moves. The annotations includes: The upper body detection of nine panelists in bounding box form. Associated VAD ground-truth (speaking, not-speaking) for nine panelists. Acoustic features extracted from the video: MFCC and raw filterbank energies. All info regarding the annotations are given in the ReadMe.txt and Acoustic Features README.txt files. When using this dataset for your research, please cite the following paper in your publication: C. Beyan, M. Shahid and V. Murino, "RealVAD: A Real-world Dataset and A Method for Voice Activity Detection by Body Motion Analysis", in IEEE Transactions on Multimedia, 2020.

RealVAD：用于语音活动检测的真实世界数据集

自动识别“何人何时发声”的任务被统称为语音活动检测（Voice Activity Detection，VAD）。自动VAD是一项至关重要的基础任务，广泛支撑人际交互、人机/机器人/AI智能体交互分析以及工业应用等多个领域。

RealVAD数据集源自一段时长约83分钟的YouTube专题讨论视频，仅提供单声道音频。场景由一台静态摄像机覆盖，可捕捉所有专题讨论嘉宾、主持人与观众。

RealVAD数据集的独特之处在于：参与讨论的嘉宾来自不同国籍，涵盖英国、荷兰、法国、德国、意大利、美国、墨西哥、哥伦比亚、泰国，可用于研究种族背景多样性对自动VAD系统的影响；嘉宾性别比例均衡，共有4名女性与5名男性讨论嘉宾。

讨论嘉宾分两排就座，发言或静止时，视线可朝向观众、其他嘉宾、个人笔记本电脑、主持人或房间内任意方向。因此，摄像机不仅能捕捉正面视角，还会根据嘉宾实时姿态与头部朝向，捕获不同角度的侧面视角。嘉宾可自由活动，会做出各类自发动作（如饮水、查看手机、使用笔记本电脑等），呈现出多样化的肢体姿态。嘉宾的身体部位有时会被自身或其他嘉宾的肢体、随身物品（如笔记本电脑）部分遮挡。

后排嘉宾身后的墙面还会出现自然光照明变化与阴影自然移动的情况。尤其对于前排嘉宾而言，当其身后人员移动时，背景常会出现动态变化。

数据集标注内容包括：以边界框形式标注的9名讨论嘉宾的上半身区域；对应9名嘉宾的VAD真值标签（发声/未发声）；从视频中提取的声学特征：梅尔频率倒谱系数（MFCC）与原始滤波器组能量。所有标注相关信息均收录于ReadMe.txt与Acoustic Features README.txt文件中。

若将本数据集用于科研工作，请在您的出版物中引用以下论文：C. Beyan、M. Shahid与V. Murino，《RealVAD：基于肢体动作分析的语音活动检测数据集与方法》，发表于《IEEE多媒体汇刊》，2020年。

搜集汇总

背景与挑战

背景概述

RealVAD是一个用于语音活动检测的真实世界数据集，基于约83分钟的YouTube小组讨论视频构建，包含9位不同国籍和性别平衡的小组成员。该数据集的特点是真实场景下的多样性，包括多种姿势、遮挡、光照变化和背景运动，并提供了上体检测边界框、VAD真值以及声学特征注释，适用于研究多民族背景下的自动语音活动检测。