Multimodal Vision-Audio-Language Dataset

The Multimodal Vision-Audio-Language Dataset is a large-scale dataset for multimodal learning. It contains 2M video clips with corresponding audio and a textual description of the visual and auditory content. The dataset is an ensemble of existing datasets and fills the gap of missing modalities. Details can be found in the attached report. Annotation The annotation files are provided as Parquet files. They can be read using Python and the pandas and pyarrow library. The split into train, validation and test set follows the split of the original datasets. Installation pip install pandas pyarrow Example import pandas as pddf = pd.read_parquet('annotation_train.parquet', engine='pyarrow')print(df.iloc[0]) dataset AudioSet filename train/---2_BBVHAA.mp3 captions_visual [a man in a black hat and glasses.] captions_auditory [a man speaks and dishes clank.] tags [Speech] Description The annotation file consists of the following fields:filename: Name of the corresponding file (video or audio file)dataset: Source dataset associated with the data pointcaptions_visual: A list of captions related to the visual content of the video. Can be NaN in case of no visual contentcaptions_auditory: A list of captions related to the auditory content of the videotags: A list of tags, classifying the sound of a file. It can be NaN if no tags are provided Data files The raw data files for most datasets are not released due to licensing issues. They must be downloaded from the source. However, due to missing files, we provide them on request. Please contact us at [e-mail]本多模态视觉-音频-语言数据集（Multimodal Vision-Audio-Language Dataset）是一款面向多模态学习的大规模数据集。其收录200万条视频片段，每条片段均配套对应音频与视觉、听觉内容的文本描述。该数据集整合现有公开数据集资源，填补了模态缺失的研究空白。详细信息可参阅随附报告。 标注文件 标注文件以Parquet（Parquet）格式提供，可通过Python结合pandas与pyarrow库读取。 训练集、验证集与测试集的划分遵循原始数据集的划分规则。 安装 pip install pandas pyarrow 使用示例 import pandas as pd df = pd.read_parquet('annotation_train.parquet', engine='pyarrow') print(df.iloc[0]) 示例输出： dataset AudioSet filename train/---2_BBVHAA.mp3 captions_visual [a man in a black hat and glasses.] captions_auditory [a man speaks and dishes clank.] tags [Speech] 标注字段说明 该标注文件包含以下字段： filename：对应文件（视频或音频文件）的名称 dataset：该数据点所属的源数据集 captions_visual：视频视觉内容相关的字幕列表，若无视觉内容则该字段值可为NaN captions_auditory：视频听觉内容相关的字幕列表 tags：用于分类音频内容的标签列表，若未提供标签则该字段值可为NaN 数据文件 由于授权许可问题，多数数据集的原始数据文件未予公开，需从源地址自行下载。若存在文件缺失情况，可按需向我们申请获取，请发送邮件至[e-mail]联系。