Synthetic Multimodal Dataset for Daily Life Activities

Outline This dataset is originally created for the Knowledge Graph Reasoning Challenge for Social Issues (KGRC4SI) Video data that simulates daily life actions in a virtual space from Scenario Data. Knowledge graphs, and transcriptions of the Video Data content ("who" did what "action" with what "object," when and where, and the resulting "state" or "position" of the object). Knowledge Graph Embedding Data are created for reasoning based on machine learning This data is open to the public as open data Details Videos mp4 format 203 action scenarios For each scenario, there is a character rear view (file name ending in 0), an indoor camera switching view (file name ending in 1), and a fixed camera view placed in each corner of the room (file name ending in 2-5). Also, for each action scenario, data was generated for a minimum of 1 to a maximum of 7 patterns with different room layouts (scenes). A total of 1,218 videos Videos with slowly moving characters simulate the movements of elderly people. Knowledge Graphs RDF format 203 knowledge graphs corresponding to the videos Includes schema and location supplement information The schema is described below SPARQL endpoints and query examples are available Script Data txt format Data provided to VirtualHome2KG to generate videos and knowledge graphs Includes the action title and a brief description in text format. Embedding Embedding Vectors in TransE, ComplEx, and RotatE. Created with DGL-KE (https://dglke.dgl.ai/doc/) Embedding Vectors created with jRDF2vec (https://github.com/dwslab/jRDF2Vec). Specification of Ontology Please refer to the specification for descriptions of all classes, instances, and properties: https://aistairc.github.io/VirtualHome2KG/vh2kg_ontology.htm Related Resources KGRC4SI Final Presentations with automatic English subtitles (YouTube) VirtualHome2KG (Software) VirtualHome-AIST (Unity) VirtualHome-AIST (Python API) Visualization Tool (Software) Script Editor (Software)### 数据集大纲 本数据集最初为面向社会议题的知识图谱推理挑战赛（Knowledge Graph Reasoning Challenge for Social Issues，KGRC4SI）研发。 视频数据基于场景数据集，模拟虚拟空间内的日常生活行为，包含知识图谱（Knowledge Graph）以及视频内容的转录文本，记录了“何人”对“何物”实施了何种“动作”、发生的时间与地点，以及该物体最终的“状态”或“位置”。 知识图谱嵌入（Knowledge Graph Embedding）数据面向基于机器学习的推理任务构建。 本数据集以开放数据形式面向公众公开。 ### 详细说明 #### 视频文件 格式为MP4，涵盖203个动作场景。 每个场景包含四类视角的视频：角色后方视角（文件名以0结尾）、室内机位切换视角（文件名以1结尾），以及布置于房间四角的固定机位视角（文件名以2至5结尾）。此外，每个动作场景针对1至7种不同的房间布局生成对应数据，最终总计1218段视频。其中角色移动节奏较慢的视频用于模拟老年人的行动特征。 #### 知识图谱 格式为资源描述框架（Resource Description Framework，RDF），包含与上述视频一一对应的203份知识图谱，涵盖本体模式与位置补充信息，本体模式详见下文。本数据集提供SPARQL端点与查询示例。 #### 脚本数据 格式为纯文本（TXT），为用于向VirtualHome2KG输入以生成视频与知识图谱的脚本数据，以文本形式包含动作标题与简要描述。 #### 嵌入向量数据 包含基于TransE、ComplEx与RotatE模型生成的嵌入向量，由DGL-KE工具（https://dglke.dgl.ai/doc/）生成；另有基于jRDF2vec工具（https://github.com/dwslab/jRDF2Vec）生成的嵌入向量。 ### 本体规范 有关所有类、实例与属性的详细说明，请参阅以下本体规范文档：https://aistairc.github.io/VirtualHome2KG/vh2kg_ontology.htm ### 相关资源 1. 带自动英文字幕的KGRC4SI最终演示文稿（YouTube平台） 2. VirtualHome2KG（软件工具） 3. VirtualHome-AIST（Unity引擎版本） 4. VirtualHome-AIST（Python应用程序接口） 5. 可视化工具（软件） 6. 脚本编辑器（软件）