MineDojo Internet Knowledge Base (YouTube)

<strong>Project website:</strong> minedojo.org <strong>Paper:</strong> arxiv.org/abs/2206.08853 <strong>GitHub:</strong> github.com/MineDojo/MineDojo Minecraft is among the most streamed games on YouTube. Human players have demonstrated a stunning range of creative activities and sophisticated missions that take hours to complete. We collect 730K+ narrated Minecraft videos, which add up to <strong>33 years of duration and 2.2B words</strong> in English transcripts. The time-aligned transcripts enable the agent to ground free-form natural language in video pixels and learn the semantics of diverse activities without laborious human labeling. There are two files in our YouTube knowledge base. <strong>youtube_tutorial.json</strong> (tutorial videos): Minecraft tutorial videos include step-by-step demonstrations and sometimes detailed verbal explanations. They also serve as a rich source of creative missions that humans find interesting. We harvest thousands of tasks from these videos in our benchmarking suite. <strong>youtube_full.json</strong> (general gameplay videos): Unlike tutorials, general gameplay videos do not necessarily provide guidance on particular tasks. Instead, they capture the “in-the-wild” human experiences that are much larger in quantity, diverse in contents, and rich in learning signals. Data Structure <pre><code class="language-python">list[ { "id": str, # video id "title": str, # video title "link": str, # video link "view_count": int # number of times the video has been viewed "like_count": int # number of users who have indicated that they liked the video "duration": float # video duration in seconds "fps": float, # video FPS } ]</code></pre> Check out our paper! <pre><code class="language-markdown">@article{fan2022minedojo, title = {MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge}, author = {Linxi Fan and Guanzhi Wang and Yunfan Jiang and Ajay Mandlekar and Yuncong Yang and Haoyi Zhu and Andrew Tang and De-An Huang and Yuke Zhu and Anima Anandkumar}, year = {2022}, journal = {arXiv preprint arXiv: Arxiv-2206.08853} }</code></pre>