mOSCAR

mOSCAR是由法国国家信息与自动化研究所等机构创建的第一个大规模多语言和多模态文档级语料库，涵盖163种语言，包含315M文档、214B tokens和1.2B images。该数据集通过从Common Crawl中爬取数据，并经过一系列的过滤和评估步骤确保数据的安全性、多样性和质量。mOSCAR的创建旨在推动多语言和多模态研究，特别是在解决全球7000多种语言的mLLM研究限制问题。数据集的应用领域包括提升多语言图像-文本任务的少样本学习性能，验证了在多语言环境中训练模型的有效性。mOSCAR is the first large-scale multilingual and multimodal document-level corpus created by institutions such as the French National Institute for Research in Computer Science and Automation (INRIA). It covers 163 languages, and contains 315 million documents, 214 billion tokens and 1.2 billion images. This dataset is sourced via web crawling from Common Crawl, with a series of filtering and evaluation procedures applied to guarantee data security, diversity and quality. The development of mOSCAR aims to advance multilingual and multimodal research, particularly to address the research limitations of multilingual large language models (mLLMs) across the more than 7,000 languages globally. Application fields of this dataset include enhancing the few-shot learning performance of multilingual image-text tasks, which has validated the effectiveness of training models in multilingual environments.

搜集汇总

背景与挑战

背景概述

mOSCAR是一个大规模的多语言和多模态文档语料库，从网络爬取而来，覆盖163种语言，包含3.15亿份文档、2140亿个标记和12亿张图像。该数据集经过仔细过滤和评估，以确保其安全性、多样性和高质量，适用于多语言和多模态研究。