MNBVC

MNBVC (海量永无止境的BT海量中文语料库) 超大规模中文语料集。对标chatGPT训练的40t数据。MNBVC数据集不但包括主流文化，也包括各个小众文化甚至火星文的数据。MNBVC数据集包括新闻、作文、小说、书籍、杂志、论文、台词、帖子、维基、古诗、歌词、商品介绍、笑话、糗事、聊天记录等一切形式的纯文本中文数据。

MNBVC (Massive Never-ending BT Chinese Corpus) is an ultra-large-scale Chinese corpus dataset. It is benchmarked against the 40 trillion tokens of training data used for ChatGPT. The MNBVC dataset covers not only mainstream cultural content but also data from various niche cultures and even martian language, a type of non-standard Chinese internet slang. It includes all forms of pure-text Chinese data, such as news, compositions, novels, books, magazines, academic papers, dialogue lines, posts, Wikipedia articles, ancient Chinese poems, song lyrics, product introductions, jokes, embarrassing anecdotes, chat logs, and more.

搜集汇总

背景与挑战

背景概述

MNBVC是一个超大规模的中文语料库，包含新闻、作文、小说等多种形式的纯文本数据，对标chatGPT的训练数据规模。该数据集由OpenDataLab于2023年发布，采用MIT许可证。