开放新闻库新闻文本预训练数据集

开放新闻库（OpenNewsArchive）是一个大规模新闻文本预训练数据集，包含880万篇来自公开网站的多领域新闻，涵盖财经、健康、军事、体育、房产、社会、学术等类别。数据经清洗去重，以jsonlines格式存储，包含标题、内容、发布日期和语言等字段。内容客观无偏见，多数发布于2023年，保障时效性与多样性，适用于大模型预训练任务。

OpenNewsArchive is a large-scale news text pre-training dataset. It consists of 8.8 million multi-domain news articles sourced from public websites, spanning categories including finance, health, military affairs, sports, real estate, society, and academia. The dataset has been cleaned and deduplicated, stored in JSONLines format, and contains fields such as title, content, publication date, and language. The content is objective and unbiased, with most articles published in 2023 to ensure both timeliness and diversity, making it suitable for pre-training tasks of large language models.

搜集汇总

背景与挑战

背景概述

开放新闻库新闻文本预训练数据集是一个包含880万篇多领域新闻的大规模语料库，涵盖财经、健康等多个类别，数据经过严格清洗处理并以jsonlines格式存储。该数据集具有时效性强、内容覆盖面广的特点，适合用于文本分析和预训练任务。