textcleanlm/textclean-corpus-1M
收藏Hugging Face2025-07-18 更新2025-08-09 收录
下载链接：
https://hf-mirror.com/datasets/textcleanlm/textclean-corpus-1M
下载链接
链接失效反馈官方服务：
资源简介：
TextClean-Corpus-1M是一个包含100万个token的预处理网络文本数据集，旨在降低语言模型处理下游任务时的计算成本。该数据集通过移除无关元素（如导航链接、广告和模板内容），同时保留核心信息，使得语言模型能够更高效地处理。数据集来源于EssentialWeb 1.0的随机样本，使用OpenAI的o4-mini模型进行清理，并以Markdown格式呈现。
TextClean-Corpus-1M is a preprocessed web text dataset containing 1 million tokens, designed to reduce computational costs in downstream language model applications. By removing irrelevant elements such as navigation links, advertisements, and boilerplate content while preserving the core information, this dataset enables more efficient processing. The dataset is sourced from random samples of EssentialWeb 1.0 and cleaned using OpenAIs o4-mini model, presented in Markdown format.
提供机构：
textcleanlm
