hynky/betterweb

BetterWeb是FineWeb数据集的改进版本，通过集成质量评分方法对原始数据进行质量过滤。该数据集结合了三种互补的质量信号：DCLM FastText质量分类器（衡量一般指令跟随质量和推理清晰度）、FineWeb-Edu教育分类器（基于460K LLM标注样本训练）和写作质量启发式方法（包括词汇丰富度、信息密度、句子质量和结构质量等指标）。数据集包含多个质量评分指标和质量等级（0-4），每个文档都有一个quality_tier属性用于灵活过滤。数据集是从FineWeb sample-10BT（10亿token）过滤而来，主要用于预训练目的。

BetterWeb is a quality-filtered version of the FineWeb dataset that applies ensemble quality scoring. It combines three complementary quality signals: the DCLM FastText quality classifier (measuring general instruction-following quality and reasoning clarity), the FineWeb-Edu educational classifier (trained on 460K LLM-annotated samples), and writing quality heuristics (including vocabulary richness, information density, sentence quality, and structural quality metrics). The dataset includes multiple quality scoring metrics and quality tiers (0-4), with each document having a quality_tier attribute for flexible filtering. It is filtered from FineWeb sample-10BT (10 billion tokens) and is primarily intended for pretraining purposes.