EleutherAI/dclm-dedup_20250227-004105-filters-only

该数据集包含了多个字段，用于表示文本过滤的相关信息。具体包括：唯一标识符id，是否为目标过滤的is_filter_target字段，以及表示单词过滤、BERT过滤和组合过滤的布尔字段。每个过滤字段还可能包含额外的元数据，例如单词过滤的关键词和BERT过滤的得分及触发段。数据集分为训练集，共有约4亿多个示例，大小约为30GB。同时，提供了默认配置，指定了训练数据的文件路径。

The dataset consists of multiple fields related to text filtering. It includes an id field for unique identification, a boolean field is_filter_target indicating whether it is a target for filtering, and boolean fields for word filtering, BERT filtering, and combined filtering. Each filter field may also contain additional metadata, such as keywords for word filtering and score and triggered passage for BERT filtering. The dataset is split into a training set with approximately 409 million examples and is about 30GB in size. Additionally, a default configuration is provided specifying the file path for the training data.