tomekkorbak/pile-curse-full

该数据集是通过从The Pile文档中使用LDNOOBW词表进行评分构建的，评分标准是每个字符的诅咒词数量。生成过程包括两个步骤：首先从The Pile中随机抽取100k个文档并评分，然后从整个The Pile中选出评分最高的100k个文档。最后，数据集被随机打乱并按9:1的比例划分为训练集和测试集。基本统计信息显示，平均评分为0.013，中位数为0.019。

This dataset was constructed by scoring documents sourced from The Pile using the LDNOOBW lexicon, where the scoring metric is the count of curse words per character. The construction process involves two key steps: first, 100,000 documents were randomly sampled from The Pile and scored; second, the top 100,000 highest-scoring documents were selected from the full corpus of The Pile. Finally, the dataset was randomly shuffled and split into training and test subsets at a 9:1 ratio. Basic statistical analysis reveals that the mean score is 0.013 and the median score is 0.019.

原始信息汇总

数据集构建过程

1. 数据来源：数据集基于the Pile的文档构建。
2. 评分方法：使用LDNOOBW词表对文档进行评分，评分标准为每字符的诅咒词数量。
3. 数据划分：
   • 前半部分：随机从the Pile中抽取100,000份文档并赋予评分。
   • 后半部分：从the Pile中选取评分最高的100,000份文档。
4. 数据处理：数据集经过洗牌处理，并进行了9:1的训练集与测试集分割。

基本统计数据

• 平均评分：0.013
• 中位数评分：0.019