Common Corpus
收藏Toxic Commons 数据集概述
数据集简介
Toxic Commons 是一个用于过滤大型语言模型（LLM）预训练数据的框架，由 PleIAs 创建。该数据集和相关模型 Celadon 是论文 Toxicity of the Commons: Curating Open Source Pre-Training Data 的研究成果。Celadon 模型旨在更高效地分类有毒数据，以节省计算资源和时间，用于更大规模的模型训练。
数据集结构
Toxic Commons 数据集包含以下主要部分：
- Celadon 模型：用于初步过滤数据，评估数据的毒性水平。
- 数据分类：根据 Celadon 模型的输出，数据被分为三类：
- 无毒性：总分在 0-3 之间，且没有任何单项得分超过 2。
- 轻度毒性：总分在 4-6 之间，或总分为 3 且仅有一个单项得分为 3。
- 有毒内容：总分达到 7 或更高，需要进一步审查。
- LLM 标注：使用 Llama 3.1 8B Instruct 模型对数据进行进一步标注和重写，以减少有害内容。
数据集使用
安装
Celadon 模型兼容 HuggingFace Transformers，安装步骤如下：
- 下载模型：
git clone https://huggingface.co/PleIAs/celadon - 导入模型类：
from celadon.model import MultiHeadDebertaForSequenceClassification - 导入 AutoTokenizer：
from transformers import AutoTokenizer - 加载 tokenizer：
tokenizer = AutoTokenizer.from_pretrained("celadon") - 加载模型：
model = MultiHeadDebertaForSequenceClassification.from_pretrained("celadon")
使用示例
以下是一个使用 Celadon 模型分类单个样本的示例脚本： python from transformers import AutoTokenizer from celadon.model import MultiHeadDebertaForSequenceClassification
tokenizer = AutoTokenizer.from_pretrained("celadon") model = MultiHeadDebertaForSequenceClassification.from_pretrained("celadon") model.eval()
sample_text = "This is an example of a normal sentence"
inputs = tokenizer(sample_text, return_tensors="pt", padding=True, truncation=True) outputs = model(input_ids=inputs[input_ids], attention_mask=inputs[attention_mask])
categories = [Race/Origin, Gender/Sex, Religion, Ability, Violence] predictions = outputs.argmax(dim=-1).squeeze().tolist()
打印每个类别的分类结果
print(f"Text: {sample_text}") for i, category in enumerate(categories): print(f"Prediction for Category {category}: {predictions[i]}")
引用
@article{arnett2024toxicity, title={{Toxicity of the Commons: Curating Open-Source Pre-Training Data}}, author={Arnett, Catherine and Jones, Eliot and Yamshchikov, Ivan P. and Langlais, Pierre-Carl}, journal={arXiv preprint arXiv:2410.22587}, url={https://arxiv.org/pdf/2410.22587}, year={2024} }
许可证
MIT License
Copyright (c) 2024 pleias
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
