DataProvenanceInitiative/Everything

Data Provenance Initiative - Everything数据集是一个旨在解决当前语言模型训练中数据透明度和理解不足问题的数据集。它提供了关于数据许可、来源、出处以及细粒度特征（如语言、文本领域、主题、用途、收集时间和任务组成）的详细元数据。数据集包含多个子数据集，每个子数据集都有详细的描述和来源链接。该数据集由机器学习和法律领域的专家共同编制，旨在为未来的语言模型开发提供更透明和负责任的数据支持。

The Data Provenance Initiative - Everything dataset is a collection of large natural language datasets used to train recent language models. The purpose of this dataset is to address the crisis in data transparency and understanding by providing the most detailed and reliable metadata available, including data licenses, sources, and provenance, as well as fine-grained characteristics such as language, text domains, topics, usage, collection time, and task compositions. The dataset includes nearly 40 popular instruction (or alignment) tuning collections and releases a suite of open-source tools for downloading, filtering, and examining this training data.