Rostlab/ProstT5Dataset

Hugging Face2023-12-04 更新2024-03-04 收录

下载链接：

https://hf-mirror.com/datasets/Rostlab/ProstT5Dataset

下载链接

链接失效反馈

资源简介：

--- dataset_info: features: - name: input_id_x sequence: int64 - name: input_id_y sequence: int64 splits: - name: test num_bytes: 1087504 num_examples: 474 - name: valid num_bytes: 1124160 num_examples: 474 - name: train num_bytes: 65391887792 num_examples: 17070828 download_size: 810671738 dataset_size: 65394099456 license: mit task_categories: - text-generation tags: - biology size_categories: - 10M<n<100M --- # Dataset Card for "ProstT5Dataset" * **Contributors:** Michael Heinzinger and Konstantin Weissenow, Joaquin Gomez Sanchez and Adrian Henkel, Martin Steinegger and Burkhard Rost * **Licence:** MIT ## Table of Contents - [Overview](#overview) - [Dataset Description](#dataset-description) - [Data Collection and Annotation](#data-collection-and-annotation) - [Data Splits](#data-splits) - [Dataset Structure](#dataset-structure) - [Data Fields](#data-fields) - [Data Instances](#data-instances) - [Data Considerations](#data-considerations) - [Social Impact of Dataset](#social-impact-of-dataset) - [Discussion of Biases](#discussion-of-biases) - [Other Known Limitations](#other-known-limitations) - [Licensing Information](#licensing-information) - [Citation Information](#citation-information) - [Contributions](#contributions) ## Overview The ProstT5Dataset is a curated collection of *tokenized* protein sequences and their corresponding structure sequences (3Di). It is derived from the [AlphaFold Protein Structure Database](https://alphafold.ebi.ac.uk/) and includes various steps of clustering and quality filtering. To capture 3D information of the sequence, the [3Di structure string representation](https://www.nature.com/articles/s41587-023-01773-0#Sec2) is leveraged. This format captures the spatial relationship of each residue to its neighbors in 3D space, effectively translating the 3D information of the sequence. The sequence tokens are generated using the [ProstT5 Tokenizer](https://huggingface.co/Rostlab/ProstT5). ## Data Fields - **input_id_x** (3Di Tokens): Corresponding tokenized 3Di structure representation sequences derived from the proteins. - **input_id_y** (Amino Acid Tokens): Tokenized amino acid sequences of proteins. ## Dataset Description ![image/png](https://cdn-uploads.huggingface.co/production/uploads/62c412251f45e8bdb2b05855/BgiKOoFUGjlHDPjbxJWOX.png) We compare basic protein properties (sequence length, amino acid composition, 3Di-distribution) between our dataset (training, validation, test sets) and proteins obtained from the [Protein Data Bank (PDB)](https://www.rcsb.org/). Key findings include similar amino acid distributions across datasets, an overrepresentation of certain 3Di-tokens (d, v, p) and helical structures in AlphaFold2 predictions compared to PDB, and a tendency for shorter protein lengths in this dataset (average 206-238) relative to PDB proteins (average 255). The analysis also highlights the relationship between 3Di states and secondary structures, with a notable distinction in strand-related tokens between datasets. ## Data Collection and Annotation The dataset began with the AlphaFold Protein Structure Database , undergoing a two-step clustering process and one step of quality filtering: 1. *First Clustering:* 214M UniprotKB protein sequences were clustered using MMseqs2, resulting in 52M clusters based on pairwise sequence identity. 2. *Second Clustering:* Foldseek further clustered these proteins into 18.8M clusters, expanded to 18.6M proteins by adding diverse members. 3. *Quality Filtering:* Removed proteins with low pLDDT scores, short lengths, and highly repetitive 3Di-strings. The final training split contains 17M proteins. ## Data Splits Data splits into train, test, and, validation were created by moving whole clusters (after quality filtering - see above), to either of the sets. For validation and test, we only kept representatives to avoid bias towards large families. This resulted in 474 proteins for test, 474 proteins for validation and around 17M proteins for training. ## Citation ``` @article{heinzinger2023prostt5, title={ProstT5: Bilingual language model for protein sequence and structure}, author={Heinzinger, Michael and Weissenow, Konstantin and Sanchez, Joaquin Gomez and Henkel, Adrian and Steinegger, Martin and Rost, Burkhard}, journal={bioRxiv}, pages={2023--07}, year={2023}, publisher={Cold Spring Harbor Laboratory} } ``` ## Tokens to Character Mapping | Amino Acid Representation | 3DI | Special Tokens | |---------------------------|-----------|--------------------| | 3: A | 128: a | 0: \<pad\> | | 4: L | 129: l | 1: \</s\> | | 5: G | 130: g | 2: \<unk\> | | 6: V | 131: v | 148: \<fold2AA\> | | 7: S | 132: s | 149: \<AA2fold\> | | 8: R | 133: r | | | 9: E | 134: e | | | 10: D | 135: d | | | 11: T | 136: t | | | 12: I | 137: i | | | 13: P | 138: p | | | 14: K | 139: k | | | 15: F | 140: f | | | 16: Q | 141: q | | | 17: N | 142: n | | | 18: Y | 143: y | | | 19: M | 144: m | | | 20: H | 145: h | | | 21: W | 146: w | | | 22: C | 147: c | | | 23: X | | | | 24: B | | | | 25: O | | | | 26: U | | | | 27: Z | | |

数据集信息：特征： - 名称：input_id_x，类型：int64序列 - 名称：input_id_y，类型：int64序列划分集： - 名称：测试集（test），字节数：1087504，样本数：474 - 名称：验证集（valid），字节数：1124160，样本数：474 - 名称：训练集（train），字节数：65391887792，样本数：17070828 下载大小：810671738 数据集总大小：65394099456 许可协议：MIT 任务类别： - 文本生成标签： - 生物学样本量区间： - 10M<n<100M # "ProstT5数据集"数据集卡片 * **贡献者：** Michael Heinzinger、Konstantin Weissenow、Joaquin Gomez Sanchez、Adrian Henkel、Martin Steinegger与Burkhard Rost * **许可证：** MIT ## 目录 - [概述](#overview) - [数据集描述](#dataset-description) - [数据收集与标注](#data-collection-and-annotation) - [数据划分](#data-splits) - [数据集结构](#dataset-structure) - [数据字段](#data-fields) - [数据实例](#data-instances) - [数据考量](#data-considerations) - [数据集的社会影响](#social-impact-of-dataset) - [偏差讨论](#discussion-of-biases) - [其他已知局限性](#other-known-limitations) - [许可信息](#licensing-information) - [引用信息](#citation-information) - [贡献者](#contributions) ## 概述本数据集ProstT5Dataset是经过精心整理的**标记化（tokenized）**蛋白质序列及其对应结构序列（3Di）的集合。其数据源为[AlphaFold蛋白质结构数据库（AlphaFold Protein Structure Database）](https://alphafold.ebi.ac.uk/)，并经过多轮聚类与质量过滤处理。为保留序列的三维结构信息，本数据集采用了[3Di结构字符串表示法（3Di structure string representation）](https://www.nature.com/articles/s41587-023-01773-0#Sec2)，该格式可捕获每个残基与其相邻残基在三维空间中的空间关联，从而将蛋白质的三维结构信息转化为可处理的序列形式。序列标记（Token）的生成借助了[ProstT5分词器（ProstT5 Tokenizer）](https://huggingface.co/Rostlab/ProstT5)。 ## 数据字段 - **input_id_x**（3Di标记）：源自蛋白质的标记化3Di结构表示序列。 - **input_id_y**（氨基酸标记）：蛋白质的标记化氨基酸序列。 ## 数据集描述 ![image/png](https://cdn-uploads.huggingface.co/production/uploads/62c412251f45e8bdb2b05855/BgiKOoFUGjlHDPjbxJWOX.png) 本研究对比了本数据集（训练集、验证集与测试集）与[蛋白质数据银行（Protein Data Bank, PDB）](https://www.rcsb.org/)获取的蛋白质的基础属性，包括序列长度、氨基酸组成、3Di分布。分析结果显示：不同数据集的氨基酸分布整体相似；相较于PDB数据库，AlphaFold2预测结果中部分3Di标记（d、v、p）与螺旋结构的占比偏高；本数据集的蛋白质平均长度（206~238）短于PDB数据库的蛋白质（平均255）。此外，分析还揭示了3Di状态与二级结构之间的关联，且不同数据集间与链相关的标记存在显著差异。 ## 数据收集与标注本数据集的原始数据源自AlphaFold蛋白质结构数据库，先后经过两步聚类与一步质量过滤处理： 1. **第一轮聚类**：使用MMseqs2对2.14亿条UniprotKB蛋白质序列进行聚类，基于序列两两同一性得到5200万个聚类簇。 2. **第二轮聚类**：借助Foldseek将上述聚类簇进一步划分为1880万个聚类簇，并通过添加多样化成员将总蛋白数量扩充至1860万条。 3. **质量过滤**：移除pLDDT评分较低、长度过短以及3Di字符串高度重复的蛋白质。最终训练集包含1700万条蛋白质序列。 ## 数据划分本数据集通过将完整聚类簇（经过上述质量过滤后）分配至训练、验证或测试集来完成划分。为避免对大蛋白家族产生偏差，验证集与测试集仅保留聚类代表序列。最终测试集与验证集各包含474条蛋白质序列，训练集则包含约1700万条蛋白质序列。 ## 引用信息 @article{heinzinger2023prostt5, title={ProstT5: Bilingual language model for protein sequence and structure}, author={Heinzinger, Michael and Weissenow, Konstantin and Sanchez, Joaquin Gomez and Henkel, Adrian and Steinegger, Martin and Rost, Burkhard}, journal={bioRxiv}, pages={2023--07}, year={2023}, publisher={Cold Spring Harbor Laboratory} } ## 标记与字符映射表 | 氨基酸表示 | 3DI | 特殊标记 | |---------------------------|-----------|--------------------| | 3: A | 128: a | 0: <pad> | | 4: L | 129: l | 1: </s> | | 5: G | 130: g | 2: <unk> | | 6: V | 131: v | 148: <fold2AA> | | 7: S | 132: s | 149: <AA2fold> | | 8: R | 133: r | | | 9: E | 134: e | | | 10: D | 135: d | | | 11: T | 136: t | | | 12: I | 137: i | | | 13: P | 138: p | | | 14: K | 139: k | | | 15: F | 140: f | | | 16: Q | 141: q | | | 17: N | 142: n | | | 18: Y | 143: y | | | 19: M | 144: m | | | 20: H | 145: h | | | 21: W | 146: w | | | 22: C | 147: c | | | 23: X | | | | 24: B | | | | 25: O | | | | 26: U | | | | 27: Z | | |

提供机构：

Rostlab

原始信息汇总

数据集概述

数据集描述

ProstT5Dataset 是一个经过精心筛选的蛋白质序列及其对应结构序列（3Di）的标记化集合。该数据集源自 AlphaFold Protein Structure Database，并包括多个聚类和质量过滤步骤。为了捕捉序列的3D信息，采用了 3Di结构字符串表示，这种格式有效地翻译了序列的3D信息。序列标记使用 ProstT5 Tokenizer 生成。

数据字段

input_id_x (3Di Tokens): 从蛋白质中派生的标记化3Di结构表示序列。
input_id_y (氨基酸 Tokens): 蛋白质的标记化氨基酸序列。

数据收集和标注

数据集从 AlphaFold Protein Structure Database 开始，经过两步聚类和一步质量过滤：

第一步聚类：使用 MMseqs2 对 214M UniprotKB 蛋白质序列进行聚类，基于成对序列相似性得到 52M 个聚类。
第二步聚类：Foldseek 进一步将这些蛋白质聚类成 18.8M 个聚类，并通过添加多样成员扩展到 18.6M 个蛋白质。
质量过滤：移除低 pLDDT 分数、短长度和高度重复的 3Di 字符串的蛋白质。最终训练集包含 17M 个蛋白质。

数据分割

数据被分割为训练集、测试集和验证集，通过将整个聚类（经过质量过滤）移动到任一集合中。为了防止对大型家族的偏见，验证集和测试集仅保留代表性蛋白质。这导致了 474 个蛋白质用于测试，474 个蛋白质用于验证，约 17M 个蛋白质用于训练。

数据集信息

特征：
- input_id_x：序列类型为 int64
- input_id_y：序列类型为 int64
分割：
- test：字节数为 1087504，示例数为 474
- valid：字节数为 1124160，示例数为 474
- train：字节数为 65391887792，示例数为 17070828
下载大小：810671738 字节
数据集大小：65394099456 字节
许可证：MIT
任务类别：
- 文本生成
标签：
- 生物学
大小类别：
- 10M<n<100M

AI搜集汇总

数据集介绍

构建方式

ProstT5Dataset的构建始于AlphaFold Protein Structure Database，经过两步聚类和一步质量过滤。首先，使用MMseqs2对214M UniprotKB蛋白质序列进行聚类，形成52M个基于成对序列相似性的簇。随后，Foldseek进一步将这些蛋白质聚类为18.8M个簇，并通过添加多样成员扩展至18.6M蛋白质。最后，通过去除低pLDDT评分、短长度和高度重复的3Di字符串的蛋白质，进行质量过滤，最终训练集包含17M蛋白质。

使用方法

ProstT5Dataset主要用于蛋白质序列和结构的双语生成任务。用户可以通过加载数据集中的'input_id_x'和'input_id_y'字段，分别获取3Di结构表示和氨基酸序列的标记化形式。这些数据可用于训练和验证基于Transformer的模型，如ProstT5，以生成或预测蛋白质的结构信息。此外，数据集的MIT许可证允许广泛的研究和应用，促进了生物信息学领域的创新和发展。

背景与挑战

背景概述

在生物信息学领域，蛋白质序列与其三维结构之间的关系一直是研究的核心问题。ProstT5Dataset由Michael Heinzinger等人创建，旨在通过整合AlphaFold Protein Structure Database中的数据，提供一个包含蛋白质序列及其三维结构信息的大规模数据集。该数据集通过两步聚类和质量过滤过程，从214M UniprotKB蛋白质序列中筛选出17M高质量蛋白质，用于训练、验证和测试。ProstT5Dataset的构建不仅有助于推动蛋白质结构预测和功能分析的研究，还为生物信息学领域的模型训练提供了宝贵的资源。

当前挑战

ProstT5Dataset在构建过程中面临多个挑战。首先，从AlphaFold Protein Structure Database中提取和处理大量蛋白质数据需要高效的算法和计算资源。其次，确保数据的质量和一致性，特别是在去除低pLDDT得分、短长度和高度重复的3Di字符串时，需要精细的过滤策略。此外，数据集的划分需避免大族群偏差，确保验证和测试集的代表性。最后，如何有效地将三维结构信息转化为可处理的序列表示，以便于模型训练和预测，也是一个重要的技术难题。

常用场景

经典使用场景

在生物信息学领域，Rostlab/ProstT5Dataset 数据集的经典使用场景主要集中在蛋白质序列与结构的生成和预测任务中。该数据集通过整合 AlphaFold Protein Structure Database 中的蛋白质序列及其对应的 3Di 结构表示，为研究人员提供了一个丰富的资源库。通过训练模型，研究人员可以预测蛋白质的三维结构，从而在药物设计、蛋白质工程等领域中发挥重要作用。

解决学术问题

Rostlab/ProstT5Dataset 数据集解决了生物信息学中长期存在的蛋白质结构预测难题。传统的蛋白质结构预测方法依赖于大量的实验数据，而该数据集通过提供高质量的 3Di 结构表示，使得基于机器学习的预测模型能够更准确地捕捉蛋白质的空间关系。这不仅提高了预测精度，还为蛋白质功能研究提供了新的视角，推动了相关领域的学术进展。

实际应用

在实际应用中，Rostlab/ProstT5Dataset 数据集被广泛用于药物设计和蛋白质工程。通过精确预测蛋白质的三维结构，研究人员可以更有效地设计针对特定蛋白质靶点的药物，从而加速新药的研发过程。此外，该数据集还可用于蛋白质工程，帮助设计具有特定功能的蛋白质，如酶的改造和优化，以满足工业和医疗需求。

数据集最近研究