ProteinBERT Trained model

Trained ProteinBERT model weights for the paper "ProteinBERT: A universal deep-learning model of protein sequence and function". https://github.com/nadavbra/protein_bert Also available via FTP: ftp://ftp.cs.huji.ac.il/users/nadavb/protein_bert/epoch_92400_sample_23500000.pkl ProteinBERT is a protein language model pretrained on ~106M proteins from UniRef90. The pretrained model can be fine-tuned on any protein-related task in a matter of minutes. ProteinBERT achieves state-of-the-art performance on a wide range of benchmarks. ProteinBERT is built on Keras/TensorFlow. ProteinBERT's deep-learning architecture is inspired by BERT, but contains several innovations such as global-attention layers that have linear complexity for sequence length (compared to self-attention's quadratic/n^2 growth). As a result, the model can process protein sequences of almost any length, including extremely long protein sequences (of over tens of thousands of amino acids). The model takes protein sequences as inputs, and can also take protein GO annotations as additional inputs (to help the model infer about the function of the input protein and update its internal representations and outputs accordingly). This pretrained Tensorflow/Keras model was produced by training for 28 days over ~670M records (~6.4 epochs over the entire UniRef90 training dataset of ~106M proteins).本研究配套论文《ProteinBERT：一种通用的蛋白质序列与功能深度学习模型》所使用的训练后ProteinBERT模型权重，其项目仓库地址为：https://github.com/nadavbra/protein_bert，同时可通过FTP获取：ftp://ftp.cs.huji.ac.il/users/nadavb/protein_bert/epoch_92400_sample_23500000.pkl。 ProteinBERT是一款基于UniRef90数据库中约1.06亿条蛋白质序列预训练得到的蛋白质语言模型（protein language model）。该预训练模型可在数分钟内完成微调，适配各类蛋白质相关任务，并在大量基准测试中取得了当前最优性能（state-of-the-art）。 ProteinBERT基于Keras/TensorFlow框架构建，其深度学习架构灵感源自BERT，但引入了多项创新设计，例如针对序列长度具有线性复杂度的全局注意力层（相较自注意力（self-attention）的二次方/n²复杂度增长）。该设计使得模型能够处理几乎任意长度的蛋白质序列，包括长达数万个氨基酸残基的超长序列。 该模型以蛋白质序列作为输入，同时支持将蛋白质基因本体（Gene Ontology, GO）注释作为额外输入，以辅助模型推断输入蛋白质的功能，并相应更新其内部表征与输出结果。 本次发布的预训练TensorFlow/Keras模型经过约28天的训练，覆盖约6.7亿条数据记录，在总规模约1.06亿条蛋白质的UniRef90训练数据集上完成了约6.4个训练轮次。