BiXT model weights (Perceiving Longer Sequences with Bi-Directional Cross-Attention Transformers)

Name: BiXT model weights (Perceiving Longer Sequences with Bi-Directional Cross-Attention Transformers)
Creator: The University of Melbourne
Published: 2025-03-10T07:32:20+08:00

DataCite Commons2025-03-10 更新2025-04-17 收录

计算机视觉

深度学习

数据链接：

https://figshare.unimelb.edu.au/articles/dataset/_b_BiXT_model_weights_b_Perceiving_Longer_Sequences_with_Bi-Directional_Cross-Attention_Transformers_/28561820 数据链接链接失效反馈

官方服务：

资源简介：

BiXT model weightsThis collection includes PyTorch weights of various BiXT models trained on the ImageNet dataset, as introduced in the paper: Markus Hiller, Krista A. Ehinger, and Tom Drummond. "Perceiving Longer Sequences With Bi-Directional Cross-Attention Transformers." The Thirty-eighth Annual Conference on Neural Information Processing Systems (NeurIPS), 2024. (available here)These weights are in support of the following github repository containing all code: https://github.com/mrkshllr/BiXT/ In detail, weights for the following models are available: ImageNet Classification ModelsDefault BiXT-Tiny models with 64 latents:BiXT-Ti/16: bixt_ti_l64_p16.zipBiXT-Ti/8: bixt_ti_l64_p16s8.zipBiXT-Ti/4: bixt_ti_l64_p16s4.zipPrevious BiXT-Tiny models fine-tuned on larger 384x384 images:BiXT-Ti/16-ft384: bixt_ti_l64_p16_ft384.zipBiXT-Ti/8-ft384: bixt_ti_l64_p16s8_ft384.zipBiXT-Ti/4-ft384: bixt_ti_l64_p16s4_ft384.zipConvolutional Alternative: BiXT-Tiny w/ conv-tokeniser:BiXT-Ti/16 (conv): bixt_conv_ti_l64_p16.zipSlightly larger models with embedding dimension of 256 instead of 192 (default tiny):BiXT-d256/16: bixt_ed256_l64_p16.zipBiXT-d256/8: bixt_ed256LS_l64_p16s8.zipBiXT-d256/8-ft384: bixt_ed256LS_l64_p16s8_ft384.zip Models for Dense Downstream TasksNote that for standard ImageNet training, we simply use a standard classification loss on the average-pooled latent embeddings for training. This means that for a 12 layer BiXT network, the refined patch tokens only receive a gradient until layer 11 -- which is why we employ only a one-sided cross-attention for the last layer (see BiXT model file here).For simplicity and easy transfer to dense downstream tasks, we therefore simply create and train BiXT-models with a depth of 13 and train these on ImageNet (see here); Afterwards, the last one-sided cross-attention that exclusively refines the latent vectors is simply discarded and the remaining (fully-trained) 12-layer network is used for finetuning on downstream tasks.Note: It is, of course, entirely possible to replace or extend our simple classification loss on the averaged latent vectors through other token-side losses (e.g. Masked Image Modelling) to provide a gradient signal for the token side and thereby directly train both, the latent and token refinement for all layers.Dense (d13) BiXT-Tiny models with 64 latents:BiXT-Ti/16 (d13): bixt_ti_l64_d13_p16.zipBiXT-Ti/8 (d13): bixt_ti_l64_d13_p16s8.zipBiXT-Ti/4 (d13): bixt_ti_l64_d13_p16s4.zipPrevious dense (d13) BiXT-Tiny models fine-tuned on larger 384x384 images:BiXT-Ti/16-ft384 (d13): bixt_ti_l64_d13_p16_ft384.zipBiXT-Ti/8-ft384 (d13): bixt_ti_l64_d13_p16s8_ft384.zipDense Convolutional Alternative: BiXT-Tiny (d13) w/ conv-tokeniser:BiXT-Ti/16 (conv, d13): bixt_conv_ti_l64_d13_p16.zipBiXT-Ti/8 (conv, d13): bixt_conv_ti_l64_d13_p8.zip NoteIn case you need any further information or model files, please make sure to check out the code in the github repository here and/or reach out! Additional information as well as details on the performance of each pre-trained model is also provided in the repository.

BiXT模型权重本集合包含在ImageNet数据集上训练的多种BiXT模型的PyTorch权重，相关模型已在论文中介绍：Markus Hiller、Krista A. Ehinger与Tom Drummond合著的《基于双向交叉注意力Transformer感知更长序列》 第三十八届神经信息处理系统年会（NeurIPS），2024年。（可在此处获取）这些权重用于支持包含所有代码的以下GitHub仓库：https://github.com/mrkshllr/BiXT/ 具体而言，以下模型的权重可供使用：ImageNet分类模型含64个潜在变量的默认BiXT-Tiny模型：BiXT-Ti/16：bixt_ti_l64_p16.zipBiXT-Ti/8：bixt_ti_l64_p16s8.zipBiXT-Ti/4：bixt_ti_l64_p16s4.zip先前在更大的384×384图像上微调的BiXT-Tiny模型：BiXT-Ti/16-ft384：bixt_ti_l64_p16_ft384.zipBiXT-Ti/8-ft384：bixt_ti_l64_p16s8_ft384.zipBiXT-Ti/4-ft384：bixt_ti_l64_p16s4_ft384.zip卷积替代方案：含卷积标记器的BiXT-Tiny模型：BiXT-Ti/16（conv）：bixt_conv_ti_l64_p16.zip嵌入维度为256（而非默认tiny模型的192）的稍大模型：BiXT-d256/16：bixt_ed256_l64_p16.zipBiXT-d256/8：bixt_ed256LS_l64_p16s8.zipBiXT-d256/8-ft384：bixt_ed256LS_l64_p16s8_ft384.zip 密集下游任务模型需要注意的是，在标准ImageNet训练中，我们仅对平均池化的潜在嵌入使用标准分类损失进行训练。这意味着对于12层BiXT网络，经过优化的patch标记仅在第11层前接收梯度——因此我们仅对最后一层采用单侧交叉注意力（详见此处的BiXT模型文件）。为简化操作并便于迁移至密集下游任务，我们构建并训练了深度为13的BiXT模型，并在ImageNet上对其进行训练（详见此处）；随后，仅用于优化潜在向量的最后一层单侧交叉注意力被直接舍弃，剩余的（已完全训练的）12层网络则用于下游任务的微调。注：当然，完全可以通过其他标记侧损失（如掩码图像建模）替换或扩展我们在平均潜在向量上使用的简单分类损失，为标记侧提供梯度信号，从而直接训练所有层的潜在向量和标记优化。含64个潜在变量的密集型（d13）BiXT-Tiny模型：BiXT-Ti/16（d13）：bixt_ti_l64_d13_p16.zipBiXT-Ti/8（d13）：bixt_ti_l64_d13_p16s8.zipBiXT-Ti/4（d13）：bixt_ti_l64_d13_p16s4.zip先前在更大的384×384图像上微调的密集型（d13）BiXT-Tiny模型：BiXT-Ti/16-ft384（d13）：bixt_ti_l64_d13_p16_ft384.zipBiXT-Ti/8-ft384（d13）：bixt_ti_l64_d13_p16s8_ft384.zip密集卷积替代方案：含卷积标记器的BiXT-Tiny（d13）模型：BiXT-Ti/16（conv，d13）：bixt_conv_ti_l64_d13_p16.zipBiXT-Ti/8（conv，d13）：bixt_conv_ti_l64_d13_p8.zip 注如需更多信息或模型文件，请务必查看此处GitHub仓库中的代码，或直接联系我们！仓库中还提供了每个预训练模型的额外信息及性能细节。

提供机构：

The University of Melbourne

创建时间：

2025-03-10