Transformers Model Zoos and Soups: A Population of Language and Vision Models

Model Zoos submitted to the NeurIPS 2024 Dataset & Benchmark track: "Transformer Model Zoos and Soups: A Population of Language and Vision Models" We generate two model zoos, one for computer vision built on the ViT-S architecture, and one for language modeling based on the BERT architecture. For each, we train several backbone models with varying hyperparameters, and further fine-tune them using multiple hyperparameter combinations. We further annotate every model with performance metrics. These include test accuracy and F1-score, as well as the generalization gap. For the vision models, we also include the robust accuracy after a FGSM attack.