EuroSAT Model Zoo: A Dataset of Diverse Populations of Neural Network Models - EuroSAT

<strong>Abstract</strong> In the last years, neural networks have evolved from laboratory environments to the state-of-the-art for many real-world problems. Our hypothesis is that neural network models (i.e., their weights and biases) evolve on unique, smooth trajectories in weight space during training. Following, a population of such neural network models (refereed to as “model zoo”) would form topological structures in weight space. We think that the geometry, curvature and smoothness of these structures contain information about the state of training and can be reveal latent properties of individual models. With such zoos, one could investigate novel approaches for (i) model analysis, (ii) discover unknown learning dynamics, (iii) learn rich representations of such populations, or (iv) exploit the model zoos for generative modelling of neural network weights and biases. Unfortunately, the lack of standardized model zoos and available benchmarks significantly increases the friction for further research about populations of neural networks. With this work, we publish a novel dataset of model zoos containing systematically generated and diverse populations of neural network models for further research. In total the proposed model zoo dataset is based on six image datasets, consist of 27 model zoos with varying hyperparameter combinations are generated and includes 50’360 unique neural network models resulting in over 2’585’360 collected model states. Additionally, to the model zoo data we provide an in-depth analysis of the zoos and provide benchmarks for multiple downstream tasks as mentioned before. <strong>Dataset</strong> This dataset is part of a larger collection of model zoos and contains the zoos trained on EuroSAT. All zoos with extensive information and code can be found at www.modelzoos.cc. This repository contains two types of model populations: the base model zoo ("eurosat_cnn_kaiming_uniform.zip"), as well as a collection of sparsified model zoos (filenames ending in "magn_XX.zip" or "ard.zip"). Zoos are trained with CNN models in configurations varying the seed only (seed), and sparsification is done through magnitude-based weight pruning ("magn_XX.zip") or varational dropout ("ard.zip"). For more information on the zoos and code to access and use the zoos, please see www.modelzoos.cc.**摘要** 近年来，神经网络已从实验室环境逐步演进，成为众多现实世界问题的前沿解决方案。本研究提出如下假设：神经网络模型（即其权重与偏置）在训练过程中，会在权重空间中沿独特且平滑的轨迹演化。据此，此类神经网络模型的集合（被称为“模型动物园（model zoo）”）将在权重空间中形成拓扑结构。我们认为，这些结构的几何特性、曲率与平滑度蕴含了训练状态的相关信息，且能够揭示单个模型的潜在属性。借助此类模型动物园，研究者可探索全新的研究路径：(i) 开展模型分析；(ii) 发掘未知的学习动力学特性；(iii) 学习此类模型集合的丰富表征；或(iv) 利用模型动物园实现神经网络权重与偏置的生成式建模。然而，目前缺乏标准化的模型动物园与可用基准数据集，大幅提升了神经网络模型集合相关后续研究的门槛。本研究公开了一套全新的模型动物园数据集，其中包含系统生成的多样化神经网络模型集合，以供后续研究使用。本数据集基于6个图像数据集，生成了27个包含不同超参数组合的模型动物园，涵盖50360个独特的神经网络模型，总计收集了超过2585360个模型状态。此外，除模型动物园数据集之外，本研究还提供了对模型动物园的深入分析，并为前文提及的多项下游任务提供基准测试方案。**数据集** 本数据集为更大规模模型动物园集合的一部分，包含基于EuroSAT数据集训练得到的模型动物园。所有附带详细信息与代码的模型动物园均可通过www.modelzoos.cc获取。本仓库包含两类模型集合：基础模型动物园（文件名为"eurosat_cnn_kaiming_uniform.zip"），以及一系列经过稀疏化处理的模型动物园（文件名以"magn_XX.zip"或"ard.zip"）。基础模型动物园使用仅调整随机种子（seed）的卷积神经网络（Convolutional Neural Network, CNN）训练得到；稀疏化处理则通过基于权重幅值的剪枝（对应"magn_XX.zip"）或变分 dropout（variational dropout，对应"ard.zip"）完成。如需了解更多关于模型动物园的相关信息，以及访问和使用模型动物园的代码，请访问www.modelzoos.cc。