Model Zoo: A Dataset of Diverse Populations of Neural Network Models - SVHN

Abstract In the last years, neural networks have evolved from laboratory environments to the state-of-the-art for many real-world problems. Our hypothesis is that neural network models (i.e., their weights and biases) evolve on unique, smooth trajectories in weight space during training. Following, a population of such neural network models (refereed to as “model zoo”) would form topological structures in weight space. We think that the geometry, curvature and smoothness of these structures contain information about the state of training and can be reveal latent properties of individual models. With such zoos, one could investigate novel approaches for (i) model analysis, (ii) discover unknown learning dynamics, (iii) learn rich representations of such populations, or (iv) exploit the model zoos for generative modelling of neural network weights and biases. Unfortunately, the lack of standardized model zoos and available benchmarks significantly increases the friction for further research about populations of neural networks. With this work, we publish a novel dataset of model zoos containing systematically generated and diverse populations of neural network models for further research. In total the proposed model zoo dataset is based on six image datasets, consist of 24 model zoos with varying hyperparameter combinations are generated and includes 47’360 unique neural network models resulting in over 2’415’360 collected model states. Additionally, to the model zoo data we provide an in-depth analysis of the zoos and provide benchmarks for multiple downstream tasks as mentioned before. Dataset This dataset is part of a larger collection of model zoos and contains the zoos trained on the labelled samples from SVHN. All zoos with extensive information and code can be found at www.modelzoos.cc. This repository contains two types of files: the raw model zoos as collections of models (file names beginning with "svhn_"), as well as preprocessed model zoos wrapped in a custom pytorch dataset class (filenames beginning with "dataset"). Zoos are trained in three configurations varying the seed only (seed), varying hyperparameters with fixed seeds (hyp_fix) or varying hyperparameters with random seeds (hyp_rand). The index_dict.json files contain information on how to read the vectorized models. For more information on the zoos and code to access and use the zoos, please see www.modelzoos.cc.Abstract 近年来，神经网络已从实验室环境演进为诸多现实世界问题的前沿解决方案。我们提出的核心假设为：神经网络模型（即其权重与偏置）在训练过程中，会在权重空间内沿独特且平滑的轨迹进行演化。由此，这类神经网络模型的集合（被称为“模型动物园（model zoo）”）将在权重空间中形成拓扑结构。我们认为，这些结构的几何特性、曲率与平滑度蕴含着训练状态相关的信息，且能够揭示单个模型的潜在属性。借助这类模型动物园，研究者可探索多种全新研究路径：(i) 开展模型分析；(ii) 挖掘未知的学习动态；(iii) 学习该模型集合的丰富表征；(iv) 利用模型动物园实现神经网络权重与偏置的生成式建模。但遗憾的是，缺乏标准化的模型动物园与可用基准集，大幅提升了神经网络集合相关后续研究的门槛。本研究发布了一个全新的模型动物园数据集，其中包含系统生成的多样化神经网络模型集合，以供后续研究使用。本数据集基于6个图像数据集构建，共生成24个包含不同超参数组合的模型动物园，涵盖47360个独特的神经网络模型，累计收集的模型状态超过2415360个。此外，除模型动物园数据外，我们还提供了对各模型动物园的深度分析，并为前文提及的多项下游任务提供了基准测试集。 Dataset 本数据集属于更大规模的模型动物园集合的一部分，包含基于街景房屋号码数据集（SVHN）的标注样本训练得到的模型动物园。所有附带详细信息与代码的模型动物园均可在www.modelzoos.cc获取。 本仓库包含两类文件：一是以模型集合形式存储的原始模型动物园（文件名以"svhn_"开头），二是封装于自定义PyTorch数据集类中的预处理后模型动物园（文件名以"dataset"开头）。模型动物园的训练共包含三种配置：仅随机种子不同（seed配置）、固定随机种子但超参数不同（hyp_fix配置），以及随机种子与超参数均不同（hyp_rand配置）。index_dict.json文件包含如何读取向量化模型的相关说明。 如需了解更多关于模型动物园的信息，以及获取与使用模型动物园的代码，请访问www.modelzoos.cc。