Collected dataset for DEBAR

We share our two collected datasets and evaluation results online.

The first dataset is a set of 9 buggy architectures collected by existing studies. The buggy architectures come from two studies: eight architectures were collected by a previous empirical study on TensorFlow bugs (Github/StackOverflow-IPS-id.pbtxt) and one architecture was obtained from the study that proposes and evaluates TensorFuzz (TensorFuzz.pbtxt).

The second dataset contains 48 architectures from a large collection of research projects in TensorFlow Models repository. Overall, our second dataset contains a great diversity of neural architectures like CNN, RNN, GAN, HMM, and so on. Please note that we have no knowledge about whether the architectures in this dataset contain numerical bugs when collecting the dataset.

For every architecture in two datasets, we extract the computation graph by using a TensorFlow API. Each extracted computation graph is represented by a Protocol Buffer file, which provides the operations (nodes) and the data flow relations (edges).

我们在线公开了所收集的两类数据集与评估结果。
第一类数据集包含从已有研究中收集的9个存在缺陷的神经网络架构。这些缺陷架构源自两项相关研究：其中8个架构来自一项针对TensorFlow缺陷的前期实证研究（对应文件为Github/StackOverflow-IPS-id.pbtxt），剩余1个架构则取自提出并评估TensorFuzz的相关研究（对应文件为TensorFuzz.pbtxt）。
第二类数据集包含从TensorFlow Models代码仓库的大量研究项目中收集的48个神经网络架构。整体而言，该数据集涵盖了丰富多样的神经网络架构类型，例如卷积神经网络（CNN）、循环神经网络（RNN）、生成对抗网络（GAN）、隐马尔可夫模型（HMM）等。需要说明的是，在收集该数据集时，我们无法确认其中的架构是否存在数值型缺陷。
针对两类数据集中的每一个架构，我们均通过TensorFlow应用程序编程接口（TensorFlow API）提取其计算图。每个提取得到的计算图均以协议缓冲区（Protocol Buffer）文件的形式存储，该文件包含了计算操作（节点）与数据流关联（边）的相关信息。