Scaled and Translated Image Recognition (STIR)

<strong>Paper:</strong> [2211.10288] Just a Matter of Scale? Reevaluating Scale Equivariance in Convolutional Neural Networks (arxiv.org)<br> <strong>Code:</strong> taltstidl/scale-equivariant-cnn: Official code for "Just a Matter of Scale? Reevaluating Scale Equivariance in Convolutional Neural Networks" (github.com) While convolutions are known to be invariant to (discrete) translations, scaling continues to be a challenge and most image recognition networks are not invariant to them. To explore these effects, we have created the Scaled and Translated Image Recognition (STIR) dataset. This dataset contains objects of size \(s \in [17,64]\), each randomly placed in a \(64 \times 64\) pixel image. <strong>Using the dataset</strong> Depending on which data you are planning to use, download one or more of the following files. Data is stored in compressed <code>.npz</code> format and can be loaded as documented here. File Description <code>emoji.npz</code> Emoji vector icons rendered as white icon on black background <code>mnist.npz</code> Classic MNIST handwritten digits rescaled to varying sizes <code>trafficsign.npz</code> Traffic signs from street imagery downscaled to varying sizes <code>aerial.npz</code> Objects in aerial imagery downscaled to varying sizes Each file contains multiple arrays that can be accessed in a dictionary-like fashion. The keys are documented below, where <code>n</code> is the number of classes for a given file and <code>m</code> is the number of instances for each class. Both <code>emoji.npz</code> (36 classes, 1 instance) and <code>mnist.npz</code> (10 classes, 50 instances) are in black &amp; white while <code>trafficsign.npz</code> (16 classes, 25 instances) and <code>aerial.npz</code> (9 classes, 25 instances) are in color. Key Shape Description <code>imgs</code> <code>(3, 48, n, m, 64, 64)</code> black &amp; white, <code>(3, 48, n, 64, 64, 3)</code> color Images grouped into 3 sets (training, validation, testing) and 48 different scales. Values will be in range <code>0</code> to <code>255</code>. <code>lbls</code> <code>(3, 48, n, m)</code> Indices referencing ground truth labels. See <code>lbldata</code> for descriptive names. Values will be in range <code>0</code> to <code>n - 1</code>. <code>scls</code> <code>(3, 48, n, m)</code> Known scales as given by bounding box size. Values will be in range <code>17</code> to <code>64</code>. <code>psts</code> <code>(3, 48, n, m, 2)</code> Known position of bounding box. First value is distance to left edge, second value distance to top edge. <code>metadata</code> <code>(6, 2)</code> Metadata on title, description, author, license, version and date. <code>lbldata</code> <code>(n,)</code> Descriptive names for each ground truth labels. For use in Python a dataset class is provided that implements the basic functionality for loading a certain split and scale selection, as illustrated in the code below. It ensures shuffling is done in a consistent manner such that ground truth scales and positions can be retrieved. Metadata and label descriptions can be retrieved via <code>metadata</code> and <code>labeldata</code>, respectively. <pre><code class="language-python">from data.dataset import STIRDataset dataset = STIRDataset('data/emoji.npz') # Obtain images and labels for training images, labels = dataset.to_torch(split='train', scales=[32, 64], shuffle=True) # Obtain known scales and positions for above scales, positions = dataset.get_latents(split='train', scales=[32, 64], shuffle=True) # Get metadata and label descriptions metadata = dataset.metadata label_descriptions = dataset.labeldata</code></pre> <strong>License and Attribution</strong> When using this dataset for your own research, please respect the individual licenses of the original data. These are distributed within the data files' metadata. For attribution in papers, we recommend the following citations. D. Gandy, J. Otero, E. Emanuel, F. Botsford, J. Lundien, K. Jackson, M. Wilkerson, R. Madole, J. Raphael, T. Chase, G. Taglialatela, B. Talbot, and T. Chase. Font Awesome. https://fontawesome.com/v5/download, Nov. 2022. Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. <em>Proc. IEEE</em>, 86(11):2278–2324, Nov. 1998. C. Ertler, J. Mislej, T. Ollmann, L. Porzi, G. Neuhold, and Y. Kuang. The Mapillary Traffic Sign Dataset for Detection and Classification on a Global Scale. In <em>2020 16th Eur. Conf. Comput. Vision (ECCV)</em>, Glasgow, UK, Aug. 2020. G.-S. Xia, X. Bai, J. Ding, Z. Zhu, S. Belongie, J. Luo, M. Datcu, M. Pelillo, and L. Zhang. DOTA: A Large-Scale Dataset for Object Detection in Aerial Images. In <em>2018 IEEE/CVF Conf. Comput. Vision and Pattern Recognition (CVPR)</em>, pages 3974–3983, Salt Lake City, UT, USA, June 2018.