WhichDog: A crowdsourced dataset including candidate set-based labelling

A dataset with crowdsourced labels for aggregation and supervised classification. It contains 400 images of dogs from the Stanford Dogs dataset (http://vision.stanford.edu/aditya86/ImageNetDogs/). Images of dogs that belong to 32 different breeds (classes) are included. Annotators were asked to provide two types of labelling: full labelling (each labeler is allowed to provide a single label for each image) and candidate labelling (each labeler is allowed to provide a set of candidate labels for each image). It includes a total of 61227 annotations (30628 full and 30599 candidate) obtained from a set of 1028 different labelers. The labels were collected through the online crowdsourcing platform Amazon mTurk thanks to funds provided by the Basque Government through the Elkartek program (KK-2018/00071). The assignments were designed as sequences of 64 images that were given to the annotators. Each image in the sequence was provided together with a specific subset of possible labels (with the number of options ranging from 4 to 32), and a instruction for the annotator to perform a specific type of labelling (full or candidate). Each labeler performed at least one assignment. Not all the labelers completed the 64 annotations in their assignments. The file 'whichdog.zip' contains a folder ('images') with the 400 images of dogs, a text file ('breed_names.txt') that indicates the names of the different breeds and their assigned label (a number in the interval from 0 to 31) and a CSV file ('whichdog_all_annots.csv') that contains the information about the annotations. Each row of the CSV file represents a single annotation, and each column shows: - image_id: ID number of the image. - is_candidate: indicates whether the requested labelling is full (0) or candidate (1). - labeler_id: ID number of the labeler. - time: time employed by the labeler to perform the annotation. - answer: label or set of labels provided by the labeler as annotation. - options: subset of possible labels shown to the labeler. - assignment_id: ID number of the assignment - sequence_point: number that indicates the point of the sequence of images of the assignment in which the annotation was provided. - class: ground truth label of the image.