Recycling Data Game_Waste Annotations

FORTH_db_export files: A dataset of users' annotations of RGB images, collected as part of the Recycling Data Game ( https://play.google.com/store/apps/details?id=com.InstituteofDigitalGames.ReclaimDataGame&hl=en&pli=1 ). The dataset is produced as part of the EU funded project RECLAIM: GA-101070524. This dataset contains the raw files, i.e. user annotations for every image in each annotation challenge, provided in JSON format across seven files. The examples below illustrate the structure of each file: Detect: {"IMAGE_NAME": {"MATERIAL": [{"user_id": "USER_1", "user_response": 0, "date": "2024-03-12 10:47:12"}]}Count: {"IMAGE_NAME": {"MATERIAL": [{"user_id": "USER_1", "user_response": N, "date": "2024-03-12 10:47:12"}]}Locate: {"IMAGE_NAME": {"MATERIAL": [{"user_id": "USER_1", "user_response": [X1, Y1, X2, Y2], "date": "2024-03-12 10:47:12"}]}Outline: {"IMAGE_NAME": {"MATERIAL": [{"user_id": "USER_1", "user_response": [X_MIN1, Y_MIN1, X_MAX1, Y_MAX1, X_MIN2, Y_MIN2, X_MAX2, Y_MAX2], "date": "2024-03-12 10:47:12"}]}Choose: {"IMAGE_NAME": {"MATERIAL": { "user_responses": [{"user_id": "USER_1", "user_response": "DETECTION_PROCESS_ID", "date": "2024-03-12 10:47:12"}], "detection_processes": {"DETECTION_PROCESS_ID": [[MASK_1], ...]}}}Categorize: {"IMAGE_NAME": {"user_responses": [{"user_id": "USER_1", "user_response": {"MASK_1_ID": "PET", "MASK_2_ID": "PET", "MASK_3_ID": "PET", "MASK_4_ID": "GLASS"}, "date": "2024-03-08 18:02:09"}], "detection_processes": {"DETECTION_PROCESS_ID": {"MASK_1_ID": [coordinates], "MASK_2_ID": [coordinates], "MASK_3_ID": [coordinates], "MASK_4_ID": [coordinates]}}Paint: {"IMAGE_NAME": {"MATERIAL": [{"user_id": "USER_1", "user_response": “FILE_NAME.npz”, "date": "2024-03-12 10:47:12"}]}For the Paint challenge an additional directory named “paint_response_files” is generated and provided. This directory contains paint game responses serialized to .npz format, and names of files are referenced in JSON export for Paint. IDG_processed_report files: A dataset of users' annotations of RGB images, collected as part of the Recycling Data Game ( https://play.google.com/store/apps/details?id=com.InstituteofDigitalGames.ReclaimDataGame&hl=en&pli=1 ). This captured the game's database state on the final date in the file's suffix. The dataset is produced as part of the EU funded project RECLAIM: GA-101070524. This dataset contains summary reports on a per-challenge, per-image basis. The summary reports also consider, where possible, the ground truth and the percentage of annotations that confirm this ground truth. We calculate ground truths for each image based on the majority answer extracted from user annotations for each of the image annotation challenges. The results are presented as a .csv file for all 7 challenges. Locate, Outline and Paint challenges have additional files with our proposed object masks/bounding boxes.Ground truths are calculated using the same logic and algorithms as game scoring. Namely: Detect - Player responses for this challenge are binary (True/False). To create a ground truth we aggregate all responses and calculate the majority answer Count - Player responses for this challenge are positive integers. To create a ground truth for a single image, material pair we aggregate all responses and calculate the majority answer. Locate - Player responses for this challenge are arrays of X, Y coordinates. To create a ground truth for a single image, material pair we apply kernel density estimation (KDE). The process involves constructing a Gaussian kernel density model using previous responses for calibration. Subsequently, the log-likelihood of the responses is computed and normalised by the highest extant density value. The final result of this process is a heatmap that can be used for object detection Outline - Player responses for this challenge are arrays of object bounding box coordinates. We offer two types of ground truths for this challenge: Pixel map - To create a ground truth for a single image, material pair we sum all responses and normalize the summed array by the number of responses. The final result is a grayscale pixel map that displays player determined object boundaries Computed bounding boxes - To compute bounding boxes, we apply clustering, where we calculate the distance between points using Intersection over Union (IoU). IoU describes the extent of overlap of two boxes and its values range between 0 and 1. A higher IoU value indicates better overlap between two boxes. To use IoU as a distance metric it needs to be inverted, so distance between two points is calculated as 1 - IoU. Since the number of ground truths isn’t known beforehand, we apply the Density-Based Spatial Clustering of Applications with Noise (DBSCAN) algorithm. This algorithm can find clusters of points that are close to each other within a certain distance threshold. Choose - Player responses for this challenge are identifiers of the generated masks that users believe to be the best. To create a ground truth for a single image, material pair we aggregate all responses and calculate the majority answer. Categorize - Player responses for this challenge are material names associated with boundaries of a number of objects. Ground truths are determined for each region separately, by selecting the majority response. Paint - Player responses for this challenge are pixel maps. To create a ground truth for a single image, material pair we sum all responses and normalize the summed array by the number of responses. The final result is a grayscale pixel map that displays player determined object boundaries.