Standardized Labeling Guidelines and Quantitative Content Analysis of New York State Mesonet Daytime Camera Images

The New York State Mesonet (NYSM, nysmesonet.org) captures and archives images every 5 minutes during the daytime from a suite of 127 surface-based cameras. The images are used to create a hand-labeled dataset of current precipitation conditions visible within the image: 1) Clear, 2) Rain, 3) Snow, or 4) Obstructed. Five Labelers (authors Horan, Wirz, Sutter, Evans, and Radford) performed a series of 2 labeling trials where reliability was assessed using the metric Krippendorff’s alpha (Hayes and and Krippendorff 2007). A website created by Dr. Freelon (Freelon, 2013; Freelon, 2010) is used to calculate metrics after labeling, which include Krippendorff’s alpha. This work follows a framework for Qualitative Content Analysis for trustworthy AI in Earth system science (Wirz et al. 2024), which follows recommendations for generating trust, and trustworthiness, in AI for the environmental sciences (Bostrom et al. 2024). This work also closely follows, and takes inspiration from, the framework set forth in Sutter et al. (2023), which performed a similar methodology for New York State Department of Transportation images. The group of labelers reaches a Krippendorff’s alpha of 0.926 on the second trial across 30 trials, or 150 total decisions. Shared in this dataset is the codebook used to assist in label decision making, examples for each class in the codebook, including difficult cases, images used for inter-coder reliability trials, and spreadsheets with inter-coder reliability results and statistics. For inter-coder reliability trials, the images are supplemented with NYSM data and co-located ASOS data (ASOS Tech. Note, 1998). The purpose of this work is to create a trustworthy dataset using preset labeling techniques to create a large dataset for use in a supervised machine learning model. This model aims to predict current precipitation type (if occurring) and to provide nowcasting output of all 127 NYSM standard sites. Additional image data and standard site data is accessible via nysmesonet.org via request. Included with this dataset are also figures that show the completed distribution of data, which will be used in machine learning. This material is based on work supported by the U.S. National Science Foundation under Grant No. RISE-2019758