Measuring agreement among experts in classifying camera images of similar species

Camera trapping and solicitation of wildlife images through citizen science have become common tools in ecological research. Such studies collect many wildlife images for which correct species classification is crucial; even low misclassification rates can result in erroneous estimation of the geographic range or habitat use of a species, potentially hindering conservation or management efforts. However, some species are difficult to tell apart, making species classification challenging - but the literature on classification agreement rates among experts remains sparse. Here, we measure agreement among experts in distinguishing between images of two similar congeneric species, bobcats (Lynx rufus) and Canada lynx (L. canadensis). We asked experts to classify the species in selected images to test whether the season, background habitat, time of day, and the visible features of each animal (e.g., face, legs, tail) affected agreement among experts about the species in each image. Overall, experts had moderate agreement (Fleiss’ kappa = 0.64), but experts had varying levels of agreement depending on these image characteristics. Most images (71%) had ≥1 expert classification of ‘unknown’, and many images (39%) had some experts classify the image as ‘bobcat’ while others classified it as ‘lynx’. Further, experts were inconsistent even with themselves, changing their classifications of numerous images when they were asked to reclassify the same images months later. These results suggest that classification of images by a single expert is unreliable for similar-looking species. Most of the images did obtain a clear majority classification from the experts, although we emphasize that even majority classifications may be incorrect. We recommend that researchers using wildlife images consult multiple species experts to increase confidence in their image classifications of similar sympatric species. Still, when the presence of a species with similar sympatrics must be conclusive, physical or genetic evidence should be required.

借助公民科学（citizen science）开展红外相机诱捕（camera trapping）与野生动物影像征集工作，现已成为生态学研究的常用工具。此类研究会收集大量野生动物影像，而准确的物种分类是其中的核心环节；即便分类错误率极低，也可能导致物种地理分布范围或栖息地利用情况的估算出现偏差，进而阻碍物种保护与管理工作的推进。然而部分物种外观相似度极高，使得物种分类颇具挑战——但目前关于专家间分类一致性率的相关研究仍较为匮乏。本研究针对两种外观相似的同属物种——短尾猫（Lynx rufus）与加拿大猞猁（L. canadensis）的影像，评估专家间的分类一致性。我们邀请专家对筛选出的影像进行物种分类，以检验季节、背景生境、拍摄时段以及动物可见特征（如面部、四肢、尾巴）是否会影响专家对单张影像的物种分类一致性。整体而言，专家间的分类一致性处于中等水平（弗莱伊斯Kappa系数（Fleiss’ kappa）= 0.64），且一致性程度会随上述影像特征的不同而有所差异。多数影像（71%）存在至少1位专家标注为‘未知’，且有39%的影像出现了部分专家标注为‘短尾猫’、其余专家标注为‘加拿大猞猁’的分歧情况。此外，专家自身也存在分类不一致的情况：在时隔数月后重新对同一批影像进行分类时，多位专家对大量影像的分类结果发生了改变。上述结果表明，针对外观相似的物种，仅依靠单一位专家进行影像分类并不可靠。尽管多数影像确实获得了专家群体的明确多数分类结果，但我们需强调：即便多数分类结果也可能存在错误。我们建议，使用野生动物影像开展研究的人员应咨询多位物种分类专家，以提升对同域分布物种（sympatric species）影像分类结果的可信度。但若需对某物种的存在做出确定性判定（尤其是存在同域分布的相似物种时），则应要求提供实物或遗传学证据。