Humans outperform Merlin SoundID in field-based point-count surveys

Point counts are integral to bird population monitoring, and ​multiple methods have been developed to improve their reliability. However, these improved protocols are time-consuming, expensive, and often require hiring surveyors with inconsistent abilities. Merlin Sound ID, an automated bird-sound classifier that can run in the field on mobile devices, could be a part of the solution to the problems associated with point counts by functioning as a no-cost observer with consistent performance. Here, we compare the accuracy of Merlin and human observers in 144 paired field-based point-counts. Our goals were to evaluate how the species identified, number of individual detections, and precision for humans and Merlin compared. We also examined the consistency of Merlin across different devices. Human observers reported 382 individual detections over the course of the study, 72% more than the 222 detections reported by Merlin. Precision was similar with 92% and 86% for humans and Merlin, respectively. Merlin detected four species missed by humans, but also detected 12 false positive species. Merlin disagreed with itself on 57% of point counts when running on two different devices at the same time. We propose Merlin not as a replacement for human point counters, but as an aid to increase detection probability of point-count surveys as secondary observers, with humans acting as reviewers and arbitrating disagreements between devices when needed. 

This study required no ethical permits.  

Methods
This study took place at the Schoodic Forest in Winter Harbor, Maine, USA (Figure 1). This 687.97 hectare forest parcel is owned and managed by Maine Coast Heritage Trust and co-managed with Schoodic Institute. The Schoodic Forest property is a coastal spruce-fir forest with abundant Acer rubrum (Red Maple) that experienced partial harvest last in the late 1990s. The land is used by the local community for hunting, hiking, and other recreation. The survey was conducted along Curtis Road, a small gravel road that is now predominantly maintained as an ATV trail and access road and is the only road through the property.

The survey route consisted of 10 points along the unpaved Curtis Road, starting ~500 m off from the paved road (Figure 2). Due to turns in the road, not all points were separated by 500 m, but no two points were closer than 300 m. Surveys were conducted beginning at point one and continuing in increasing order along Curtis Road to point 10 over the course of one day, except one instance, where we began the survey at point 10 and continued in decreasing order to point one. We conducted these surveys from July to October in 2023, and visited all 10 points during each weekly survey, unless conditions deteriorated, to where detection probability would be impaired as indicated in the National Parks Service Breeding Landbird Monitoring Volunteer Resource Manual (Faccio et al. 2024). This happened on one survey day, when we only conducted counts at points 1-4 before ending the survey. Whenever possible, surveys were scheduled in accordance with ideal weather conditions. Each point count consisted of a one-minute settlement period, during which no data were collected, when observers stood silently to allow their presence to minimize disturbance on bird behavior. At the end of this one minute, we began a three-minute point count, where a human observer auditorily and visually identified birds alongside another observer operating Merlin on a handheld device (see Merlin Sound ID Software and Recorder Hardware for device make and models). Observers stood within 1-2 m of each other. The Merlin operator aimed the microphone of their handheld device away from themselves and remained still for the three minutes to minimize sound interference. The human observer did not consult with the Merlin operator during the count regarding birds identified by the software. Occasionally there were two Merlin operators on separate devices during the same count. There were two human observers over the course of the study, both deemed intermediately skilled as they were considered level two birders according to the standards outlined in the Birder Certification Online program (Howe & Giese 2021). All point counts were conducted between 1100 and 1800 UTC.

Six device types were used over the course of the study: Apple iPhone SE, Samsung Galaxy Tab A, Apple iPhone 12 Pro, Apple iPhone 14, Apple iPhone 13 Mini, and Apple iPhone 13. Three versions of the Merlin software (ver. 2.1.8, 3.0, and 3.0.1) were used during the four-month study period on Apple iPhones, and one version (ver. 2.1.4) on the Samsung Galaxy. To our knowledge, no changes to the classifier used in the Sound ID feature were made during the duration of the study. Over the course of the study (144 point counts), we generated 214 three-minute audio files, accounting for point counts where we had two devices running Merlin at the same time. Points 1-4 had 22 audio files each, while points 5-10 had 21 each, resulting in 642 minutes of audio in total.