Evaluation of an automated acoustic beaked whale detection algorithm using multiple validation and assessment methods

"Currently, the acoustic detection of beaked whales during passive acoustic surveys requires trained acousticians to identify beaked whale signals with the aid of various software programs. The development of reliable automated detection and classification methods will enable passive acoustic approaches to better meet monitoring needs for real-time mitigation of industry and military impacts. During ongoing development of automated beaked whale detectors and classifiers it will be important for researchers at different institutions to utilize standardized metrics of performance. At the Southwest Fisheries Science Center (SWFSC), automated detection algorithms for Cuvier's beaked whale (Ziphius cavirostris) and Baird's beaked whale (Berardius bairdii) were developed using PAMGUARD software (Douglas Gillespie: www.pamguard.org). To evaluate the performance of these beaked whale detectors, 15 ten-minute recording segments were processed in PAMGUARD, and the resulting signal detections were compared to manual logs of beaked whale signals confirmed by an experienced acoustician. The comparison was conducted using three methods: precise timestamp matching between manual and automated detections, detection counts from one-minute time bins, and binary presence/absence detection classification of one-minute bins. The detections were scored as true positive, false positive, false negative or false classification. Detector efficacy was quantified using measures developed for information retrieval systems (precision, recall, and F-score) as well as the Receiver Operating Characteristic. Calculated performance scores were compared across evaluation methods. We found that the method used to evaluate detector functionality greatly influences the resulting performance scores and subsequently our perception of detector ability. Therefore, it will be important for researchers to clearly communicate methods and results of detector evaluation. To allow for greatest precision and applicability to different recording datasets, we recommend that beaked whale detectors be evaluated using timestamp matching between manual and automated detections in trial datasets and that F-scores be used to compare detectors. This approach avoids problems associated with binning datasets by eliminating the need for a measure of false negatives"--Abstract. Eiren K. Jacobson, Tina M. Yack, Jay Barlow. "March 2013." System requirements: Adobe Acrobat Reader. Includes bibliographical references (pages 17-19).