Using a new video rating tool to crowd-source analysis of behavioural reaction to stimuli

Quantifying the intensity of animals’ reaction to stimuli is notoriously difficult as classic unidimensional measures of responses such as latency or duration of looking, can fail to capture the overall strength of behavioural responses. More holistic rating can be useful but have the inherent risks of subjective bias and lack of repeatability. Here, we explored whether crowdsourcing could be used to efficiently and reliably overcome these potential flaws. A total of 396 participants watched online videos of dogs reacting to auditory stimuli and provided 23,248 ratings of the strength of the dogs’ responses from zero (default) to 100 using an online survey form. We found that raters achieved very high inter-rater reliability across multiple datasets (although their responses were affected by their sex and attitude towards animals) and that as few as 10 raters could be used to achieve a reliable result. A linear mixed model applied to PCA factors of behaviours discovered that the dogs’ f...