Replication Data for: Crowd sourcing remote comparative lameness assessments for dairy cattle

Lameness assessments are rarely conducted routinely on dairy farms and when completed typically underestimate lameness prevalence, hampering early diagnosis and treatment. A well-known feature of many perceptual tasks is that relative assessments are more accurate than absolute assessments, suggesting that creating methods that allow for the relative scoring of ‘which cow is more lame’ will allow for reliable lameness assessments. Here we developed and tested a remote comparative lameness assessment method: we recruited non-experienced crowd workers via an online platform and asked them to watch two videos side-by-side, each showing a cow walking, and to identify which cow was more lame and by how much (on a scale of -3 to 3). We created 11 tasks, each with 10 video pairs for comparison, and recruited 50 workers per task. All tasks were also completed by 5 experienced cattle lameness assessors. We evaluated data filtering and clustering methods based on worker responses and determined the agreement among workers, among experienced assessors, and between these groups. A moderate to high interobserver reliability was observed (intraclass correlation coefficient, ICC=0.46 to 0.77) for crowd workers and agreement was high among the experienced assessors (ICC=0.87). Average crowd worker responses showed excellent agreement with the average of experienced assessor responses (ICC= 0.89 to 0.91), regardless of data processing method. To investigate if we could use fewer workers per task while still retaining high agreement with experienced assessors, we randomly sub-sampled 2 to 43 (1 less than the minimum number of workers retained per task after data cleaning) workers from each task. The agreement with experienced assessors increased substantially as we increased the number of workers from 2 to 10, but little increase was observed after 10 or more workers were used (ICC>0.80). The proposed method provides a fast and cost-effective way to assess lameness in commercial herds. In addition, this method allows for large-scale data collection useful for training computer vision algorithms that could be used to automate lameness assessments on farm.

奶牛场极少常规开展跛行（lameness）评估，即便完成评估，通常也会低估跛行患病率，进而阻碍早期诊断与治疗。诸多感知任务的公认特性之一，是相对评估相较于绝对评估更为准确，据此，开发能够对“哪头奶牛跛行更严重”开展相对评分的方法，即可实现可靠的跛行评估。本研究开发并验证了一种远程比较性跛行评估方法：我们通过在线平台招募无相关经验的众包工作者，要求其并排观看两段分别展示单头奶牛行走的视频，并判断哪头奶牛跛行更严重，同时给出严重程度评分（评分区间为-3至3）。本研究共设置11项评估任务，每项任务包含10组待比较的视频对，并为每项任务招募50名工作者。所有任务同时由5名经验丰富的奶牛跛行评估者完成。我们基于工作者的应答数据对数据过滤与聚类方法进行了评估，并分析了工作者组内、经验丰富的评估者组内，以及这两组人群之间的一致性水平。结果显示，众包工作者的观察者间信度为中等至较高水平（组内相关系数（intraclass correlation coefficient, ICC）=0.46~0.77）；经验丰富的评估者组内一致性较高（ICC=0.87）。无论采用何种数据处理方法，众包工作者的平均应答结果与经验丰富的评估者的平均应答结果均具有极佳的一致性（ICC=0.89~0.91）。为探究在减少每项任务的工作者数量的同时，仍可保持与经验丰富的评估者较高一致性的可行性，我们从每项任务中随机抽取2至43名工作者（该数值为数据清洗后保留的每项任务的最少工作者数量减1）。当工作者数量从2增加至10时，与经验丰富的评估者之间的一致性显著提升；但当工作者数量达到10及以上时，一致性未出现明显提升（ICC>0.80）。本研究提出的方法可为商业牛群的跛行评估提供一种快速且经济高效的途径。此外，该方法可支持大规模数据采集，可用于训练可自动化实现奶牛场跛行评估的计算机视觉算法。