replicAnt - Plum2023 - 3D Models - Unreal Engine 5

This dataset contains the 3D models used to generate all synthetic data presented in the <em>replicAnt - generating annotated images of animals in complex environments using Unreal Engine </em>manuscript. The models have been generated with the open-source photogrammetry platform <em>scAnt</em> peerj.com/articles/11155/ and have been pre-processed and converted into Unreal Engine 5 compatible .uasset files, to be used with the associated <em>replicAnt</em> project available from https://github.com/evo-biomech/replicAnt. <strong>Abstract:</strong> Deep learning-based computer vision methods are transforming animal behavioural research. Transfer learning has enabled work in non-model species, but still requires hand-annotation of example footage, and is only performant in well-defined conditions. To overcome these limitations, we created <em>replicAnt</em>, a configurable pipeline implemented in Unreal Engine 5 and Python, designed to generate large and variable training datasets on consumer-grade hardware instead. <em>replicAnt</em> places 3D animal models into complex, procedurally generated environments, from which automatically annotated images can be exported. We demonstrate that synthetic data generated with <em>replicAnt</em> can significantly reduce the hand-annotation required to achieve benchmark performance in common applications such as animal detection, tracking, pose-estimation, and semantic segmentation; and that it increases the subject-specificity and domain-invariance of the trained networks, so conferring robustness. In some applications, <em>replicAnt</em> may even remove the need for hand-annotation altogether. It thus represents a significant step towards porting deep learning-based computer vision tools to the field. <strong>Funding</strong> This study received funding from Imperial College’s President’s PhD Scholarship (to Fabian Plum), and is part of a project that has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation program (Grant agreement No. 851705, to David Labonte). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.