Lightning Pose dataset: mirror-fish

Weakly-electric Mormyrid fish (of the species Gnathonemus petersii) swim freely in and out of an experimental tank, capturing worms from a well. The tank has a side mirror and a top mirror, both at 45◦, providing three different views seen from a single camera at 300 Hz. 17 body parts are labeled across all three views for a total of 51 keypoints.
The labeled data are partitioned into disjoint "in-distribution" (InD) and "out-of-distribution" (OOD) sets. Each set contains different sessions / animals. The InD data contain 373 labeled frames, and 28 unlabeled videos; the OOD data contain 94 labeled frames, and 10 unlabeled videos.
For the InD data only, we use a probabilistic variant of multi-view PCA (PPCA) to infer keypoints that are occluded in one out of the three views, effectively similar to the triangulation-reprojection protocols used for multi-view tracking by Dunn et al., 2021 and Chettih et al., 2024. This results in a 30% increase in the number of keypoints usable for training, with more occluded keypoints included in the augmented label set.
Data collected by Federico Pedraja, David Ehrlich, and Dillon Noone in the Sawtell Lab, Columbia University.