Habitat complexity alters the strength of sexual selection on female brain size in a livebearing fish

Animals often reproduce in complex environments, which should generate selection for both enhanced detectability in signaling traits and improved cognitive processing abilities. However, the extent to which signaling and cognitive traits have evolved to overcome the challenges of interacting in complex habitats remains understudied. We examined whether habitat complexity influences sexual selection in the pygmy halfbeak, Dermogenys collettei, a small livebearing freshwater fish. Using free-swimming arenas, we created low- and high-complexity environments and observed mating behaviors in mixed-sex groups. While the opportunity for sexual selection did not differ significantly between environments for either sex, we observed positive selection gradients for female brain size in open arenas, but not in complex habitats. Selection on morphological traits associated with visual signaling was also primarily detected in open environments, particularly in females. These results suggest that habitat complexity may reduce selection pressures on both cognitive traits, such as brain size, and signaling traits relevant to mate choice. Together, our findings highlight the importance of integrating cognitive traits into sexual selection theory and considering sex-specific selection across ecologically relevant contexts.

动物通常在复杂环境中繁衍，这会对信号传递性状的可检测性提升以及认知加工能力的改善产生选择作用。然而，信号传递与认知性状为克服复杂生境中社交互动的挑战所演化的程度，目前仍未得到充分研究。我们以小型卵胎生淡水鱼类矮氏半齿鳉（pygmy halfbeak，*Dermogenys collettei*）为研究对象，探究生境复杂度是否会影响其性选择过程。本研究通过设置自由游动试验池，构建了低复杂度与高复杂度两类生境，并对混合性别群体的交配行为进行观察。尽管两类生境中雌雄个体的性选择机会均无显著差异，但我们观察到，在开放试验池中雌性的脑尺寸存在正向选择梯度，而在复杂生境中则未出现该现象。与视觉信号传递相关的形态性状所受的选择压力，也主要在开放生境中被检测到，且该现象在雌性群体中尤为显著。上述结果表明，生境复杂度可能会降低认知性状（如脑尺寸）以及与配偶选择相关的信号传递性状所受的选择压力。综上，本研究结果凸显了将认知性状纳入性选择理论，并在符合生态背景的情境中考虑性别特异性选择的重要性。