Data from: Habitat complexity alters the strength of sexual selection on female brain size in a livebearing fish

Animals often reproduce in complex environments, which should generate
selection for both enhanced detectability in signaling traits and improved
cognitive processing abilities. However, the extent to which signaling and
cognitive traits have evolved to overcome the challenges of interacting in
complex habitats remains understudied. We examined whether habitat
complexity influences sexual selection in the pygmy halfbeak, Dermogenys
collettei, a small livebearing freshwater fish. Using free-swimming
arenas, we created low- and high-complexity environments and observed
mating behaviors in mixed-sex groups. While the opportunity for sexual
selection did not differ significantly between environments for either
sex, we observed positive selection gradients for female brain size in
open arenas, but not in complex habitats. Selection on morphological
traits associated with visual signaling was also primarily detected in
open environments, particularly in females. These results suggest that
habitat complexity may reduce selection pressures on both cognitive
traits, such as brain size, and signaling traits relevant to mate choice.
Together, our findings highlight the importance of integrating cognitive
traits into sexual selection theory and considering sex-specific selection
across ecologically relevant contexts.