Estimating ancestral states of complex characters: A case study on the evolution of feathers

Feathers are a key novelty underpinning the evolutionary success of birds, yet the origin of feathers remains poorly understood. Debates about feather evolution hinge upon whether filamentous integument has evolved once or multiple times independently in the lineage leading to modern birds. These contradictory results stem from methodological differences in statistical ancestral state estimates. Here, we conduct a comprehensive comparison of ancestral state estimation methodologies applied to stem-group birds, testing the role of outgroup inclusion, tree time scaling method, model choice, and character coding strategy. Models are compared based on their Akaike Information Criteria (AIC), mutual information, as well as the uncertainty of marginal ancestral state estimates. Our results demonstrate that ancestral state estimates of stem-bird integument are strongly influenced by tree time scaling method, outgroup selection, and model choice, while character coding strategy seems to have less effect on the ancestral estimates produced. We identify the best-fitting and most generalizable models using AIC scores and leave-one-out cross-validation (LOOCV), respectively. Our analyses broadly support the independent origin of filamentous integument in dinosaurs and pterosaurs and support a younger evolutionary origin of feathers than has been suggested previously. In terms of model selection, we observe little correlation between AIC/AICc and LOOCV error, suggesting that, for our dataset, model fit does not reliably predict generalizability. However, both approaches favor models that infer a similar pattern of feather evolution. More globally, our study highlights that special care must be taken in selecting the outgroup, tree, and model when conducting ASE analyses.