Meta-Analyzing International Large-Scale Assessment Data: An Application of the Split, Analyze, and Meta-Analyze (SAM) Approach

International large-scale assessments (ILSAs) provide educationally relevant findings from different populations on a common topic. Researchers can use ILSA data to examine whether an effect size is consistent across the body of data, summarize the relationships among variables, and quantify possible heterogeneity in the data. Common multilevel modeling approaches face computational issues with synthesizing the data from multiple ILSAs or ILSA cycles and cannot quantify explicitly the heterogeneity of effect sizes that are derived from the parameters of a specific model. However, meta-analytic approaches, such as the Split, Analyze, and Meta-analyze (SAM) approach, offer ways to circumvent these issues by bringing together the best features of multilevel modeling and meta-analysis. In this study, we showcase how the SAM approach can be used to synthesize findings from ILSA data, focusing on the Big-Fish-Little-Pond-Effect (BFLPE) in mathematics, that is, the negative contextual effect of classroom mathematics achievement on students’ mathematics selfconcept. We analyzed the data from fourth-grade students across five TIMSS cycles. As part of the SAM approach, we performed multi-group multilevel confirmatory factor analysis (MGMCFA) and two-level MSEM to obtain the BFLPE per country and synthesized the resultant effect sizes via cross-classified random-effects meta-analysis. Our results shed light on the average effect size of the BFLPE, the heterogeneity within and between countries and cycles, and the extent to which country-level variables can explain variation in the BFLPE. Overall, this study illustrates how researchers can use the SAM approach for the analysis and synthesis of research evidence using ILSA data. unknown unknown