Data from: Correction for bias in meta-analysis of little-replicated studies

1. Meta-analyses conventionally weight study estimates on the inverse of
their error variance, in order to maximize precision. Unbiased variability
in the estimates of these study-level error variances increases with the
inverse of study-level replication. Here we demonstrate how this
variability accumulates asymmetrically across studies in
precision-weighted meta-analysis, to cause undervaluation of the
meta-level effect size or its error variance (the meta-effect and
meta-variance). 2. Small samples, typical of the ecological literature,
induce big sampling errors in variance estimation, which substantially
bias precision-weighted meta-analysis. Simulations revealed that biases
differed little between random- and fixed-effects tests. Meta-estimation
of a one-sample mean from 20 studies, with sample sizes of 3 to 20
observations, undervalued the meta-variance by ~20%. Meta-analysis of
two-sample designs from 20 studies, with sample sizes of 3 to 10
observations, undervalued the meta-variance by 15-20% for the log response
ratio (lnR); it undervalued the meta-effect by ~10% for the standardised
mean difference (SMD). 3. For all estimators, biases were eliminated or
reduced by a simple adjustment to the weighting on study precision. The
study-specific component of error variance prone to sampling error and not
parametrically attributable to study-specific replication was replaced by
its cross-study mean, on the assumption of random sampling from the same
population variance for all studies, and sufficient studies for averaging.
Weighting each study by the inverse of this mean-adjusted error variance
universally improved accuracy in estimation of both the meta-effect and
its significance, regardless of number of studies. For comparison,
weighting only on sample size gave the same improvement in accuracy, but
could not sensibly estimate significance. 4. For the one-sample mean and
two-sample lnR, adjusted weighting also improved estimation of
between-study variance by DerSimonian-Laird and REML methods. For
random-effects meta-analysis of SMD from little-replicated studies, the
most accurate meta-estimates obtained from adjusted weights following
conventionally-weighted estimation of between-study variance. 5. We
recommend adoption of weighting by inverse adjusted-variance for
meta-analyses of well- and little-replicated studies, because it improves
accuracy and significance of meta-estimates, and it can extend the scope
of the meta-analysis to include some studies without variance estimates.