Supplementary File (JAP Letter to Editor)

Supplementary data and analysis file for letter to editor regarding a paper in Journal of Applied Physiology - (Caldwell et al., 2025). Importantly, the analyses presented in the document file involve deriving the difference in response variances between treatment vs control group variances (and thereby deriving the SDir) for a parallel-group trial. Mills et al. (2021) used the abbreviation DiV for this difference in variances statistic. The SDir is simply the square root of this DiV (an advantage being that the original units are retained for interpretation purposes). Crucially, Mills et al. (2021) reported both a “by hand” calculation and a modelling approach for deriving the DiV. Mills et al. (2021) termed the modelling approach "a linear model with non-constant variance (LMNCV)". Both these "by hand" and modelling approaches in Mills et al. (2021) are the same as those reported previously and separately for the SDir (Atkinson and Batterham, 2015; Atkinson et al., 2019). Caldwell et al. make false claims about variance comparison statistics in general, and conveniently but falsely focus these claims only on the SDir. They decided not to critique Mills et al. (2021) for covering the by-hand and model-derived estimates of variance difference (DiV), which is the basis of the SDir. To counter another of the many unjustified claims in Caldwell et al., and just like any other variance comparison statistic, the SDir is not restricted for use only on pre-post changes. It can be used, for example, to compare variances at follow-up. Mills et al. (2021) did exactly this when they modelled and calculated the DiV using some previously-published trial data (follow-up variances). As is the case for the mean treatment effect in a parallel group trial, when the outcome is measured on a continuous, ratio scale, and with a sufficiently large sample size, then the same estimate of variance difference is obtained whether one uses the pre-post changes or the post-only values as the outcome as long as a baseline-adjusted ANCOVA linear mixed model is applied (detailed in our file). We have one comment about the modelling approach in Mills et al. (2021). They stated that to derive the difference in response variances using a linear mixed model ‘the software must allow the variances to be negative.’ Like most statistics software packages (other than SAS with the ‘nobound’ option), Stata has no option to remove the constraint that variances must always be positive. In the context of treatment response heterogeneity, it is useful to remove this constraint to better illustrate and embrace the uncertainty in its estimation. We can do this manually when constructing the confidence interval for the difference in the variance of the change scores. Conventional practice (reproducing an analysis in SAS with the ‘nobound’ option) is to simply multiply the standard error (SE) by 1.96, and then calculate confidence limits in the usual way. To derive the confidence limits for the SDir, we take the square root of the calculated confidence limits for DiV. For the negative lower limit, we first ignore the sign, take the square root, and then replace the sign, giving a 95% confidence interval. Note that the confidence intervals are not symmetric in the metric of the SDir but are for the variances. References: Atkinson G, Batterham AM. True and false interindividual differences in the physiological response to an intervention. Exp Physiol 100(6): 577–588, 2015. https://doi.org/10.1113/ep08507 Atkinson G, Williamson P, Batterham AM. Issues in the determination of “responders” and “non-responders” in physiological research. Exp Physiol 104(8): 1215–1225, 2019. https://doi.org/10.1113/ep087712 Mills HL, Higgins JPT, Morris RW, Kessler D, Heron J, Wiles N, Davey Smith G, Tilling K. Detecting heterogeneity of intervention effects using analysis and meta-analysis of differences in variance between trial arms. Epidemiology 32(6): 846–854, 2021. https://doi.org/10.1097/ede.0000000000001401