Supplementary Material (JAP Letter to Editor)

Supplementary data and analysis files for letter to editor regarding a paper in Journal of Applied Physiology - (Caldwell et al., 2025). We present: "Analysis of BP data.doc" - Demonstration of how the DiV and SDir are obtained from a study design specific model using the BP dataset from Gadbury GL, Iyer HK, Allison DB. Evaluating subject-treatment interaction when comparing two treatments. J Biopharm Stat. 2001 Nov;11(4):313-33 "Mills et al comparison.pptx" - to demonstrate congruence bewteen Mills et al and our work regarding how the DiV and SDir are obtained from a statistical model or a calculation. "Caldwell data.jpg" file which shows the compromising identical values of change in 1RM for the majority of participants, both within-person and between-person, rendering this dataset sub-optimal for demonstrating any analysis. "SDD observations.doc" - Demonstration of how an SDD > 0 can falsely indicate the presence of treatment response heterogeneity even when the two trial groups have identical sample distributions and no active treatment is present. N.B. In tjis document, the following sentence is incomplete "Without any additional effect modifier variables, e.g. sex, being incorporated into the scenario, then this cannot explain the perceived TRH, but what about multiplicative effects where larger effects are observed for larger baseline values, and vice versa?". It should have ended with "But there are no multiplicative effects at all in this data simulation". "The SDD and variance Component.doc" - Explanation for the above "SDD observations" trial simulation in the context of trial variance components. Neither the SD of differences (in a crossover trial correlated data context) nor the SDD (in a parallel trial potential correlated outcomes context) provide reliable indications of treatment response heterogeneity. This is because, as the modelled "error term", neither the SDdiff nor the SDD cannot isolate true treatment response heterogeneity from random within-person variability. Some special notes about the model-derived SDir and DiV Importantly, the analyses presented in the document file "Analysis of BP data.doc" involve deriving the difference in response variances between treatment vs control group variances (and thereby deriving the SDir) for a parallel-group trial. Mills et al. (2021) used the abbreviation DiV for this difference in variances statistic (see Powerpoint file "Mills et al comparison.pptx". The SDir is simply the square root of this DiV (an advantage being that the original units are retained for interpretation purposes). Crucially, Mills et al. (2021) reported both a “by hand” calculation and a modelling approach for deriving the DiV. Mills et al. (2021) termed the modelling approach "a linear model with non-constant variance (LMNCV)". Both these "by hand" and modelling approaches in Mills et al. (2021) are the same as those reported previously and separately for the SDir (Atkinson and Batterham, 2015; Atkinson et al., 2019). Caldwell et al. make false claims about variance comparison statistics in general, and conveniently but falsely focus these claims only on the SDir. They decided not to critique Mills et al. (2021) for covering the by-hand and model-derived estimates of variance difference (DiV), which is the basis of the SDir. Caldwell et al completely overlooked that, when trial data are at hand to analyse, the DiV and the SDir are obtained from a statistical model, not "seems akin" to a statistical model (as Caldwell erroneously stated), but an actual statistical model. The variance difference outcome is modelled in a synonymous way as the mean treatment effect is modelled for an RCT, e.g. with baseline covariate adjustment, and this can be undertaken in all popular statistical software packages, e.g., STATA, SPSS. Other variance difference statistics do not have this design-specific modelling advantage. To counter another of the many unjustified claims in Caldwell et al., and just like any other variance comparison statistic, the SDir is not restricted for use only on pre-post changes. It can be used, for example, to compare variances at follow-up. Mills et al. (2021) did exactly this when they modelled and calculated the DiV using some previously-published trial data (follow-up variances). As is the case for the mean treatment effect in a parallel group trial, when the outcome is measured on a continuous, ratio scale, and with a sufficiently large sample size, then the same estimate of variance difference is obtained whether one uses the pre-post changes or the post-only values as the outcome as long as a baseline-adjusted ANCOVA linear mixed model is applied. We have one comment about the modelling approach in Mills et al. (2021). They stated that to derive the difference in response variances using a linear mixed model ‘the software must allow the variances to be negative.’ Like most statistics software packages (other than SAS with the ‘nobound’ option), Stata has no option to remove the constraint that variances must always be positive. In the context of treatment response heterogeneity, it is useful to remove this constraint to better illustrate and embrace the uncertainty in its estimation. We can do this manually when constructing the confidence interval for the difference in the variance of the change scores. Conventional practice (reproducing an analysis in SAS with the ‘nobound’ option) is to simply multiply the standard error (SE) by 1.96, and then calculate confidence limits in the usual way. To derive the confidence limits for the SDir, we take the square root of the calculated confidence limits for DiV. For the negative lower limit, we first ignore the sign, take the square root, and then replace the sign, giving a 95% confidence interval. Note that the confidence intervals are not symmetric in the metric of the SDir but are for the variances. References: Atkinson G, Batterham AM. True and false interindividual differences in the physiological response to an intervention. Exp Physiol 100(6): 577–588, 2015. https://doi.org/10.1113/ep08507 Atkinson G, Williamson P, Batterham AM. Issues in the determination of “responders” and “non-responders” in physiological research. Exp Physiol 104(8): 1215–1225, 2019. https://doi.org/10.1113/ep087712 Mills HL, Higgins JPT, Morris RW, Kessler D, Heron J, Wiles N, Davey Smith G, Tilling K. Detecting heterogeneity of intervention effects using analysis and meta-analysis of differences in variance between trial arms. Epidemiology 32(6): 846–854, 2021. https://doi.org/10.1097/ede.0000000000001401