What exactly is random about random effects?

The central question of the paper is: When should stimuli be treated as random effects in random- or mixed-effects ANOVA? The conceptual starting point is the view of significance testing in experimental research as following the logic of randomization tests. Three designs are discussed: (1) Stimuli nested under treatment conditions, (2) stimuli and treatment conditions crossed, and (3) stimuli and treatment conditions counterbalanced. The treatment of stimuli as random effects in the first two designs is shown to be inadequate but is necessary in the third design. Moreover, the analysis has implications for at least two additional topics: (1) The conceptualization of the significance test as a randomization test when no randomization takes place, e.g., in quasi-experiments and correlational studies, and (2) the validity of aggregating over subjects and stimuli (referred to as aggregation validity). Aggregation validity has to be distinguished from internal validity and is shown to be one aspect of the concept formerly known as external validity, albeit without appeal to some underlying population. unknown publishedVersion