Improving the Design of Evaluations that Include Students, Teachers, and Schools: An Empirical Investigation of Key Design Parameters

The
learning environment is a complex environment for conducting evaluations. Students,
teachers, and schools all play a role in shaping the environment, which can
lead to differences in student achievement. Hence designing an evaluation with
the capacity to detect the effectiveness of an educational intervention on
student achievement requires careful planning. A key part of the planning is a
statistical power analysis that considers the multilevel nature of the design. This
study provides empirical estimates of design parameters necessary for planning
adequately powered cluster randomized trials that include the student, teacher,
and school level and are focused on reading, mathematics, or science
achievement. The sample in our study includes administrative state datasets
from Michigan, North Carolina, Kentucky, and Maryland for grades 3 through 8. The
results showed that, with few exceptions, the variance in student test scores
is larger between teachers within schools than between schools.