California under the microscope: an evaluation of the practical effect of different approaches to accountability

Restricted until 2 April 2009. The purpose of this study was to examine five practical approaches to educational accountability that are utilized or are within the current constraints of California's accountability system. Various approaches were compared and analyzed as potential comparative assessments for elementary schools that serve a large population of English language learners (ELL). The study specifically compared and analyzed the interpretability, reliability, fairness, and the validity of: (a) State decile ranks; (b) Similar-schools rank; (c) Similar-schools difference scores; (d) Academic Performance Index (API) for English learners; and (e) Residualized difference scores.; The population included elementary schools that had API Base and Growth data for any of the 2003-2005 school years with at least 130 students with valid test scores. For the API for English learners and the Residualized difference score approaches, the sample was inclusive of schools that had a separate ELL API in 2005 and 2006.; Results of the study found state decile ranks, which are utilized in California, to be strongly correlated to socioeconomic status and thus were of limited utility as measures of accountability for schools that have high a proportion of ELLs. Similar-schools rank, which is the state's recommended method in evaluating schools against those that are similar in demographic characteristics, was a more equitable method for evaluating schools that serve ELLs. An analysis of the API for English learners, which was recently added by the state, showed that it was a promising method for comparing schools with a large population of ELLs.; Two other comparative assessments within the current constraints of California's accountability system were examined (Similar-schools difference scores and Residualized difference scores). Results of the evaluation of comparative assessments suggest Residualized difference scores to be the best method in evaluating schools that serve a large number of ELLs.; This study only examined approaches to accountability that are currently being utilized or are feasible within the constraints of California's accountability system. Further research is still necessary to enhance the accuracy of school-level-achievement data especially in schools that serve a large population of English learners.本资料于2009年4月2日前处于保密限制状态。本研究旨在考察当前加州教育问责体系中已采用或符合其现行规范的五种实用教育问责方法。本研究针对服务大量英语学习者（English Language Learners, ELL）的小学，将各方法作为潜在对比评估工具展开比较与分析，重点比较分析了以下五种方法的可解释性、信度、公平性与效度：(a) 州层级十分位排名；(b) 同质性学校排名；(c) 同质性学校差异得分；(d) 英语学习者学业表现指数（Academic Performance Index, API）；(e) 残差化差异得分。研究样本涵盖2003至2005学年任意一学年拥有API基准值与增长数据、且至少有130名学生拥有有效考试成绩的小学。针对英语学习者API与残差化差异得分这两种方法，研究样本额外纳入了2005年与2006年拥有单独ELL API数据的学校。研究结果显示，加州现行的州层级十分位排名与社会经济地位强相关，因此在服务高比例ELL学生的学校问责评估中，其应用价值较为有限。同质性学校排名作为加州推荐的评估方法，即基于人口统计学特征匹配同类学校进行对比评估，在服务ELL学生的学校评估中展现出更强的公平性。针对加州新近增设的英语学习者API的分析表明，该方法在对比服务大量ELL学生的学校时颇具应用前景。本研究还考察了加州问责体系现行规范下的另外两种对比评估工具：同质性学校差异得分与残差化差异得分。评估结果显示，残差化差异得分是服务大量ELL学生的学校的最优评估方法。本研究仅考察了加州教育问责体系中现行采用或符合其规范的问责方法。未来仍需开展进一步研究，以提升学校层面的学业成就数据准确性，尤其针对服务大量ELL学生的学校。