Measuring the Quality of University Lectures: Development and Validation of the Instructional Skills Questionnaire (ISQ)

In higher education, student ratings are often used to evaluate and improve the quality of courses and professors’ instructional skills. Unfortunately, student-rating questionnaires rarely generate specific feedback for professors to improve their instructional skills. The impact of student ratings on professors’ instructional skills has proven to be low. This study concerns the psychometric properties of the Instructional Skills Questionnaire (ISQ), a new theory-based student-rating-of-teaching questionnaire with specific questions concerning lecturing skills. The ISQ is administered after a single lecture. This way, it serves as a formative feedback instrument for university professors during courses to assist them to improve and (re-) evaluate their skills if necessary. The ISQ contains seven dimensions of professors’ instructional skills and three student (self perceived) learning outcomes. In this study, Dutch students in 75 courses rated three 90-minute lectures (T1, T2 and T3) of their respective professors using the ISQ. In total, 14,298 ISQ-forms were used to rate 225 lectures. The teacher level reliabilities of the seven dimensions were found to be good at each measurement occasion. In addition, confirmatory multilevel factor analysis confirmed a seven dimensional factor structure at the teacher level at each measurement occasion. Furthermore, specific teacher level factors significantly predicted students’ (self-assessed) learning outcomes. These results partly supported the proposed theoretical framework on the relationship between the ISQ teaching dimensions and the student learning process, and provided evidence for the construct validity of the instrument. In sum, the ISQ is found to be a reliable and valid instrument, which can be used by professors and faculty development centers to assess and improve university teaching.在高等教育领域，学生评价常被用于评估并提升课程质量与教师的教学技能。遗憾的是，传统学生评价问卷极少能为教师提供可用于改进教学技能的具体反馈信息，且已有研究表明，学生评价对教师教学技能提升的实际作用有限。本研究聚焦于教学技能问卷（Instructional Skills Questionnaire, ISQ）的心理测量学属性，该问卷是一款基于理论构建的新型学生教学评价工具，内含针对授课技能的具体问题项。该问卷在单次授课结束后发放，可作为高校教师在课程周期内的形成性反馈工具，在必要时辅助教师改进并（重新）评估自身教学技能。教学技能问卷涵盖教师教学技能的7个维度，以及3项学生（自我感知的）学习成果指标。本研究中，修读75门课程的荷兰学生采用ISQ，对各自任课教师的3场时长90分钟的授课（分别标记为T1、T2、T3）进行评价。本次研究共回收有效ISQ问卷14298份，用于评价225场授课活动。各测量时点下，7个维度的教师内部一致性信度均表现良好。此外，验证性多层因子分析结果证实，在各测量时点下，教师层面均存在稳定的七维因子结构。进一步分析显示，教师层面的特定因子可显著预测学生（自我评估的）学习成果。上述结果部分验证了ISQ教学维度与学生学习过程间关系的理论框架，并为该工具的结构效度提供了实证支持。综上，教学技能问卷（ISQ）被证实为一款具备良好信效度的评价工具，可被教师与教师发展中心用于评估并改进高校教学工作。