PSCR+ interrater agreement testing scores

In this work, the authors document an expansion of the Public Speaking Competency Rubric (PSCR). First developed in 2012 by Schreiber, et al., the original rubric has only one item related to non-verbal communication. The authors of this work expanded the rubric to include 10 items related to the non-verbal aspects of public speaking and had it critiqued by 10 outside experts. The rubric was tested on recorded speeches given by college students. The expanded rubric, dubbed the PSCR+, has been found to be both valid and reliable. Minimal training is needed to apply the rubric in a classroom setting, and it has the benefit of being useful for both formative and summative assessment. Finally, this rubric is complete enough that it can be used to provide students with detailed feedback regarding their speaking skills without the addition of further notes or comments.

Methods
The authors used informative speeches from a university introduction to public speaking class to test the interrater agreement of the PSCR+. The students had signed waivers that gave permission for speeches to be filmed and shared and the videos were publicly available online. Four raters each scored four speeches to test how well the expanded version of the PSCR instrument fared among a group of raters. Each rubric item is scored from 0–4, with 1 representing "deficient" in the rubric category through 4 reflecting "advanced".

The scores for the four speeches were analyzed for three indices of interrater agreement over j items (rwg(j), awg(j), ADM(j);  (LeBreton & Senter, 2008). The first, rwg, estimates agreement as a proportional reduction in error variance among raters relative to complete agreement and disagreement; this index tends to be the most widely used for estimating agreement. The second, awg, is an extension of rwg that is calculated to correct for possible limitations regarding sample size and number of scale anchors. ADM estimates agreement as the average deviation of scores around the mean for those scores. For the first two indices, scores over .70 are interpreted as demonstrating strong agreement, with .90 or higher representing very strong agreement, and for the third, scores below .80 are interpreted as in high agreement.