Measuring Word Correctness in Young Initial Readers: Comparing Assessments from Teachers, Phoneticians, and ASR Models

Personalized reading instruction requires continuing, up-to-date assessment through word and text reading tests. Current tests administered by teachers who have to assess for each read word whether it is correct or incorrect are time-consuming, mentally demanding, and error-prone. Automatic assessment through ASR technology would be welcome, but needs to be investigated. In this study, we analyzed speech recordings of six different first graders (7-8 years old) reading 144 words which were assessed by 51 teachers, three phoneticians, a Kaldi-based ASR model and an end-to-end ASR model. We investigated to what extent the teachers agree, how to optimally interpret the ASR output, and how readings with phonetic transcriptions that deviate from the prompt are assessed by the teachers and the ASR models. The analysis scripts can be found on GitHub: https://github.com/WiekeHarmsen/SLaTE23-WordCorrectness.
In this data collection you can find a CSV document with the following information for each read aloud prompt word: 1) binary correctness assessments by 51 teachers; 2) phonetic transcription and binary correctness assessment by 3 phoneticians; 3) binary correctness assessments by two different ASR systems (Kaldi-based and End2End). The audio data is not included.