WOZ Uncertainty Adaptation

Most existing tutoring systems respond based on the correctness of student answers. Although the tutoring community has argued that student incorrectness and uncertainty both represent learning impasses (and thus opportunities to learn), and has also shown correlations between uncertainty and learning, to date very few controlled experiments have investigated whether system responses to student uncertainty improve learning. Thus, this controlled experiment tests whether this hypothesis holds true, under "ideal" system conditions.

This study uses a Wizard of Oz (WOZ) version of a qualitative physics spoken dialogue tutoring system, called ITSPOKE, which shares technology with TuTalk. This version of ITSPOKE tutors one qualitative physics problem involving basic physics concepts (e.g. Newton's Second Law) student and tutor interact via spoken dialogue that has a (fixed) Tutor Question - (expected) Student Answer format. A human "Wizard" performs speech recognition, natural language understanding, and recognition of uncertainty, for each student answer.

This study has 3 conditions. In the experimental condition, the Wizard tells the system that all correct but uncertain student answers are incorrect, thereby causing the system to respond to both uncertain and incorrect student answers in the same way, namely with further dialogue to reinforce the student's understanding of the principle(s) under discussion. In the first control condition, the system responds only to incorrect student answers in this way. In the second control condition, the system also responds to a percentage of correct answers in this way, to control for the additional tutoring in the experimental condition.

Data collection for this experiment completed in May, 2007. Because
this was a small 8-month experiment, we use an existing WOZ infrastructure and
measure only normal learning, with the expectation that a larger subsequent study measuring robust learning can be performed in either the Physics LearnLab or in other LearnLabs developing spoken dialogue systems.