Evaluating Dialogue Systems via an Opinion

Dialogue systems are a significant field of research and development in artificial intelligence. Until today, the evaluation of such algorithms happens in one fundamental way. They solve "hypothetical problems," i.e., dialog systems are tested by being asked to respond in specific scenarios and provide a "solution", i.e. a reply, to a "problem". The replies are either explicitly compared to reference responses using overlap-based metrics (e.g., BLEU) or are evaluated by human annotators, which can also be seen as compared to (implicit) references. Instead, we propose to ask dialogue systems to tell whether a sample "solution" to a sample "problem" is good or bad. In other words, we ask another dialog system whether a conversation is fluent and coherent or not, and to what degree. In our experiments, we show how to evaluate dialogue systems by "asking for an opinion" and that it indeed offers an additional perspective on assessing these methods.