Date of Original Version



Technical Report

Abstract or Description

In the recent years, automated speech recognition has been the main drive behind the advent of spoken language interfaces, but at the same time a severe limiting factor in the development of these systems. We believe that increased robustness in the face of recognition errors can be achieved by making the systems aware of their own misunderstandings, and employing appropriate recovery techniques when breakdowns in interaction occur. In
this paper we address the rst problem: the development of an utterance-level con dence annotator for a spoken dialog system. After a brief introduction to the CMU Communicator spoken dialog system (which provided the target platform for the developed annotator), we cast the con dence annotation problem as a machine learning classi cation task, and focus on selecting relevant features and on empirically identifying the best classi cation techniques for this task. The results indicate that signi cant reductions in classi cation error rate can be obtained using several di erent classi ers. Furthermore, we propose a data driven approach to assessing the impact of the errors committed by the con dence annotator on dialog performance, with a view to optimally ne-tuning the annotator. Several models were constructed, and the resulting error costs were in accordance with our intuition. We found, surprisingly, that, at least for a mixed-initiative spoken dialog system as
the CMU Communicator, these errors trade-o equally over a wide operating characteristic range.