Titre du document / Document title

Assessing the usability of a dialogue management system designed in the framework of a rapid dialogue prototyping methodology

Auteur(s) / Author(s)

RAJMAN Martin ⁽¹⁾ ; BUI Trung H. ⁽¹⁾ ; RAJMAN Andréa ⁽¹⁾ ; SEYDOUX Florian ⁽¹⁾ ; TRUTNEV Alex ⁽¹⁾ ; QUARTERONI Silvia ⁽¹⁾ ;

Affiliation(s) du ou des auteurs / Author(s) Affiliation(s)

⁽¹⁾ Artificial Intelligence Laboratory, Swiss Federal Institute of Technology Lausanne (EPFL), 1015, Lausanne, SUISSE

Résumé / Abstract

The goal of this contribution is to present how the notion of dialogue management evaluation was integrated in the rapid dialogue prototyping methodology (RDPM), designed and experimented by the authors in the framework of the InfoVox project. We first describe the proposed RDPM. The general idea of this methodology is to produce, for any given application, a quickly deployable dialogue-driven interface and to improve this interface through an iterative process based on Wizard-of-Oz experiments (i.e. dialogue simulations). More precisely, the methodology is decomposed into five consecutive steps, namely: (1) producing a task model for the targeted application; (2) automatically deriving an initial dialogue model (and the associated dialogue-driven interface) from the produced task model; (3) using the generated interface to carry out Wizard-of-Oz experiments to improve the initial dialogue model; (4) carrying out an internal field test to further refine the dialogue model; and (5) carrying out an external field test to evaluate the final dialogue model and dialogue-driven interface. In a second part, we more specifically focus on evaluation. We describe the integration of various evaluation aspects in the design steps corresponding to the internal and external field tests. The internal field test involved 20 internal users (typically the designers of the system, other members of the involved laboratories, willing relatives, etc.) and was used to further adapt the prototype. The various modifications made were validated during a second internal field test with a reduced group of seven users. The external field test involved a larger population of 50 external users (i.e. users that did not have any a priori knowledge of the system), randomly selected in all French speaking Swiss cantons. Its main goal was the final evaluation of the produced dialogue model, dialogue-driven interface, and evaluation procedure. Both the internal and external field tests were carried out over the phone. Finally, we concentrate on the assessment results obtained during the external field test for the InfoVox prototype. We first present the retrospective predominant modality and correlation analyses that were integrated in the external field test and then show how the obtained evaluation data can also be used in a more prospective way to derive useful hints for a better dialogue design (prospective correlation analysis). All the presented concepts are systematically illustrated by examples taken from the experiments that were carried out in the InfoVox project.

Revue / Journal Title

Acta acustica united with acustica   ISSN 1610-1928 

Source / Source

Congrès
ISCA Tutorial and Research Workshop on Auditory Quality of Systems N^o1, Herne , ALLEMAGNE (23/04/2003)
2004, vol. 90, n^o 6 (209 p.)  [Document : 16 p.] (35 ref.), [Notes: Selected papers], pp. 1096-1111 [16 page(s) (article)]

Langue / Language

Anglais

Editeur / Publisher

Hirzel, Stuttgart, ALLEMAGNE  (2001) (Revue)

Mots-clés anglais / English Keywords

AWord ; Prototype tests ; Experimental study ; Correlation analysis ; A priori estimation ; Modeling ; Iterative method ; User interface ; Audio acoustics ; Sound reproduction ; Rapid prototyping ; Man machine dialogue ; Sound quality ;

Mots-clés français / French Keywords

. ; Essai qualification ; Etude expérimentale ; Analyse corrélation ; Estimation a priori ; Modélisation ; Méthode itérative ; Interface utilisateur ; Acoustique audio ; Reproduction son ; Prototypage rapide ; Dialogue homme machine ; Qualité sonore ;

Mots-clés espagnols / Spanish Keywords

APalabra ; Prueba calificación ; Estudio experimental ; Análisis correlación ; Estimación a priori ; Modelización ; Método iterativo ; Interfase usuario ; Reproducción sonido ; Prototipificación rápida ; Diálogo hombre máquina ; Calidad sonora ;

Localisation / Location

INIST-CNRS, Cote INIST : 6827, 35400012637147.0090