Ebook: Spoken Multimodal Human-Computer Dialogue in Mobile Environments

This book is based on publications from the ISCA Tutorial and Research Workshop on Multi-Modal Dialogue in Mobile Environments held at Kloster Irsee, Germany, in 2002. The workshop covered various aspects of devel- ment and evaluation of spoken multimodal dialogue systems and components with particular emphasis on mobile environments, and discussed the state-- the-art within this area. On the development side the major aspects addressed include speech recognition, dialogue management, multimodal output gene- tion, system architectures, full applications, and user interface issues. On the evaluation side primarily usability evaluation was addressed. A number of high quality papers from the workshop were selected to form the basis of this book. The volume is divided into three major parts which group together the ov- all aspects covered by the workshop. The selected papers have all been - tended, reviewed and improved after the workshop to form the backbone of the book. In addition, we have supplemented each of the three parts by an invited contribution intended to serve as an overview chapter.

---

The ongoing migration of computing and information access from stationary environments to mobile computing devices for eventual use in mobile environments, such as Personal Digital Assistants (PDAs), tablet PCs, next generation mobile phones, and in-car driver assistance systems, poses critical challenges for natural human-computer interaction. Spoken dialogue is a key factor in ensuring natural and user-friendly interaction with such devices which are meant not only for computer specialists, but also for everyday users.

Speech supports hands-free and eyes-free operation, and becomes a key alternative interaction mode in mobile environments, e.g. in cars where driver distraction by manually operated devices may be a significant problem. On the other hand, the use of mobile devices in public places, may make the possibility of using alternative modalities possibly in combination with speech, such as graphics output and gesture input, preferable due to e.g. privacy issues. Researchers’ interest is progressively turning to the integration of speech with other modalities such as gesture input and graphics output, partly to accommodate more efficient interaction and partly to accommodate different user preferences.

This book:

combines overview chapters of key areas in spoken multimodal dialogue (systems and components, architectures, and evaluation) with chapters focussed on particular applications or problems in the field.

focusses on the influence of the environment when building and evaluating an application.

Audience:

Computer scientists, engineers, and others who work in the area of spoken multimodal dialogue systems in academia and in the industry.

Graduate students and Ph.D. students specialising in spoken multimodal dialogue systems in general, or focusing on issues in these systems in mobile environments in particular

---

The ongoing migration of computing and information access from stationary environments to mobile computing devices for eventual use in mobile environments, such as Personal Digital Assistants (PDAs), tablet PCs, next generation mobile phones, and in-car driver assistance systems, poses critical challenges for natural human-computer interaction. Spoken dialogue is a key factor in ensuring natural and user-friendly interaction with such devices which are meant not only for computer specialists, but also for everyday users.

Speech supports hands-free and eyes-free operation, and becomes a key alternative interaction mode in mobile environments, e.g. in cars where driver distraction by manually operated devices may be a significant problem. On the other hand, the use of mobile devices in public places, may make the possibility of using alternative modalities possibly in combination with speech, such as graphics output and gesture input, preferable due to e.g. privacy issues. Researchers’ interest is progressively turning to the integration of speech with other modalities such as gesture input and graphics output, partly to accommodate more efficient interaction and partly to accommodate different user preferences.

This book:

combines overview chapters of key areas in spoken multimodal dialogue (systems and components, architectures, and evaluation) with chapters focussed on particular applications or problems in the field.

focusses on the influence of the environment when building and evaluating an application.

Audience:

Computer scientists, engineers, and others who work in the area of spoken multimodal dialogue systems in academia and in the industry.

Graduate students and Ph.D. students specialising in spoken multimodal dialogue systems in general, or focusing on issues in these systems in mobile environments in particular

Content:
Front Matter....Pages i-xxviii
Multimodal Dialogue Systems....Pages 3-11
Speech Recognition Technology in Multimodal/Ubiquitous Computing Environments....Pages 13-36
A Robust Multimodal Speech Recognition Method using Optical Flow Analysis....Pages 37-53
Feature Functions for Tree-Based Dialogue Course Management....Pages 55-76
A Reasoning Component for Information-Seeking and Planning Dialogues....Pages 77-91
A Model for Multimodal Dialogue System Output Applied to an Animated Talking Head....Pages 93-113
Overview of System Architecture....Pages 117-132
XISL: A Modality-Independent MMI Description Language....Pages 133-148
A Path to Multimodal Data Services for Telecommunications....Pages 149-167
Multimodal Spoken Dialogue with Wireless Devices....Pages 169-184
The Smartkom Mobile Car Prototype System for Flexible Human-Machine Communication....Pages 185-202
LARRI: A Language-Based Maintenance and Repair Assistant....Pages 203-218
Overview of Evaluation and Usability....Pages 221-246
Evaluating Dialogue Strategies in Multimodal Dialogue Systems....Pages 247-268
Enhancing the Usability of Multimodal Virtual Co-drivers....Pages 269-285
Design, Implementation and Evaluation of the SENECA Spoken Language Dialogue System....Pages 287-310
Segmenting Route Descriptions for Mobile Devices....Pages 311-328
Effects of Prolonged Use on the Usability of a Multimodal Form-Filling Interface....Pages 329-348
User Multitasking with Mobile Multimodal Systems....Pages 349-377
Speech Convergence with Animated Personas....Pages 379-397
Back Matter....Pages 399-406

---

The ongoing migration of computing and information access from stationary environments to mobile computing devices for eventual use in mobile environments, such as Personal Digital Assistants (PDAs), tablet PCs, next generation mobile phones, and in-car driver assistance systems, poses critical challenges for natural human-computer interaction. Spoken dialogue is a key factor in ensuring natural and user-friendly interaction with such devices which are meant not only for computer specialists, but also for everyday users.

Speech supports hands-free and eyes-free operation, and becomes a key alternative interaction mode in mobile environments, e.g. in cars where driver distraction by manually operated devices may be a significant problem. On the other hand, the use of mobile devices in public places, may make the possibility of using alternative modalities possibly in combination with speech, such as graphics output and gesture input, preferable due to e.g. privacy issues. Researchers’ interest is progressively turning to the integration of speech with other modalities such as gesture input and graphics output, partly to accommodate more efficient interaction and partly to accommodate different user preferences.

This book:

combines overview chapters of key areas in spoken multimodal dialogue (systems and components, architectures, and evaluation) with chapters focussed on particular applications or problems in the field.

focusses on the influence of the environment when building and evaluating an application.

Audience:

Computer scientists, engineers, and others who work in the area of spoken multimodal dialogue systems in academia and in the industry.

Graduate students and Ph.D. students specialising in spoken multimodal dialogue systems in general, or focusing on issues in these systems in mobile environments in particular

Content:
Front Matter....Pages i-xxviii
Multimodal Dialogue Systems....Pages 3-11
Speech Recognition Technology in Multimodal/Ubiquitous Computing Environments....Pages 13-36
A Robust Multimodal Speech Recognition Method using Optical Flow Analysis....Pages 37-53
Feature Functions for Tree-Based Dialogue Course Management....Pages 55-76
A Reasoning Component for Information-Seeking and Planning Dialogues....Pages 77-91
A Model for Multimodal Dialogue System Output Applied to an Animated Talking Head....Pages 93-113
Overview of System Architecture....Pages 117-132
XISL: A Modality-Independent MMI Description Language....Pages 133-148
A Path to Multimodal Data Services for Telecommunications....Pages 149-167
Multimodal Spoken Dialogue with Wireless Devices....Pages 169-184
The Smartkom Mobile Car Prototype System for Flexible Human-Machine Communication....Pages 185-202
LARRI: A Language-Based Maintenance and Repair Assistant....Pages 203-218
Overview of Evaluation and Usability....Pages 221-246
Evaluating Dialogue Strategies in Multimodal Dialogue Systems....Pages 247-268
Enhancing the Usability of Multimodal Virtual Co-drivers....Pages 269-285
Design, Implementation and Evaluation of the SENECA Spoken Language Dialogue System....Pages 287-310
Segmenting Route Descriptions for Mobile Devices....Pages 311-328
Effects of Prolonged Use on the Usability of a Multimodal Form-Filling Interface....Pages 329-348
User Multitasking with Mobile Multimodal Systems....Pages 349-377
Speech Convergence with Animated Personas....Pages 379-397
Back Matter....Pages 399-406
....