Multimodal and Mobile Interfaces

Mobile Multimodal Dialogue Systems (Prof. Wolfgang Wahlster)

Mobile multimodal dialogue systems allow the user and the system to adapt their choice of input and output modality according to various technical and cognitive resource limitations and the task at hand. They combine speech, gestures, haptics and facial expressions for input and/or output. Some of these systems like SMARTKOM and PEACH provide an anthropomorphic and affective interface through the personification of an interface agent. A dialogue system with symmetric multimodality must not only understand and represent the user's multimodal input, but also its own multimodal output. We present the anatomy of the SMARTKOM system, that provides full symmetric multimodality in a mixed-initiative dialogue system with an embodied conversational agent. SMARTKOM represents a new generation of multimodal dialogue systems, that deal not only with simple modality integration and synchronization, but cover the full spectrum of dialogue phenomena that are associated with symmetric multimodality (including crossmodal references, one-anaphora, and backchannelling). The application of this technology is especially motivated in non-desktop scenarios, such as smart rooms, kiosks, or mobile environments. We discuss the situated understanding of possibly imprecise, ambiguous or incomplete multimodal input and the generation of coordinated, cohesive, and coherent multimodal presentations based on advanced methods for multimodal fusion and fission. We show that plug-an-play architectures support multiple recognizers for a single modality, eg. the user's speech signal can be processed by three unimodal recognizers in parallel (speech recognition, emotional prosody, boundary prosody). The mutual disambiguation of modalities and the resolution of multimodal anaphora are based on a three-tiered discourse model, that consists of a domain, a discourse and a modality layer. We argue that intelligent multimodal interfaces are key to the consumers’ acceptance of new location-based web services for 3G UMTS smartphones and present various industrial spin-off products, e.g. the BPN system that can be used as a mobile travel companion for car drivers and pedestrians.

Wolfgang Wahlster is the Director and CEO of the German Research Center for Artificial Intelligence (DFKI GmbH) and a Professor of Computer Science at Saarland University, Saarbrücken. He received his diploma and doctoral degree (1981) in Computer Science from the University of Hamburg. He has published more than 160 technical papers and 7 books on intelligent user interfaces. His current research includes multimodal and perceptive user interfaces, user modeling, embodied conversational agents, smart navigation systems, semantic web services, and resource-adaptive cognitive technologies. He is a AAAI Fellow, a ECCAI Fellow, and a recipient of the Fritz Winter Award (1991), an IST Prize (1995), and the Beckurts prize (2000) for his research on cooperative user interfaces. In 2001, the President of the Federal Republic of Germany, Dr. Johannes Rau, presented the German Future Prize to Prof. Wahlster for his work on language technology and intelligent user interfaces. Prof. Wahlster was the first computer scientist to receive Germany's highest scientific prize that is awarded each year for outstanding innovations in technology, engineering, or the natural sciences. He was elected Full Member of the German Academy of Sciences and Literature, and Foreign Member of the Royal Swedish Society of Sciences, Stockholm. He is the Scientific Editor-in-Chief of ETAI, the Electronic Transactions on Artificial Intelligence, and a member of the Editorial Board or Advisory Board of the following journals: Annals of Mathematics and Artificial Intelligence, Applied Artificial Intelligence, Artificial Intelligence, Cognitive Science Quarterly, User Modeling and User-Adapted Interaction, Web Semantics.

Tangible Interfaces (Dr. Brygg Ullmer)

This tutorial will give an introduction to the field of tangible interfaces. The first day will present an overview, theory, and context of this research area. The second day will provide a practical introduction to some of the physical and electronic tools and technologies which support and enable research in this area, including RFID tagging, rapid physical prototyping, and embedded networked computing. Tangible interfaces are a genre of human-computer interaction where spatially reconfigurable physical objects are used as representations and controls for digital information. This approach has recently drawn considerable interest, with several dozen embodying systems published over the past few years. Tangible interfaces have demonstrated special potential for contexts and domains such as collocated collaboration, simulation, and education, often situated within physical and social contexts that are poorly supported by traditional graphical interfaces. This research area has been enabled and fueled by technological advances in embedded computing, sensing, and networking; RFID tagging; and rapid physical prototyping, among others. This tutorial should give participants the theoretical and practical exposure both to understand this body of work, as well as to begin designing and realizing their own novel systems.

Brygg Ullmer is a postdoctoral researcher in the visualization department of the Zuse Institute Berlin. He completed his Ph.D. at the MIT Media Lab in 2002, where he studied with Prof. Hiroshi Ishii in the Tangible Media group. He also holds a B.S. in computer engineering from the University of Illinois, Urbana-Champaign (1994), and an M.S. from the MIT Media Lab (1997). He has been a visiting lecturer in Hong Kong Polytechnic University's School of Design (2002), and has held internships at Sony Computer Science Labs, Tokyo (2000) and Interval Research Corporation (1993-95). His research interests include tangible and graphical user interfaces for interaction with online media, complex infrastructure, biological systems, and group interaction contexts, as well as rapid physical prototyping.

Building Practical Dialogue Systems (Prof. Johanna Moore & Dr. Oliver Lemon)

This course will cover a variety of techniques for building dialogue systems of different types: from simple form-filling applications to more complex task management and tutorial dialogue systems. The focus will be on choosing the right technology for a particular application to be developed, and its desired linguistic and interactive features. We will also cover a range of research issues in the field. In the first part of the course we will explain important issues in dialogue system design, such as as the representation and management of dialogue context, and the various linguistic and interactive properties which a system can exhibit (e.g. initiative, question accommodation). Then we compare the main approaches to building dialogue systems: finite state systems, form-based systems, Voice XML, agenda-based systems, task-based systems, and the "Information State Update" approach of TRINDI and DIPPER. We will present a variety of architectures, components, and resources that can be used to build working systems, many of which are freely available. We will also present a demonstration of a task-based multimodal dialogue system which uses Information State updates. The second part of the course will discuss ongoing research issues in dialogue system design, for example: Natural Language Generation in dialogue, task modelling and task switching, reconfigurable systems, context-sensitive speech recognition, and learning and adaptivity.

Johanna D. Moore is Professor of Artificial Intelligence and Director of the Human Communication Research Centre at the University of Edinburgh. Her research spans the areas of natural language processing, dialogue and discourse, planning, and knowledge management. Prof. Moore has directed and participated in numerous externally funded interdisciplinary research projects applying basic research results to applications in intelligent computer-based educational systems; customised information presentation for personalised healthcare and consumer services; and automatic generation of interactive multimedia presentations for data analysis in complex domains. Current projects focus on dialogue for tutoring systems, information extraction and summarisation of multiparty meetings, and learning and adaptivity for spoken dialogue systems. Prof. Moore is a Fellow of the Royal Society of Edinburgh and a recipient of a National Science Foundation National Young Investigator Award. She is the author of "Participating in Explanatory Dialogues" (MIT Press, 1995). She is currently president of the Association for Computational Linguistics, and is on the Governing Board of the Cognitive Science Society. She has just completed a three-year term on the AAAI Executive Council, and has served as Guest Editor of Computational Linguistics and Knowledge Based Systems, Associate Editor of the Journal of Artificial Intelligence Research, and as editorial board member for Computational Linguistics and JAIR. She has been Program Chair for several major conferences including Cognitive Science, the International Conference on Intelligent User Interfaces, and the World Conference on Artificial Intelligence and Education. Oliver Lemon is a Research Fellow at the School of Informatics, University of Edinburgh. He was formerly a Senior Project Engineer at the Center for the Study of Language and Information (CSLI) at Stanford University, California, working on multimodal dialogue systems (2000-2002). He holds a PhD from the Centre for Cognitive Science, Edinburgh University, for work in dialogue modelling, logic, and formal semantics (1996), and BSc in Mathematics and Philosophy from Manchester University (1992). He was Research Associate in the AI group at Manchester University (1995-98), working on spatial logics and graphical reasoning, and was a visiting scientist at NASA (Ames Research Center) in 2002. He has lectured in AI and computational linguistics at Trinity College, Dublin (1998-99), and gave an advanced course in Logic and Linguistics at the European Summer School in Logic, Language, and Information (ESSLLI 1998). His work on dialogue systems has been presented in numerous academic and commercial settings, including SRI International, Netscape/AOL, Telia, and NASA. He is Scientific Coordinator of the FP6 project "TALK" , focusing on machine learning techniques in multimodal dialogue systems.

Instrumental interaction (Prof. Michel Beaudouin-Lafon)

This course will present the principles of graphical interaction techniques and Instrumental Interaction, a model that can describe a wide variety of interaction techniques as well as help compare them and generate new ones. I will then introduce three design principles that have proved powerful to design novel interactive systems : reification, polymorphism and reuse. I will illustrate instrumental interaction and the design principles through a real world example, the CPN2000 system developed at University of Aarhus (Denmark) Finally, I will discuss power and simplicity, two conflicting goals facing any interaction designer and how to tackle them.

Michel Beaudouin-Lafon is Professor of Computer Science at Université Paris-Sud, France, and head of LRI (Laboratoire de Recherche en Informatique, http://www.lri.fr), the research laboratory for computer science at Université Paris-Sud and CNRS with 170 faculty, PhD students and staff. Michel's research interests cover several aspects of Human-Computer Interaction (HCI): interaction techniques, computer-supported cooperative work, engineering for HCI, models and methods for designing interactive systems. He carries out his research in the In Situ project, an interdisciplinary group of computer scientists, social scientists and designers. He is the author of more than 80 publications in international conferences and journals, and editor of a book on Computer-Supported Cooperative Work now available on-line from his web site. Michel was conference chair for the ACM User Interface Software and Technology Symposium (UIST '02) that took place in 2002 in Paris, and is co-chair for the forthcoming European Conference on Computer-Supported Cooperative Work (E-CSCW '05). He is a member-at-large of the ACM Council and was the founder and first president of AFIHM, the French HCI association.