department of informatics

HephaisTK

Multimodal Interactive Systems: Architecture, Modeling and Fusion

PhD thesis, Bruno Dumas

 

Summary :

Multimodal interfaces have drawn a growing interest in the past decades. Nevertheless, progress has been slow due to the high number of state-of-the-art knowledge needed in order to create such an interface. This thesis work seeks to help multimodal interaction practitioners with the creation of multimodal interfaces, by focusing on the study of specific architectural and conceptual bottlenecks.

This thesis work is thus defined along three axes:

  • Study of different architectures for frameworks for multimodal interaction;
  • Description languages for multimodal interaction, and the definition of guidelines for the creation of such languages;
  • Fusion of Input Data for Multimodal Interaction, in particular, how to evaluate fusion algorithms, and creation of a novel fusion algorithm. 

These three axes have been explored through the conception of a toolkit for creation of multimodal interfaces, named HephaisTK, its scripting language, SMUIML, and a set of tools targeted at the study of interactive multimodal input fusion. 

HephaisTK toolkit is released as open source software on SourceForge Get HephaisTK multimodal toolkit at SourceForge.net. Fast, secure and Free Open Source software downloads.

 

Period : This thesis project has started on january 2006.

 

Fundings : The project is funded by The Hasler Stiftung and The IM2 project .

 

Participants :

 

For more details, see HephaisTK project on SourceForge Get HephaisTK multimodal toolkit at SourceForge.net. Fast, secure and Free Open Source software downloads

 


Publications

 

2010

  • Dumas, B., Lalanne, D., and Ingold, R. "Description languages for multimodal interaction: a set of guidelines and its illustration with SMUIML".  In Journal on Multimodal User Interfaces:  "Special Issue on The Challenges of Engineering Multimodal Interaction",  Springer Verlag,  vol. 3,  2009 , pp. 237-247. Online version

2009

  • Dumas, B., Ingold, R., Lalanne, D. "Benchmarking Fusion Engines of Multimodal Interactive Systems." In proc. of International Conference on Multimodal Interfaces and Workshop on Machine Learning for Multimodal Interaction  (ICMI-MLMI 2009), Cambridge (MA) (USA), November 02-06 2009, pp. 169-176. Online version
  • Dumas, B., Lalanne, D., Ingold R. "HephaisTK: A Toolkit for Rapid Prototyping of Multimodal Interfaces." In proc. of International Conference on Multimodal Interfaces and Workshop on Machine Learning for Multimodal Interaction  (ICMI-MLMI 2009), Cambridge (MA) (USA), November 02-06 2009, pp. 231-232. Online version
  • Dumas, B., Lalanne, D., Oviatt, S. "Multimodal Interfaces: A Survey of Principles, Models and Frameworks". In Denis Lalanne, Jürg Kohlas eds. (2009). Human Machine Interaction, LNCS 5440, Springer-Verlag, Berlin/Heidelberg, pp. 3-26. Online version
  • Mugellini, E., Lalanne, D., Dumas, B., Evequoz, F., Gerardi, S., Le Calvé, A., Boder, A, Ingold, R., Abou Khaled, O. "Memodules as tangible shortcuts to multimedia information". In Denis Lalanne, Jürg Kohlas eds (2009). Human Machine Interaction, LNCS 5440, Springer-Verlag, Berlin/Heidelberg, pp. 103-132. Online version

2008

  • Dumas, B., Lalanne, D., Ingold, R. "Démonstration: HephaisTK, une boîte à outils pour le prototypage d'interfaces multimodales." In proc. of 20e Conférence sur l'Interaction Homme-Machine (IHM 08), Metz (France), September 2-5 2008, pp. 215-216. Online version
  • Dumas, B., Lalanne, D., Ingold, R. "Prototyping Multimodal Interfaces with SMUIML Modeling Language". In proc. of CHI 2008 Workshop on UIDLs for Next Generation User Interfaces (CHI 2008 workshop), Florence (Italy), April 5-10 2008. Online version
  • Dumas, B., Lalanne, D., Guinard, D., Koenig, R., Ingold, R., "Strengths and weaknesses of software architectures for the rapid creation of tangible and multimodal interfaces." In proc. of 2nd international conference on Tangible and Embedded Interaction (TEI 2008), Bonn (Germany), February 19 - 21 2008, pp. 47-54. Online version

2007

  • Lalanne, D., Evéquoz, F., Rigamonti, M., Dumas, B., Ingold, R. "An ego-centric and tangible approach to meeting indexing and browsing." In proc. of 4th Joint Workshop on Multimodal Interaction and Related Machine Learning Algorithms (MLMI'07), Brno (Czech Republic), June 28 - 30 2007 , pp. 84-95. Online version