department of informatics

CHASM

Modelling Combined Handwriting and Speech Modalities

Summary :

  • The general context of the project is the study of advanced statistical modelling of multimodal data. More specifically, we are interested in modelling Combined Handwriting and Speech Modalities (the so-called CHASM data). In this approach, CHASM data are simultaneously recorded by asking the user to say what he is writing.
  • Our objective is then to build tools to model speech and handwriting data, simultaneously recorded by asking the user to say what he is writing. The modeling of CHASM data will be based on statistical approaches that are data-driven and CHASM modelling will be applied to identity verification and content recognition.
  • The main novelty of our approach is to handle a simultaneous recording of handwriting and speech. From the user perspective, the extra cognitive load associated to the simulatneous production of handwriting and speech is relatively small. On the other hand, from the computer perspective, having access to two streams of data will allow boosting the performance of the modelling.
  • We have finished the first and second phase of our project. The first phase was aiming at modelling both streams of data independently with state-of-the-art statistical models applied to identity verification. The second phase was aiming at modelling the two streams of data jointly. We are now entering the third phase, going deeper into advanced statistical modelling that can be applied to content recognition and identity verification.

Period : The project has started on April 2005.

Fundings : The project was partly funded by IM2-ACP (phase I of IM2) and partly by BioSecure (working group single modality speech and signature).

Participants :

  • andreas.humm AT unifr.ch
  • jean.hennebert AT unifr.ch (thesis supervisor)
  • rolf.ingold AT unifr.ch

Partners : Members of the EU BioSecure project

 


Publications related to this project

  • A. Humm, R. Ingold and J. Hennebert, "Spoken Handwriting for User Authentication using Joint Modelling Systems", In proc. of 6th International Symposium on Image and Signal Processing and Analysis (ISPA'09), Salzburg (Austria), September 16 - 18, 2009, pp. 505 - 510.
  • A. Humm, J. Hennebert and R. Ingold, "Combined Handwriting And Speech Modalities For User Authentication", In IEEE Transactions on Systems, Man, and Cybernetics, Part A: Systems and Humans, Vol. 39, No. 1, January, 2009, pp. 25 - 35.
  • A. Humm, J. Hennebert and R. Ingold, "Spoken Signature For User Authentication", In SPIE Journal of Electronic Imaging, Special Section on Biometrics: ASUI, January-March 2008 Vol. 17, No. 1, April 2 2008, pp. 011013-1 - 011013-11.
  • A. Humm, J. Hennebert and R. Ingold, "Hidden Markov Models for Spoken Signature Verification", In proc. of IEEE Conference on Biometrics: Theory, Applications and Systems (BTAS'07), Washington DC (USA), September 27 - 29, 2007.
  • A. Humm, R. Ingold and J. Hennebert, "Spoken Handwriting Verification using Statistical Models", In proc. of 9th International Conference on Document Analysis and Recognition (ICDAR'07), Curitiba (Brazil), September 23 - 26, 2007, pp. 999 - 1003.
  • A. Humm, J. Hennebert and R. Ingold, "Modelling Combined Handwriting And Speech Modalities", In proc. of 2nd International Conference on Biometrics (ICB'07), Seoul, Korea, August 27 - 29, 2007, pp. 1025 - 1034.
  • J. Hennebert, R. Loeffel, A. Humm and R. Ingold, "A New Forgery Scenario Based On Regaining Dynamics Of Signature", In proc. of 2nd International Conference on Biometrics (ICB'07), Seoul (Korea), August 27 - 29, 2007, pp. 366 - 375.
  • J. Hennebert, A. Humm and R. Ingold, "Modelling Spoken Signatures with Gaussian Mixture Model Adaptation", In proc. of 32nd International Conference on Acoustics, Speech, and Signal Processing (ICASSP'07), Honolulu, Hawaii (USA), April 15 - 20, 2007, pp. II-229 - II-232.
  • J. Hennebert, A. Humm, R. Ingold, "Vérification d'Identité par Ecriture et Parole Combinées", In proc. of Colloque International Francophone sur l'Ecrit et le Document (CIFED'06), Fribourg (Switzerland), September 18 - 22, 2006, pp. 235-240.
  • A. Wahl, J. Hennebert, A. Humm, R. Ingold, "Generation and Evaluation of Brute-Force Signature Forgeries", In proc. of Workshop on Multimedia Content Representation, Classification and Security (MRCS'06), Istanbul (Turkey), September 11 - 13, 2006, pp. 2-9.
  • A. Humm, J. Hennebert, R. Ingold, "Scenario and Survey of Combined Handwriting and Speech Modalities for User Authentication", In proc. of 6th International Conference on Recent Advances in Soft Computing (RASC'06), Canterbury (UK), July 10 - 12, 2006, pp. 496-501.
  • A. Humm, J. Hennebert, R. Ingold, "Gaussian Mixture Models for CHASM Signature Verification", In proc. of 3rd Joint Workshop on Multimodal Interaction and Related Machine Learning Algorithms (MLMI'06), Washington DC (USA), May 1 - 3, 2006, pp. 102-113.
  • B. Dumas, C. Pugin, J. Hennebert, D. Petrovska-Delacrétaz, A. Humm, F. Evéquoz, R. Ingold, D. Von Rotz, "MyIdea - Multimodal Biometrics Database, Description of Acquisition Protocols", In proc. of Third COST 275 Workshop (COST 275), Hatfield (UK), October 27 - 28, 2005, pp. 59-62.

  • A. Humm, J. Hennebert and R. Ingold, "Database and Evaluation Protocols for User Authentication using Combined Handwriting and Speech Modalities", Internal Research Report, 2007.
  • A. Humm, J. Hennebert, R. Ingold, "Combined Handwriting and Speech Modalities for User Authentication", Internal Research Report, 2006.
  • A. Wahl, J. Hennebert, A. Humm, R. Ingold, "A Novel Method to Generate Brute-Force Signature Forgeries", Internal Research Report, 2006.
  • B. Dumas, F. Evéquoz, J. Hennebert, A. Humm, R. Ingold, D. Petrovska-Delacrétaz, C. Pugin and D. Von Rotz, "MyIDea, Sensors Specifications and Acquisition Protocol", Internal Research Report, 2005.