Toward Adaptive Information Fusion in Multimodal Systems
X. Huang, and S. Oviatt. Machine Learning for Multimodal Interaction: Second International Workshop, MLMI 2005, Edinburgh, UK, July 11--13, 2005, Revised Selected Papers, volume 3869 of Lecture Notes in Computer Science, Springer, Berlin, (2006)
DOI: 10.1007/11677482_2
Abstract
In recent years, a new generation of multimodal systems has emerged as a major direction within the HCI community. Multimodal interfaces and architectures are time-critical and data-intensive to develop, which poses new research challenges. The goal of the present work is to model and adapt to users' multimodal integration patterns, so that faster and more robust systems can be developed with on-line adaptation to individual's multimodal temporal thresholds. In this paper, we summarize past user-modeling results on speech and pen multimodal integration patterns, which indicate that there are two dominant types of multimodal integration pattern among users that can be detected very early and remain highly consistent. The empirical results also indicate that, when interacting with a multimodal system, users intermix unimodal with multimodal commands. Based on these results, we present new machine-learning results comparing three models of on-line system adaptation to users' integration patterns, which were based on Bayesian Belief Networks. This work utilized data from ten adults who provided approximately 1,000 commands while interacting with a map-based multimodal system. Initial experimental results with our learning models indicated that 85 percent of users' natural mixed input could be correctly classified as either unimodal or multimodal, and 82 percent of users' mulitmodal input could be correctly classified as either sequentially or simultaneously integrated. The long-term goal of this research is to develop new strategies for combining empirical user modeling with machine learning techniques to bootstrap accelerated, generalized, and improved reliability of information fusion in new types of multimodal system.
%0 Book Section
%1 HuangOviatt05MLMI
%A Huang, Xiao
%A Oviatt, Sharon
%B Machine Learning for Multimodal Interaction: Second International Workshop, MLMI 2005, Edinburgh, UK, July 11--13, 2005, Revised Selected Papers
%C Berlin
%D 2006
%E Renals, Steve
%E Bengio, Samy
%I Springer
%K 01624 springer paper ai multimodal interaction user interface information processing adaptive analysis learn zzz.mmi
%P 15--27
%R 10.1007/11677482_2
%T Toward Adaptive Information Fusion in Multimodal Systems
%V 3869
%X In recent years, a new generation of multimodal systems has emerged as a major direction within the HCI community. Multimodal interfaces and architectures are time-critical and data-intensive to develop, which poses new research challenges. The goal of the present work is to model and adapt to users' multimodal integration patterns, so that faster and more robust systems can be developed with on-line adaptation to individual's multimodal temporal thresholds. In this paper, we summarize past user-modeling results on speech and pen multimodal integration patterns, which indicate that there are two dominant types of multimodal integration pattern among users that can be detected very early and remain highly consistent. The empirical results also indicate that, when interacting with a multimodal system, users intermix unimodal with multimodal commands. Based on these results, we present new machine-learning results comparing three models of on-line system adaptation to users' integration patterns, which were based on Bayesian Belief Networks. This work utilized data from ten adults who provided approximately 1,000 commands while interacting with a map-based multimodal system. Initial experimental results with our learning models indicated that 85 percent of users' natural mixed input could be correctly classified as either unimodal or multimodal, and 82 percent of users' mulitmodal input could be correctly classified as either sequentially or simultaneously integrated. The long-term goal of this research is to develop new strategies for combining empirical user modeling with machine learning techniques to bootstrap accelerated, generalized, and improved reliability of information fusion in new types of multimodal system.
%@ 978-3-540-32549-9
@incollection{HuangOviatt05MLMI,
abstract = {In recent years, a new generation of multimodal systems has emerged as a major direction within the HCI community. Multimodal interfaces and architectures are time-critical and data-intensive to develop, which poses new research challenges. The goal of the present work is to model and adapt to users' multimodal integration patterns, so that faster and more robust systems can be developed with on-line adaptation to individual's multimodal temporal thresholds. In this paper, we summarize past user-modeling results on speech and pen multimodal integration patterns, which indicate that there are two dominant types of multimodal integration pattern among users that can be detected very early and remain highly consistent. The empirical results also indicate that, when interacting with a multimodal system, users intermix unimodal with multimodal commands. Based on these results, we present new machine-learning results comparing three models of on-line system adaptation to users' integration patterns, which were based on Bayesian Belief Networks. This work utilized data from ten adults who provided approximately 1,000 commands while interacting with a map-based multimodal system. Initial experimental results with our learning models indicated that 85 percent of users' natural mixed input could be correctly classified as either unimodal or multimodal, and 82 percent of users' mulitmodal input could be correctly classified as either sequentially or simultaneously integrated. The long-term goal of this research is to develop new strategies for combining empirical user modeling with machine learning techniques to bootstrap accelerated, generalized, and improved reliability of information fusion in new types of multimodal system.},
added-at = {2017-05-16T09:00:57.000+0200},
address = {Berlin},
author = {Huang, Xiao and Oviatt, Sharon},
biburl = {https://www.bibsonomy.org/bibtex/2d97d0b174a03c59f97d77fc1396016a7/flint63},
booktitle = {Machine Learning for Multimodal Interaction: Second International Workshop, MLMI 2005, Edinburgh, UK, July 11--13, 2005, Revised Selected Papers},
crossref = {MLMI2005},
doi = {10.1007/11677482_2},
editor = {Renals, Steve and Bengio, Samy},
file = {SpringerLink:2005/HuangOviatt05MLMI.pdf:PDF},
groups = {public},
interhash = {1170e7ddd4fdb20baff52e7366230a8c},
intrahash = {d97d0b174a03c59f97d77fc1396016a7},
isbn = {978-3-540-32549-9},
issn = {0302-9743},
keywords = {01624 springer paper ai multimodal interaction user interface information processing adaptive analysis learn zzz.mmi},
pages = {15--27},
publisher = {Springer},
series = {Lecture Notes in Computer Science},
timestamp = {2017-07-13T17:36:23.000+0200},
title = {Toward Adaptive Information Fusion in Multimodal Systems},
username = {flint63},
volume = 3869,
year = 2006
}