Audio-Visual Group Recognition Using Diffusion Maps
Yosi Keller, Ronald R. Coifman, Stéphane Lafon et al. · IEEE Transactions on Signal Processing · 2009
Data fusion is a natural and common approach to recovering the state of physical systems. But the dissimilar appearance of different sensors remains a fundamental obstacle. We propose a unified embedding scheme for multisensory data, based on the spectral diffusion framework, which addresses this issue. Our scheme is purely data-driven and assumes noa prioristatistical or deterministic models of the data sources. To extract the underlying structure, we first embed separately each input channel; the resultant structures are then combined in diffusion coordinates. In particular, as different sensors sample similar phenomena with different sampling densities, we apply the density invariant Laplace-Beltrami embedding. This is a fundamental issue in multisensor acquisition and processing, overlooked in prior approaches. We extend previous work ongroup recognitionand suggest a novel approach to the selection of diffusion coordinates. To verify our approach, we demonstrate performance improvements in audio/visual speech recognition.
Cited by 50 works. doi:10.1109/tsp.2009.2030861
Authors
References
- The Elements of Statistical Learning: Data Mining, Inference, and Prediction
- The Elements of Statistical Learning: Data Mining, Inference, and Prediction
- Laplacian Eigenmaps for Dimensionality Reduction and Data Representation
- Alignment by Maximization of Mutual Information
- 10.1162/153244303322753616
- Alignment by maximization of mutual information
- Multiple kernel learning, conic duality, and the SMO algorithm
- Spectral grouping using the nystrom method
- Geometric diffusions as a tool for harmonic analysis and structure definition of data: Diffusion maps
- Diffusion maps