Neural Computation, Vol 11, Issue 2 443-482, Copyright © 1999 by The MIT Press

LETTERS

Mixtures of probabilistic principal component analyzers

ME Tipping and CM Bishop
Microsoft Research Limited, Saint George House, 1 Guildhall Street, CB2 3NH, United Kingdom. mtipping@microsoft.com.

Principal component analysis (PCA) is one of the most popular techniques for processing, compressing, and visualizing data, although its effectiveness is limited by its global linearity. While nonlinear variants of PCA have been proposed, an alternative paradigm is to capture data complexity by a combination of local linear PCA projections. However, conventional PCA does not correspond to a probability density, and so there is no unique way to combine PCA models. Therefore, previous attempts to formulate mixture models for PCA have been ad hoc to some extent. In this article, PCA is formulated within a maximum likelihood framework, based on a specific form of gaussian latent variable model. This leads to a well-defined mixture model for probabilistic principal component analyzers, whose parameters can be determined using an expectation-maximization algorithm. We discuss the advantages of this model in the context of clustering, density modeling, and local dimensionality reduction, and we demonstrate its application to image compression and handwritten digit recognition.

This article has been cited by other articles:

				Y. Bengio, M. Monperrus, and H. Larochelle Nonlocal Estimation of Manifold Structure. Neural Comput., October 1, 2006; 18(10): 2509 - 2528. [Abstract] [Full Text] [PDF]

				R. Yoshida, T. Higuchi, S. Imoto, and S. Miyano ArrayCluster: an analytic tool for clustering, data visualization and module finder on gene expression profiles Bioinformatics, June 15, 2006; 22(12): 1538 - 1539. [Abstract] [Full Text] [PDF]

				E. Lopez-Rubio, J. M. Ortiz-de-Lazcano-Lobato, J. Munoz-Perez, and J. Antonio Gomez-Ruiz Principal Components Analysis Competitive Learning Neural Comput., November 1, 2004; 16(11): 2459 - 2481. [Abstract] [Full Text] [PDF]

				G J McLachlan and S U Chang Mixture modelling for cluster analysis Statistical Methods in Medical Research, October 1, 2004; 13(5): 347 - 361. [Abstract] [PDF]

				J.-H. Ahn and J.-H. Oh A Constrained EM Algorithm for Principal Component Analysis Neural Comput., January 1, 2003; 15(1): 57 - 65. [Abstract] [Full Text] [PDF]

				R. A. Choudrey and S. J. Roberts Variational Mixture of Bayesian Independent Component Analyzers Neural Comput., January 1, 2003; 15(1): 213 - 252. [Abstract] [Full Text] [PDF]

				M.-a. Sato Online Model Selection Based on the Variational Bayes Neural Comput., July 1, 2001; 13(7): 1649 - 1681. [Abstract] [Full Text]

				A. Utsugi and T. Kumagai Bayesian Analysis of Mixtures of Factor Analyzers Neural Comput., May 1, 2001; 13(5): 993 - 1002. [Abstract] [Full Text] [PDF]

				P. Meinicke and H. Ritter Resolution-Based Complexity Control for Gaussian Mixture Models Neural Comput., February 1, 2001; 13(2): 453 - 475. [Abstract] [Full Text]

				W. S. DeSarbo, A. M. Degeratu, M. Wedel, and M. K. Saxton The Spatial Representation of Market Information Marketing Science, January 1, 2001; 20(4): 426 - 441. [Abstract] [PDF]

				D. Husmeier The Bayesian Evidence Scheme for Regularizing Probability-Density Estimating Neural Networks Neural Comput., November 1, 2000; 12(11): 2685 - 2717. [Abstract] [Full Text]