HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS		QUICK SEARCH:   [advanced] Author: Keyword(s): Year:  Vol:  Page:

This Article Full Text Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Services Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Sato, M.-a. Search for Related Content PubMed PubMed Citation Articles by Sato, M.-a.

Online Model Selection Based on the Variational Bayes

Information Sciences Division, ATR International, and CREST, Japan Science and Techonology Corporation, Seika-cho, Soraku-gun, Kyoto 619-0288, Japan

The Bayesian framework provides a principled way of model selection. This framework estimates a probability distribution over an ensemble of models, and the prediction is done by averaging over the ensemble of models. Accordingly, the uncertainty of the models is taken into account, and complex models with more degrees of freedom are penalized. However, integration over model parameters is often intractable, and some approximation scheme is needed.

Recently, a powerful approximation scheme, called the variational bayes (VB) method, has been proposed. This approach defines the free energy for a trial probability distribution, which approximates a joint posterior probability distribution over model parameters and hidden variables. The exact maximization of the free energy gives the true posterior distribution. The VB method uses factorized trial distributions. The integration over model parameters can be done analytically, and an iterative expectation-maximization-like algorithm, whose convergence is guaranteed, is derived.

In this article, we derive an online version of the VB algorithm and prove its convergence by showing that it is a stochastic approximation for finding the maximum of the free energy. By combining sequential model selection procedures, the online VB method provides a fully online learning method with a model selection mechanism. In preliminary experiments using synthetic data, the online VB method was able to adapt the model structure to dynamic environments.

This article has been cited by other articles:

				S. Nakajima and S. Watanabe Variational bayes solution of linear neural networks and its generalization performance. Neural Comput., April 1, 2007; 19(4): 1112 - 1153. [Abstract] [Full Text] [PDF]

				J.-M. Ye, X.-L. Zhu, and X.-D. Zhang Adaptive Blind Separation with an Unknown Number of Sources Neural Comput., August 1, 2004; 16(8): 1641 - 1660. [Abstract] [Full Text] [PDF]