HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS		QUICK SEARCH:   [advanced] Author: Keyword(s): Year:  Vol:  Page:

This Article Full Text Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Services Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Amari, S.-i. Articles by Ozeki, T. Search for Related Content PubMed PubMed Citation Articles by Amari, S.-i. Articles by Ozeki, T.

Singularities Affect Dynamics of Learning in Neuromanifolds

amari{at}brain.riken.jp RIKEN Brain Science Institute, Saitama, 351-0198, Japan

Hyeyoung Park

hypark{at}knu.ac.kr Kyungpook National University, Korea

Tomoko Ozeki

tomoko{at}brain.riken.jp RIKEN Brain Science Institute, Saitama, 351-0198, Japan

The parameter spaces of hierarchical systems such as multilayer perceptrons include singularities due to the symmetry and degeneration of hidden units. A parameter space forms a geometrical manifold, called the neuromanifold in the case of neural networks. Such a model is identified with a statistical model, and a Riemannian metric is given by the Fisher information matrix. However, the matrix degenerates at singularities. Such a singular structure is ubiquitous not only in multilayer perceptrons but also in the gaussian mixture probability densities, ARMA time-series model, and many other cases. The standard statistical paradigm of the Cramér-Rao theorem does not hold, and the singularity gives rise to strange behaviors in parameter estimation, hypothesis testing, Bayesian inference, model selection, and in particular, the dynamics of learning from examples. Prevailing theories so far have not paid much attention to the problem caused by singularity, relying only on ordinary statistical theories developed for regular (nonsingular) models. Only recently have researchers remarked on the effects of singularity, and theories are now being developed.

This article gives an overview of the phenomena caused by the singularities of statistical manifolds related to multilayer perceptrons and gaussian mixtures. We demonstrate our recent results on these problems. Simple toy models are also used to show explicit solutions. We explain that the maximum likelihood estimator is no longer subject to the gaussian distribution even asymptotically, because the Fisher information matrix degenerates, that the model selection criteria such as AIC, BIC, and MDL fail to hold in these models, that a smooth Bayesian prior becomes singular in such models, and that the trajectories of dynamics of learning are strongly affected by the singularity, causing plateaus or slow manifolds in the parameter space. The natural gradient method is shown to perform well because it takes the singular geometrical structure into account. The generalization error and the training error are studied in some examples.

This article has been cited by other articles:

				H. Wei, J. Zhang, F. Cousseau, T. Ozeki, and S.-i. Amari Dynamics of learning near singularities in layered networks. Neural Comput., March 1, 2008; 20(3): 813 - 843. [Abstract] [Full Text] [PDF]

				S. Nakajima and S. Watanabe Variational bayes solution of linear neural networks and its generalization performance. Neural Comput., April 1, 2007; 19(4): 1112 - 1153. [Abstract] [Full Text] [PDF]

				G. Dutta, P. Jha, A. K. Laha, and N. Mohan Artificial Neural Network Models for Forecasting Stock Price Index in the Bombay Stock Exchange Journal of Emerging Market Finance, December 1, 2006; 5(3): 283 - 295. [Abstract] [PDF]