HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS		QUICK SEARCH:   [advanced] Author: Keyword(s): Year:  Vol:  Page:

This Article Full Text Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Services Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Watanabe, S. Search for Related Content PubMed PubMed Citation Articles by Watanabe, S.

Algebraic Analysis for Nonidentifiable Learning Machines

P&I Laboratory, Tokyo Institute of Technology, Yokohama, 226-8503 Japan

This article clarifies the relation between the learning curve and the algebraic geometrical structure of a nonidentifiable learning machine such as a multilayer neural network whose true parameter set is an analytic set with singular points. By using a concept in algebraic analysis, we rigorously prove that the Bayesian stochastic complexity or the free energy is asymptotically equal to ₁logn-(m₁-1)loglogn+ constant, where n is the number of training samples and ₁ and m₁ are the rational number and the natural number, which are determined as the birational invariant values of the singularities in the parameter space. Also we show an algorithm to calculate ₁ and m₁ based on the resolution of singularities in algebraic geometry. In regular statistical models, 2₁ is equal to the number of parameters and m₁=1, whereas in nonregular models, such as multilayer networks, 2₁ is not larger than the number of parameters and m₁1. Since the increase of the stochastic complexity is equal to the learning curve or the generalization error, the nonidentifiable learning machines are better models than the regular ones if Bayesian ensemble learning is applied.

This article has been cited by other articles:

				H. Wei, J. Zhang, F. Cousseau, T. Ozeki, and S.-i. Amari Dynamics of learning near singularities in layered networks. Neural Comput., March 1, 2008; 20(3): 813 - 843. [Abstract] [Full Text] [PDF]

				S. Nakajima and S. Watanabe Variational bayes solution of linear neural networks and its generalization performance. Neural Comput., April 1, 2007; 19(4): 1112 - 1153. [Abstract] [Full Text] [PDF]

				S.-i. Amari, H. Park, and T. Ozeki Singularities affect dynamics of learning in neuromanifolds. Neural Comput., May 1, 2006; 18(5): 1007 - 1065. [Abstract] [Full Text] [PDF]

				S. NAKAJIMA and S. WATANABE Generalization Performance of Subspace Bayes Approach in Linear Neural Networks IEICE Trans D: Information, March 1, 2006; E89-D(3): 1128 - 1138. [Abstract] [PDF]

				S.-i. Amari and H. Nakahara Difficulty of Singularity in Population Coding Neural Comput., April 1, 2005; 17(4): 839 - 858. [Abstract] [Full Text] [PDF]

				T. Hayasaka, M. Kitahara, and S. Usui On the Asymptotic Distribution of the Least-Squares Estimators in Unidentifiable Models Neural Comput., January 1, 2004; 16(1): 99 - 114. [Abstract] [Full Text] [PDF]

				K. Tsuda, S. Akaho, M. Kawanabe, and K.-R. Muller Asymptotic Properties of the Fisher Kernel Neural Comput., January 1, 2004; 16(1): 115 - 137. [Abstract] [Full Text] [PDF]

				S. Watanabe and S.-i. Amari Learning Coefficients of Layered Models When the True Distribution Mismatches the Singularities Neural Comput., May 1, 2003; 15(5): 1013 - 1033. [Abstract] [Full Text] [PDF]

				K. Hagiwara On the Problem in Model Selection of Neural Network Regression in Overrealizable Scenario Neural Comput., August 1, 2002; 14(8): 1979 - 2002. [Abstract] [Full Text] [PDF]