HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS		QUICK SEARCH:   [advanced] Author: Keyword(s): Year:  Vol:  Page:

This Article Full Text Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Services Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Hagiwara, K. Search for Related Content PubMed PubMed Citation Articles by Hagiwara, K.

On the Problem in Model Selection of Neural Network Regression in Overrealizable Scenario

hagi{at}phen.mie-u.ac.jp, Faculty of Physics Engineering, Mie University, Tsu, 514-8507, Japan

In considering a statistical model selection of neural networks and radial basis functions under an overrealizable case, the problem of unidentifiability emerges. Because the model selection criterion is an unbiased estimator of the generalization error based on the training error, this article analyzes the expected training error and the expected generalization error of neural networks and radial basis functions in overrealizable cases and clarifies the difference from regular models, for which identifiability holds. As a special case of an overrealizable scenario, we assumed a gaussian noise sequence as training data. In the least-squares estimation under this assumption, we first formulated the problem, in which the calculation of the expected errors of unidentifiable networks is reduced to the calculation of the expectation of the supremum of the ² process. Under this formulation, we gave an upper bound of the expected training error and a lower bound of the expected generalization error, where the generalization is measured at a set of training inputs. Furthermore, we gave stochastic bounds on the training error and the generalization error. The obtained upper bound of the expected training error is smaller than in regular models, and the lower bound of the expected generalization error is larger than in regular models. The result tells us that the degree of overfitting in neural networks and radial basis functions is higher than in regular models. Correspondingly, it also tells us that the generalization capability is worse than in the case of regular models. The article may be enough to show a difference between neural networks and regular models in the context of the least-squares estimation in a simple situation. This is a first step in constructing a model selection criterion in an overrealizable case. Further important problems in this direction are also included in this article.

This article has been cited by other articles:

				H. Wei, J. Zhang, F. Cousseau, T. Ozeki, and S.-i. Amari Dynamics of learning near singularities in layered networks. Neural Comput., March 1, 2008; 20(3): 813 - 843. [Abstract] [Full Text] [PDF]

				S. Nakajima and S. Watanabe Variational bayes solution of linear neural networks and its generalization performance. Neural Comput., April 1, 2007; 19(4): 1112 - 1153. [Abstract] [Full Text] [PDF]

				K. HAGIWARA and H. ISHITANI On the Expected Prediction Error of Orthogonal Regression with Variable Components IEICE Trans A: Fundamentals, December 1, 2006; E89-A(12): 3699 - 3709. [Abstract] [PDF]

				S.-i. Amari, H. Park, and T. Ozeki Singularities affect dynamics of learning in neuromanifolds. Neural Comput., May 1, 2006; 18(5): 1007 - 1065. [Abstract] [Full Text] [PDF]

				S. NAKAJIMA and S. WATANABE Generalization Performance of Subspace Bayes Approach in Linear Neural Networks IEICE Trans D: Information, March 1, 2006; E89-D(3): 1128 - 1138. [Abstract] [PDF]

				H. Park, N. Murata, and S.-i. Amari Improving Generalization Performance of Natural Gradient Learning Using Optimized Regularization by NIC Neural Comput., February 1, 2004; 16(2): 355 - 382. [Abstract] [Full Text] [PDF]

				T. Hayasaka, M. Kitahara, and S. Usui On the Asymptotic Distribution of the Least-Squares Estimators in Unidentifiable Models Neural Comput., January 1, 2004; 16(1): 99 - 114. [Abstract] [Full Text] [PDF]

				S. Watanabe and S.-i. Amari Learning Coefficients of Layered Models When the True Distribution Mismatches the Singularities Neural Comput., May 1, 2003; 15(5): 1013 - 1033. [Abstract] [Full Text] [PDF]