| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
Neural Computation, Vol 10, 251-276, Copyright © 1998 by The MIT Press
ARTICLES |
Shun-ichi Amari
When a parameter space has a certain underlying structure, the ordinary gradient of a function does not represent its steepest direction, but the natural gradient does. Information geometry is used for calculating the natural gradients in the parameter space of perceptrons, the space of matrices (for blind source separation), and the space of linear dynamical systems (for blind source deconvolution). The dynamical behavior of natural gradient online learning is analyzed and is proved to be Fisher efficient, implying that it has asymptotically the same performance as the optimal batch estimation of parameters. This suggests that the plateau phenomenon, which appears in the backpropagation learning algorithm of multilayer perceptrons, might disappear or might not be so serious when the natural gradient is used. An adaptive method of updating the learning rate is proposed and analyzed.
This article has been cited by other articles:
|
|
 |
|
  Z. He, S. Xie, L. Zhang, and A. Cichocki A note on lewicki-sejnowski gradient for learning overcomplete representations. Neural Comput., March 1, 2008; 20(3): 636 - 643. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
H. Wei, J. Zhang, F. Cousseau, T. Ozeki, and S.-i. Amari Dynamics of learning near singularities in layered networks. Neural Comput., March 1, 2008; 20(3): 813 - 843. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 X. ZHANG, N. ZHANG, J. LU, and T. YAHAGI Independent Component Analysis for Image Recovery Using SOM-Based Noise Detection IEICE Trans A: Fundamentals, June 1, 2007; E90-A(6): 1125 - 1132. [Abstract] [PDF]
|
|
|
|
|
|
J. EVEN and K. SUGIMOTO Blind Identification for Systems Non-Invertible at Infinity IEICE Trans A: Fundamentals, June 1, 2007; E90-A(6): 1133 - 1143. [Abstract] [PDF]
|
|
|
|
|
|
S. Fiori A Study on Neural Learning on Manifold Foliations: The Case of the Lie Group SU(3) Neural Comput., April 1, 2007; 20(4): 1091 - 1117. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
K. Miura, M. Okada, and S.-i. Amari Estimating Spiking Irregularities Under Changing Environments. Neural Comput., October 1, 2006; 18(10): 2359 - 2386. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
K. IKEDA Geometric Properties of Quasi-Additive Learning Algorithms IEICE Trans A: Fundamentals, October 1, 2006; E89-A(10): 2812 - 2817. [Abstract] [PDF]
|
|
|
|
|
|
J. Basak Online adaptive decision trees: pattern classification and function approximation. Neural Comput., September 1, 2006; 18(9): 2062 - 2101. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
M. TUFAIL, M. ABE, and M. KAWAMATA An Extension to the Natural Gradient Algorithm for Robust Independent Component Analysis in the Presence of Outliers IEICE Trans A: Fundamentals, September 1, 2006; E89-A(9): 2429 - 2432. [Abstract] [PDF]
|
|
|
|
|
|
S.-i. Amari, H. Park, and T. Ozeki Singularities affect dynamics of learning in neuromanifolds. Neural Comput., May 1, 2006; 18(5): 1007 - 1065. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
X.-L. Zhu, X.-D. Zhang, and J.-M. Ye A generalized contrast function and stability analysis for overdetermined blind separation of instantaneous mixtures. Neural Comput., March 1, 2006; 18(3): 709 - 728. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
S. MAKINO, H. SAWADA, R. MUKAI, and S. ARAKI Blind Source Separation of Convolutive Mixtures of Speech in Frequency Domain IEICE Trans A: Fundamentals, July 1, 2005; E88-A(7): 1640 - 1655. [Abstract] [PDF]
|
|
|
|
|
|
A. ANDO, M. IWAKI, K. ONO, and K. KUROZUMI Separation of Sound Sources Propagated in the Same Direction IEICE Trans A: Fundamentals, July 1, 2005; E88-A(7): 1665 - 1672. [Abstract] [PDF]
|
|
|
|
|
|
S.-i. Amari and H. Nakahara Difficulty of Singularity in Population Coding Neural Comput., April 1, 2005; 17(4): 839 - 858. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
K. Zhang and L.-W. Chan Extended Gaussianization Method for Blind Separation of Post-Nonlinear Mixtures Neural Comput., February 1, 2005; 17(2): 425 - 452. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
S. Ikeda, T. Tanaka, and S.-i. Amari Stochastic Reasoning, Free Energy, and Information Geometry Neural Comput., September 1, 2004; 16(9): 1779 - 1810. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
J. Basak Online Adaptive Decision Trees Neural Comput., September 1, 2004; 16(9): 1959 - 1981. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
J.-M. Ye, X.-L. Zhu, and X.-D. Zhang Adaptive Blind Separation with an Unknown Number of Sources Neural Comput., August 1, 2004; 16(8): 1641 - 1660. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
H. Park, N. Murata, and S.-i. Amari Improving Generalization Performance of Natural Gradient Learning Using Optimized Regularization by NIC Neural Comput., February 1, 2004; 16(2): 355 - 382. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 S. Zeki, R.J. Perry, and A. Bartels The Processing of Kinetic Contours in the Brain Cereb Cortex, February 1, 2003; 13(2): 189 - 202. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
N. Ay Locality of Global Stochastic Interaction in Directed Acyclic Networks Neural Comput., December 1, 2002; 14(12): 2959 - 2980. [Abstract] [Full Text]
|
|
|
|
|
|
K. Hagiwara On the Problem in Model Selection of Neural Network Regression in Overrealizable Scenario Neural Comput., August 1, 2002; 14(8): 1979 - 2002. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
N. N. Schraudolph Fast Curvature Matrix-Vector Products for Second-Order Gradient Descent Neural Comput., July 1, 2002; 14(7): 1723 - 1738. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
M. Zlochin and Y. Baram Manifold Stochastic Dynamics for Bayesian Learning Neural Comput., November 1, 2001; 13(11): 2549 - 2572. [Abstract] [Full Text]
|
|
|
|
|
|
S. Fiori A Theory for Learning by Weight Flow on Stiefel-Grassman Manifold Neural Comput., July 1, 2001; 13(7): 1625 - 1647. [Abstract] [Full Text]
|
|
|
|
|
|
M.-a. Sato Online Model Selection Based on the Variational Bayes Neural Comput., July 1, 2001; 13(7): 1649 - 1681. [Abstract] [Full Text]
|
|
|
|
|
|
S.-i. Amari Estimating Functions of Independent Component Analysis for Temporally Correlated Signals Neural Comput., September 1, 2000; 12(9): 2083 - 2107. [Abstract] [Full Text]
|
|
|
|
|
|
S.-i. Amari, H. Park, and K. Fukumizu Adaptive Method of Realizing Natural Gradient Learning for Multilayer Perceptrons Neural Comput., June 1, 2000; 12(6): 1399 - 1409. [Abstract] [Full Text]
|
|
|
|
|
|
A. Navia-Vázquez and A. R. Figueiras-Vidal Efficient Block Training of Multilayer Perceptrons Neural Comput., June 1, 2000; 12(6): 1429 - 1447. [Abstract] [Full Text]
|
|
|
|
|
|
T. Heskes On "Natural" Learning and Pruning in Multilayered Perceptrons Neural Comput., April 1, 2000; 12(4): 881 - 901. [Abstract] [Full Text]
|
|
|
|
|
|
S.-i. Amari Natural Gradient Learning for Over- and Under-Complete Bases in ICA Neural Comput., November 15, 1999; 11(8): 1875 - 1883. [Abstract] [Full Text]
|
|
|
|
|
|
P. van de Larr and T. Heskes Pruning Using Parameter and Neuronal Metrics Neural Comput., May 15, 1999; 11(4): 977 - 993. [Abstract] [Full Text]
|
|
|
|
|
|
J. Basak and S.-i. Amari Blind Separation of a Mixture of Uniformly Distributed Source Signals: A Novel Approach Neural Comput., May 15, 1999; 11(4): 1011 - 1034. [Abstract] [Full Text]
|
|
|
|
 |
|
 S. Makeig, M. Westerfield, T.-P. Jung, J. Covington, J. Townsend, T. J. Sejnowski, and E. Courchesne Functionally Independent Components of the Late Positive Event-Related Potential during Visual Spatial Attention J. Neurosci., April 1, 1999; 19(7): 2665 - 2680. [Abstract] [Full Text] [PDF]
|
|
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
| J COGNITIVE NEUROSCIENCE | NEURAL COMPUTATION | MIT PRESS JOURNALS |
