Neural Computation, Vol 8, 451-460, Copyright © 1996 by The MIT Press

ARTICLES

The interchangeability of learning rate and gain in backpropagation neural networks

G Thimm, P Moerland and E Fiesler
IDIAP, CH-1920 Martigny, Switzerland.

The backpropagation algorithm is widely used for training multilayer neural networks. In this publication the gain of its activation function(s) is investigated. In specific, it is proven that changing the gain of the activation function is equivalent to changing the learning rate and the weights. This simplifies the backpropagation learning rule by eliminating one of its parameters. The theorem can be extended to hold for some well-known variations on the backpropagation algorithm, such as using a momentum term, flat spot elimination, or adaptive gain. Furthermore, it is successfully applied to compensate for the nonstandard gain of optical sigmoids for optical neural networks.

This article has been cited by other articles:

				M. Pedzisz and D. P. Mandic A Homomorphic Neural Network for Modeling and Prediction Neural Comput., April 1, 2007; 20(4): 1042 - 1064. [Abstract] [Full Text] [PDF]

				Z. Chen and S. Haykin On Different Facets of Regularization Theory Neural Comput., December 1, 2002; 14(12): 2791 - 2846. [Abstract] [Full Text]

				D. P. Mandic and J. A. Chambers Relationships Between the A Priori and A Posteriori Errors in Nonlinear Adaptive Neural Filters Neural Comput., June 1, 2000; 12(6): 1285 - 1292. [Abstract] [Full Text]

				D. P. Mandic and J. A. Chambers Relating the Slope of the Activation Function and the Learning Rate Within a Recurrent Neural Network Neural Comput., July 1, 1999; 11(5): 1069 - 1077. [Abstract] [Full Text]