Neural Computation, Vol 10, 1137-1156, Copyright © 1998 by The MIT Press

LETTERS

Efficient Learning in Boltzmann Machines Using Linear Response Theory

H. J. Kappen and F. B. Rodríguez

The learning process in Boltzmann machines is computationally very expensive. The computational complexity of the exact algorithm is exponential in the number of neurons. We present a new approximate learning algorithm for Boltzmann machines, based on mean-field theory and the linear response theorem. The computational complexity of the algorithm is cubic in the number of neurons. In the absence of hidden units, we show how the weights can be directly computed from the fixed-point equation of the learning rules. Thus, in this case we do not need to use a gradient descent procedure for the learning process. We showthat the solutions of this methodare close to the optimal solutions and give a significant improvement when correlations play a significant role. Finally, we apply the method to a pattern completion task and show good performance for networks up to 100 neurons.

This article has been cited by other articles:

				A. Hyvarinen Consistency of Pseudolikelihood Estimation of Fully Visible Boltzmann Machines. Neural Comput., October 1, 2006; 18(10): 2283 - 2292. [Abstract] [Full Text] [PDF]

				M. Welling and Y. W. Teh Linear Response Algorithms for Approximate Inference in Graphical Models Neural Comput., January 1, 2004; 16(1): 197 - 221. [Abstract] [Full Text] [PDF]

				P. A.d.F.R. Hojen-Sorensen, O. Winther, and L. K. Hansen Mean-Field Approaches to Independent Component Analysis Neural Comput., April 1, 2002; 14(4): 889 - 918. [Abstract] [Full Text] [PDF]