HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS		QUICK SEARCH:   [advanced] Author: Keyword(s): Year:  Vol:  Page:

This Article Full Text Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Services Similar articles in this journal Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via Google Scholar Google Scholar Articles by Schmidhuber, J. Articles by Gomez, F. Search for Related Content PubMed Articles by Schmidhuber, J. Articles by Gomez, F.

Training Recurrent Networks by Evolino

juergen{at}idsia.ch IDSIA, 6928 Manno (Lugano), Switzerland, and TU Munich, 85748 Garching, München, Germany

Daan Wierstra

daan{at}idsia.ch

Matteo Gagliolo

matteo{at}idsia.ch

Faustino Gomez

tino{at}idsia.ch IDSIA, 6928 Manno (Lugano), Switzerland

In recent years, gradient-based LSTM recurrent neural networks (RNNs) solved many previously RNN-unlearnable tasks. Sometimes, however, gradient information is of little use for training RNNs, due to numerous local minima. For such cases, we present a novel method: EVOlution of systems with LINear Outputs (Evolino). Evolino evolves weights to the nonlinear, hidden nodes of RNNs while computing optimal linear mappings from hidden state to output, using methods such as pseudo-inverse-based linear regression. If we instead use quadratic programming to maximize the margin, we obtain the first evolutionary recurrent support vector machines. We show that Evolino-based LSTM can solve tasks that Echo State nets (Jaeger, 2004a) cannot and achieves higher accuracy in certain continuous function generation tasks than conventional gradient descent RNNs, including gradient-based LSTM.