Introduction to Recurrent Neural Networks and Learning Algorithms


Leong-Kwan Li

Department of Applied Mathematics
The Hong Kong Polytechnic University
Hung Hom, Kowloon, Hong Kong

Abstract :  In this talk, we introduce the mathematical model for leaky integrator recurrent neural networks. For a given trajectory, we may use a recurrent neural network dynamic to approximate the trajectory. Learning is a process for which we adjust the neural parameters so that the network output comes close to the trajectory. This process of modifying parameters is called "learning". Williams and Zipser (1989) derived the learning algorithms for trajectories of a discrete-time recurrent neural network but the question of the capability of the recurrent network is still unknown. Then, we consider a learning algorithm for discrete-time recurrent neural networks as a nonlinear numerical optimization problem. We shall also discuss the relations between least square errors and the network sizes. We also compare our results with linear time series models.