INVERSE APPROXIMATION THEORY OF RECURRENT MODELS FOR LEARNING SEQUENCES

Learning long-term relationships is a challenging task in sequence modelling. Despite numerous empirical results demonstrating the difficulty of recurrent models in learning long-term relationships, this dissertation presents a series of theoretical studies on the learning of long-term memories using recurrent models. First, based on the concept of a generalized memory function over nonlinear functional sequences, we prove that adding nonlinear activation does not change the asymptotic exponential memory decay pattern. Then we prove the universal approximation property for state-space models. A similar inverse approximation result is established for state-space models, indicating that despite their high efficiency, changing the method of incorporating nonlinear activation fails to alleviate the memory decay constraint observed in the first part. Based on the proofs, we identify that suitable reparameterizations are the key to stably approximation. A class of stable reparameterizations enables state-space models to achieve stable approximation for any target with decaying memory.

Author dc:creator

• WANG SHIDA

• memory function
• stable approximation
• state-space model
• recurrent model
• sequence modelling
• Inverse approximation theory

Licence

• https://scholarbank.nus.edu.sg/bitstreams/1387f33c-4aca-4167-b20a-4850d030558f/download