{"id":{"repo_id":"york","oai_identifier":"oai:yorkspace.library.yorku.ca:10315/32337"},"canonical_url":"https://search.dev.ndltd.org/etd/york/oai:yorkspace.library.yorku.ca:10315/32337","repository":{"repo_id":"york","name":"York University","base_url":"https://yorkspace.library.yorku.ca/oai/request"},"display":{"title":"Higher Order Recurrent Neural Network for Language Modeling","abstract":"In this thesis, we study novel neural network structures to better model long term dependency in sequential data. We propose to use more memory units to keep track of more preceding states in recurrent neural networks (RNNs), which are all recurrently fed to the hidden layers as feedback through different weighted paths. By extending the popular recurrent structure in RNNs, we provide the models with better short-term memory mechanism to learn long term dependency in sequences. Analogous to digital filters in signal processing, we call these structures as higher order RNNs (HORNNs). Similar to RNNs, HORNNs can also be learned using the back-propagation through time method. HORNNs are generally applicable to a variety of sequence modelling tasks. In this work, we have examined HORNNs for the language modeling task using two popular data sets, namely the Penn Treebank (PTB) and English text8 data sets. Experimental results have shown that the proposed HORNNs yield the state-of-the-art performance on both data sets, significantly outperforming the regular RNNs as well as the popular LSTMs.","abstract_html":"In this thesis, we study novel neural network structures to better model long term dependency in sequential data. We propose to use more memory units to keep track of more preceding states in recurrent neural networks (RNNs), which are all recurrently fed to the hidden layers as feedback through different weighted paths. By extending the popular recurrent structure in RNNs, we provide the models with better short-term memory mechanism to learn long term dependency in sequences. Analogous to digital filters in signal processing, we call these structures as higher order RNNs (HORNNs). Similar to RNNs, HORNNs can also be learned using the back-propagation through time method. HORNNs are generally applicable to a variety of sequence modelling tasks. In this work, we have examined HORNNs for the language modeling task using two popular data sets, namely the Penn Treebank (PTB) and English text8 data sets. Experimental results have shown that the proposed HORNNs yield the state-of-the-art performance on both data sets, significantly outperforming the regular RNNs as well as the popular LSTMs.","abstract_has_math":false,"creators":["Bidgoli, Rohollah Soltani"],"institution":null,"degree_name":null,"degree_level":null,"degree_discipline":null,"degree_department":null,"school":null,"contributors":[],"advisors":["Jiang, Hui"],"committee_chairs":[],"committee_members":[],"year":2016,"date_issued":"2016-09-20","date_published":"2016-09-20","updated_at":"2026-07-24T06:33:43Z","subjects":["Computer science"],"languages":["en"],"rights":["Author owns copyright, except where explicitly noted. Please contact the author directly with licensing requests."],"rights_urls":[],"identifier_entries":[]},"links":{"outbound_url":"http://hdl.handle.net/10315/32337","outbound_label":"Handle","outbound_source":"dc:identifier.uri"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor.advisor","label":"Advisor","values":["Jiang, Hui"]},{"key":"dc:creator","label":"Author","values":["Bidgoli, Rohollah Soltani"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date.accessioned","label":"Dc Date Accessioned","values":["2016-09-20T18:56:37Z"]},{"key":"dc:date.available","label":"Dc Date Available","values":["2016-09-20T18:56:37Z"]},{"key":"dc:date.issued","label":"Date","values":["2016-09-20"]},{"key":"dc:type","label":"Dc Type","values":["Electronic Thesis or Dissertation"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Computer science"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:language.iso","label":"Language (ISO)","values":["en"]},{"key":"dc:rights","label":"Dc Rights","values":["Author owns copyright, except where explicitly noted. Please contact the author directly with licensing requests."]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier.uri","label":"Identifier URI","values":["http://hdl.handle.net/10315/32337"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description.abstract","label":"Abstract","values":["In this thesis, we study novel neural network structures to better model long term dependency in sequential data. We propose to use more memory units to keep track of more preceding states in recurrent neural networks (RNNs), which are all recurrently fed to the hidden layers as feedback through different weighted paths. By extending the popular recurrent structure in RNNs, we provide the models with better short-term memory mechanism to learn long term dependency in sequences. Analogous to digital filters in signal processing, we call these structures as higher order RNNs (HORNNs). Similar to RNNs, HORNNs can also be learned using the back-propagation through time method. HORNNs are generally applicable to a variety of sequence modelling tasks. In this work, we have examined HORNNs for the language modeling task using two popular data sets, namely the Penn Treebank (PTB) and English text8 data sets. Experimental results have shown that the proposed HORNNs yield the state-of-the-art performance on both data sets, significantly outperforming the regular RNNs as well as the popular LSTMs."]},{"key":"dc:title","label":"Title","values":["Higher Order Recurrent Neural Network for Language Modeling"]}]}],"canonical_facts":{"dc:contributor.advisor":["Jiang, Hui"],"dc:creator":["Bidgoli, Rohollah Soltani"],"dc:date.accessioned":["2016-09-20T18:56:37Z"],"dc:date.available":["2016-09-20T18:56:37Z"],"dc:date.issued":["2016-09-20"],"dc:description.abstract":["In this thesis, we study novel neural network structures to better model long term dependency in sequential data. We propose to use more memory units to keep track of more preceding states in recurrent neural networks (RNNs), which are all recurrently fed to the hidden layers as feedback through different weighted paths. By extending the popular recurrent structure in RNNs, we provide the models with better short-term memory mechanism to learn long term dependency in sequences. Analogous to digital filters in signal processing, we call these structures as higher order RNNs (HORNNs). Similar to RNNs, HORNNs can also be learned using the back-propagation through time method. HORNNs are generally applicable to a variety of sequence modelling tasks. In this work, we have examined HORNNs for the language modeling task using two popular data sets, namely the Penn Treebank (PTB) and English text8 data sets. Experimental results have shown that the proposed HORNNs yield the state-of-the-art performance on both data sets, significantly outperforming the regular RNNs as well as the popular LSTMs."],"dc:identifier.uri":["http://hdl.handle.net/10315/32337"],"dc:language.iso":["en"],"dc:rights":["Author owns copyright, except where explicitly noted. Please contact the author directly with licensing requests."],"dc:subject":["Computer science"],"dc:title":["Higher Order Recurrent Neural Network for Language Modeling"],"dc:type":["Electronic Thesis or Dissertation"]},"updated_at":"2026-07-24T06:33:43Z"}