{"id":{"repo_id":"mississippi","oai_identifier":"oai:egrove.olemiss.edu:etd-1450"},"canonical_url":"https://search.dev.ndltd.org/etd/mississippi/oai:egrove.olemiss.edu:etd-1450","repository":{"repo_id":"mississippi","name":"University of Mississippi","base_url":"https://egrove.olemiss.edu/do/oai/"},"display":{"title":"The Effect Of Hyperparameters In The Activation Layers Of Deep Neural Networks","abstract":"Deep neural networks (DNNs), and artificial neural networks (ANNs) in general, have recently received a great amount of attention from both the media and the machine learning community at large. DNNs have been used to produce world-class results in a variety of domains, including image recognition, speech recognition, sequence modeling, and natural language processing. Many of most exciting recent deep neural network studies have made improvements by hardcoding less about the network and giving the neural network more control over its own parameters, allowing flexibility and control within the network. Although much research has been done to introduce trainable hyperparameters into transformation layers (GRU [7], LSTM [13], etc), the introduction of hyperparameters into the activation layers have been largely ignored. This paper serves several purposes: to (1) equip the reader with the background knowledge, including theory and best practices for DNNs, which help contextualize the contributions of this paper, (2) to describe and verify the effectiveness of current techniques in the literature that utilize hyperparameters in the activation layer, and (3) to introduce some new activation layers that introduce hyperparameters into the model, including activation pools (APs) and parametric activation pools (PAPs), and study the effectiveness of these new constructs on popular image recognition datasets.","abstract_html":"Deep neural networks (DNNs), and artificial neural networks (ANNs) in general, have recently received a great amount of attention from both the media and the machine learning community at large. DNNs have been used to produce world-class results in a variety of domains, including image recognition, speech recognition, sequence modeling, and natural language processing. Many of most exciting recent deep neural network studies have made improvements by hardcoding less about the network and giving the neural network more control over its own parameters, allowing flexibility and control within the network. Although much research has been done to introduce trainable hyperparameters into transformation layers (GRU [7], LSTM [13], etc), the introduction of hyperparameters into the activation layers have been largely ignored. This paper serves several purposes: to (1) equip the reader with the background knowledge, including theory and best practices for DNNs, which help contextualize the contributions of this paper, (2) to describe and verify the effectiveness of current techniques in the literature that utilize hyperparameters in the activation layer, and (3) to introduce some new activation layers that introduce hyperparameters into the model, including activation pools (APs) and parametric activation pools (PAPs), and study the effectiveness of these new constructs on popular image recognition datasets.","abstract_has_math":false,"creators":["Mcleod, Clay Lafayette"],"institution":null,"degree_name":"M.S. in Engineering Science","degree_level":"Thesis","degree_discipline":"Computer and Information Science","degree_department":null,"school":null,"contributors":["Dawn Wilkins","Byunghyun Jang","Yixin Chen"],"advisors":[],"committee_chairs":[],"committee_members":[],"year":2016,"date_issued":"2016-01-01T08:00:00Z","date_published":"2016-01-01T08:00:00Z","updated_at":"2026-07-24T03:05:36Z","subjects":["Activation Function","Deep Learning","Neural Network","Computer Sciences"],"languages":[],"rights":[],"rights_urls":[],"identifier_entries":[]},"links":{"outbound_url":"https://egrove.olemiss.edu/etd/451","outbound_label":"Repository record","outbound_source":"dc:identifier"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Dawn Wilkins","Byunghyun Jang","Yixin Chen"]},{"key":"dc:creator","label":"Author","values":["Mcleod, Clay Lafayette"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date.available","label":"Dc Date Available","values":["2019-06-27T07:00:00Z"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Computer and Information Science"]},{"key":"thesis:degree_level","label":"Degree Level","values":["Thesis"]},{"key":"thesis:degree_name","label":"Degree Name","values":["M.S. in Engineering Science"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Activation Function","Deep Learning","Neural Network","Computer Sciences"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["https://egrove.olemiss.edu/etd/451"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description.abstract","label":"Abstract","values":["Deep neural networks (DNNs), and artificial neural networks (ANNs) in general, have recently received a great amount of attention from both the media and the machine learning community at large. DNNs have been used to produce world-class results in a variety of domains, including image recognition, speech recognition, sequence modeling, and natural language processing. Many of most exciting recent deep neural network studies have made improvements by hardcoding less about the network and giving the neural network more control over its own parameters, allowing flexibility and control within the network. Although much research has been done to introduce trainable hyperparameters into transformation layers (GRU [7], LSTM [13], etc), the introduction of hyperparameters into the activation layers have been largely ignored. This paper serves several purposes: to (1) equip the reader with the background knowledge, including theory and best practices for DNNs, which help contextualize the contributions of this paper, (2) to describe and verify the effectiveness of current techniques in the literature that utilize hyperparameters in the activation layer, and (3) to introduce some new activation layers that introduce hyperparameters into the model, including activation pools (APs) and parametric activation pools (PAPs), and study the effectiveness of these new constructs on popular image recognition datasets."]},{"key":"dc:title","label":"Title","values":["The Effect Of Hyperparameters In The Activation Layers Of Deep Neural Networks"]}]}],"canonical_facts":{"dc:contributor":["Dawn Wilkins","Byunghyun Jang","Yixin Chen"],"dc:creator":["Mcleod, Clay Lafayette"],"dc:date.available":["2019-06-27T07:00:00Z"],"dc:description.abstract":["Deep neural networks (DNNs), and artificial neural networks (ANNs) in general, have recently received a great amount of attention from both the media and the machine learning community at large. DNNs have been used to produce world-class results in a variety of domains, including image recognition, speech recognition, sequence modeling, and natural language processing. Many of most exciting recent deep neural network studies have made improvements by hardcoding less about the network and giving the neural network more control over its own parameters, allowing flexibility and control within the network. Although much research has been done to introduce trainable hyperparameters into transformation layers (GRU [7], LSTM [13], etc), the introduction of hyperparameters into the activation layers have been largely ignored. This paper serves several purposes: to (1) equip the reader with the background knowledge, including theory and best practices for DNNs, which help contextualize the contributions of this paper, (2) to describe and verify the effectiveness of current techniques in the literature that utilize hyperparameters in the activation layer, and (3) to introduce some new activation layers that introduce hyperparameters into the model, including activation pools (APs) and parametric activation pools (PAPs), and study the effectiveness of these new constructs on popular image recognition datasets."],"dc:identifier":["https://egrove.olemiss.edu/etd/451"],"dc:subject":["Activation Function","Deep Learning","Neural Network","Computer Sciences"],"dc:title":["The Effect Of Hyperparameters In The Activation Layers Of Deep Neural Networks"],"thesis:degree_discipline":["Computer and Information Science"],"thesis:degree_level":["Thesis"],"thesis:degree_name":["M.S. in Engineering Science"]},"updated_at":"2026-07-24T03:05:36Z"}