Global ETD Search
Search theses and dissertations gathered from participating repositories worldwide. Every result links back to the library that holds it. No account is needed.
Results
Showing 1 to 20 of 25 for “"ReLU"”.
-
Reductions of ReLU neural networks to linear neural networks and their applications
… In this thesis, we describe reductions of ReLU neural networks to linear neural networks, under various general condition of network architectures, loss functions and datasets. When such conditions are met, one can adapt techniques used in the theory of linear neural networks to study ReLU …
-
Optimal Approximation of High-dimensional Functions on Smooth Manifolds Using Deep ReLU Neural Networks
… errors by deep neural networks with ReLU activation functions do not depend on the ambient dimension $D$ of the function but only on its lower manifold dimension $d$, in a precise sense. Our result improves existing results from the literature where approximation and generalization …
-
Evaluation of the smoothing activation function in neural networks for business applications
… the Softplus and Rectified Linear Unit (ReLU) activation functions were introduced in 2000 and 2001 respectively, with the latter emerging as the more popular choice of activation function in neural networks. Notably, the ReLU activation function maintains a high degree of gradient …
-
On the computational power of RNNs
… finite precision RNNs with one hidden layer and ReLU activation and finite precision GRUs are exactly as computationally powerful as deterministic nice automata. Allowing arbitrary precision, we prove that RNNs with one hidden layer and ReLU activation are at least as computationally powerful as …
-
Optimization for Deep Learning: Bridging the Theory-Practice Gap
… tight bounds on memorization capacity of ReLU networks. We present results showing that width-$\Theta(\sqrt{n})$ ReLU networks can memorize arbitrary $n$ data points, which brings down the existing width requirement $n$ to a much more realistic number. The third part discusses implicit …
-
Mathematical Study on the Expressive Power of Machine Learning and Applications in Optimal Filtering Problems
… basis function. We specifically focus on GAT-ReLU, a variant of GAT with ReLU activation, which exhibits enriched activation patterns and smoother solutions compared to traditional ReLU activation. By selecting proper basis functions (continuous and high-variation), GAT-ReLU enhances the …
-
Optimization in Deep Learning: Structured, Realistic and Interpretable Learning for Decision-Making
… that leverage the piecewise-linear structure of ReLU networks, reducing the original hard-to-solve mixed-integer program into tractable linear programs. To ensure realism, we introduce constraints that restrict decisions to lie on the data manifold. We then extend this framework to any …
-
Completion of hinge loss has an implicit bias
… the gradient descent iterates of two-layer ReLU nets on the inner product exactly. This result is studied further to extrapolate conclusions for neural network optimization.
-
Why deep neural networks for function approximation
… use a combination of rectifier linear units (ReLUs) and binary step units, two of the most popular types of activation functions. Our analysis builds on a simple observation: the multiplication of two bits can be represented by a ReLU.
-
Joint Training Methods for Tandem and Hybrid Speech Recognition Systems using Deep Neural Networks
… the sigmoid and rectified linear unit (ReLU), are parameterised to enable automatic learning of function forms. Experiments using conversational telephone speech (CTS) Mandarin data result in an average of 3.4% and 2.2% relative character error rate (CER) reduction with sigmoid and ReLU …
-
Theory and Applications of Matrix Completion in Genomics Datasets
… infinite-width neural network with ReLU activation, and develop a framework applying the NTK to matrix completion. We explore a specific application of this framework, using the Connectivity Map dataset of gene expression data for various cells and perturbations, demonstrating …
-
PFHE: partially homomorphic encryption on CNN inference
… the implementation of nonlinear activation like ReLU in CNN with high precision, without the consumption of a significant amount of multiplications. We evaluate our framework under the ImageNet dataset. For the first six convolutional layers, our method achieves at least 4.12× speedup in latency …
-
Towards Refinement for Measuring Subcutaneously Transplanted Tumour Models in Mice
… the best results. Using the layer FC7 before RELU activation for extraction in the VGC-Face model, a tumour recognition rate of 98.86% was achieved. This was increased to 100% through a semi-automatic step with detection repeated on cropped versions of negatively classified images. Finally, …
-
Deep Reinforcement Learning for Multirotor Flight Control: A Comparative Study of Sim-to-Real Training and Real-World Performance
… % overshoot. Among activation functions tested, ReLU outperformed tanh and ELU, lowering steady-state error by up to 36 % and inference time by 26 %. Post-training quantization further reduced inference latency by ≈40 % with negligible performance loss. Incorporating trajectory tracking during …
-
Demystifying deep network architectures : from theory to applications
… and demonstrate a "no free lunch" behavior in ReLU networks. Finally, a practical use case in the industry will be discussed (Chapter 7), where we design and scale up vision foundation models, again, without any training cost.
-
Machine Learning for Physics Applications: Quantum Algorithm Design, Chaotic Dynamics, and Mass Spectrometry
… distributed in the domain, it avoids the dying ReLU problem in the first layer, and training is accelerated since the network starts in a favourable state. Collectively, these studies demonstrate how machine learning serves as a tool for quantum algorithm design and mass spectrometry image data …
-
Artificial intelligence-enabled transaction prediction.
… two hidden layers of 10 neurons each with ReLU activation, and a dropout rate of 0.2 to prevent overfitting. The output layer has 2 neurons with a sigmoid activation function to output separate probabilities, thus balancing complexity and enhancing class differentiation. The research made …
-
Approximate Inference in Bayesian Neural Networks and Translation Equivariant Neural Processes
… show that for single-hidden layer networks with ReLU activation functions, there are fundamental limitations concerning the representation of in-between uncertainty: increased uncertainty in between well separated regions of low uncertainty. We show that this theoretical limitation doesn’t apply …
-
Exploring Deep Learning Methods for Discovering Features in Speech Signals
… features learnt on raw signals using Gaussian-ReLU Restricted Boltzmann Machines can achieve accuracy close to that achieved with the best traditional features. These features are, however, learnt using a generative model that ignores domain knowledge. We develop methods to discover features …
-
Testing Federated Learning Privacy Through Gradient Leakage Attacks
… analyzes the gradients of linear layers with ReLU activations to show that the special structure of these commonly used layers allows for exact reconstruction of batched client data, simultaneously theoretically justifying the success of prior optimization-based attacks and significantly …
Page 1 of 2