Novel optimizers for multi-objective deep learning

In many real-world problems, access to solutions requires balancing (trade-off) between conflicting objectives, so they must be treated as Multi-objective Optimization problems. Multi-objective optimizers are suitable for training machine learning algorithms as they can minimize several loss functions simultaneously. Additionally, providing a set of weight configurations with Pareto front solutions in multi-objective optimization offers flexibility and a desirable trade-off options to the end user in many machine learning tasks. With that goal in mind, this thesis centers around developing novel multi-objective optimizers for training artificial neural networks to provide a variety of non-dominated model parameters across the Pareto front spectrum. Adaptive Moment Estimation (Adam) is one of the most commonly used gradient-based optimization strategies in the field of neural networks. Adam is a single-objective single-solution optimizer that emerges as the victor in the majority of competitions. However, committing to a single loss function often fails to capture the full complexity of the underlying problem and causes models to overfit to a particular objective. Additionally, with a single network parameter (weight configurations), the Adam optimizer operates without independent searching agents. Therefore, the global search is not permissible as multiple agents do not communicate with one another. In this thesis, we introduced a population-based Adam optimizer and incorporated it into the framework of multi-objective optimization to develop Multi-objective Adam Optimizer (MAdam). It required dismantling the structure of the Adam optimizer and integrating the exploitation of Adam optimizer with the exploration strategy of a population-based algorithm and the dominance strategy of multi-objective optimization. To enhance the exploratory capability of MAdam optimizer, we integrated an opposition-based scheme into the MAdam framework, as global search is necessary for escaping local optima in gradient-based multi-objective optimization. This resulted in Opposition-based Multi-objective Adam Optimizer (OMAdam). Aside from these two optimizers, we also introduced a gradient-free multi-objective optimizer, Multi-objective Coordinate Search (MOCS), which is a population-based coordinate search method. Due to the computational efficiency and memory requirements for training ANNs, we placed an emphasis on the gradient-based optimizers and provided experimental results obtained by implementing MAdam and OMAdam optimizers for deep learning tasks in medical image analysis.

Name thesis:degree_name

Doctor of Philosophy (PhD)

Discipline thesis:degree_discipline

Computer Science

Grantor

University of Ontario Institute of Technology

Year dc:date.issued

2025

Author dc:creator

• Nikbakhtsarvestani, Farzaneh

Advisors dc:contributor.advisor

• Ebrahimi, Mehran
• Rahnamayan, Shahryar

Language dc:language.iso

en

Handle dc:identifier.uri

https://hdl.handle.net/10155/2071

OAI identifier oai:identifier

oai:ontariotechu.scholaris.ca:10155/2071