Adaptive parameter servers

Machine learning (ML) has become an essential tool for solving problems that have traditionally been challenging for computers, for example in the fields of natural language processing, computer vision, and recommender systems. For large ML tasks, distributed training has become a necessity for keeping up with increasing dataset sizes and model complexity. A key challenge in distributed training is to synchronize model parameters among cluster nodes and to do so efficiently. Parameter servers (PSs) facilitate the implementation of distributed training by providing cluster-wide read and write access to the parameters, and transparently handling partitioning and synchronization in the background (either among the cluster nodes directly or via physically separate server nodes). In this thesis, we study the efficiency of PSs for ML tasks with sparse parameter access, i.e., tasks in which each update step reads and writes only a small (or tiny) part of the model. In a first step, we find that existing PSs are inefficient for such tasks: in our experiments, distributed implementations were slower than efficient single node implementations due to communication overhead. This inefficiency dramatically limits the utility of PSs and distributed training in general. With such inefficiency, distributed training in practice will be used only when it is indispensable, e.g., for very large models. And when distributed training is indeed employed, its inefficiency squanders hardware and energy resources. Starting from this observation, we investigate whether and how PS efficiency can be improved. The main idea of this thesis is to increase efficiency by making the PS adapt to the underlying ML task. We first present and evaluate a series of potential performance improvements in this direction, each making the PS more adaptive. In particular, we explore (i) to dynamically adapt the allocation of model parameters, i.e., to relocate parameters among nodes during training, according to where they are accessed, (ii) to adapt the management technique of the PS to the access patterns of individual parameters, i.e., to employ a suitable management technique for each parameter, and (iii) to adapt to the type of a parameter access, by supporting sampling (i.e., randomized) access directly. We find empirically that each of these aspects can improve PS efficiency. However, each aspect also makes the PS more complex to use, because the application (i.e., the component that interacts with the PS) needs to control adaptivity manually. To reduce complexity, we develop a mechanism that enables automatic adaptivity, i.e., adaptivity without requiring the application's manual control. With this mechanism, the application merely provides information about parameter accesses, in a way that naturally integrates into common ML systems. We describe a novel PS system—called AdaPS—that adapts to ML tasks automatically based on the information provided by this mechanism. AdaPS incorporates all adaptivity aspects presented in this thesis. It decides what to do (i.e., which management technique to use for a specific parameter and where to allocate each parameter) and when to do so. It does so automatically, i.e., without further user input, and dynamically, i.e., based on the current situation. In our experiments, AdaPS enabled efficient distributed ML training for multiple ML tasks: in contrast to previous PSs, it provided near-linear speed-ups over efficient single node implementations. With these results, we argue that PSs can be efficient for sparse ML tasks, and that this efficiency can be reached with limited additional effort from application developers. Efficient and easy-to-use PSs make distributed training (i) attractive for a wide range of use cases—thus enabling solutions to challenging problems—and (ii) squander fewer of our planet's resources.

Author dc:creator

• Renz-Wieland, Florian Alexander

Advisors dc:contributor.advisor

• Markl, Volker
• Gemulla, Rainer

Licence dc:rights.uri

• https://creativecommons.org/licenses/by/4.0/

Language dc:language.iso

en

Identifier URI

https://doi.org/10.14279/depositonce-16871

OAI identifier oai:identifier

oai:depositonce.tu-berlin.de:11303/18078