Distributed and concurrent optimization for machine learning (part 2)

EN
/ Day 2
/ 12:00
/ Track 1

Machine learning has made considerable progress over the past decade, enabled by the widespread availability of large datasets, as well as by improved algorithms and models. Given the large computational demands of machine learning workloads, parallelism, implemented either through single-node concurrency or through multi-node distribution, has been a third key ingredient to advances in machine learning.

The goal of this tutorial lecture is to provide the audience with an overview of standard distribution techniques in machine learning, with an eye towards the intriguing trade-offs between synchronization and communication costs of distributed machine learning algorithms, on the one hand, and their convergence, on the other. The tutorial will focus on parallelization strategies for the fundamental stochastic gradient descent (SGD) algorithm, which is a key tool when training machine learning models, from classical instances such as linear regression, to state-of-the-art neural network architectures.

The tutorial will describe the guarantees provided by this algorithm in the sequential case, and then move on to cover both shared-memory and message-passing parallelization strategies, together with the guarantees they provide, and corresponding trade-offs. The presentation will conclude with a broad overview of ongoing research in distributed and concurrent machine learning. The lecture will assume no prior knowledge of machine learning or optimization beyond familiarity with basic concepts in algebra and analysis.

Dan Alistarh

IST Austria

Dan Alistarh is an Assistant Professor at IST Austria. His research focuses on concurrent data structures and distributed algorithms, and spans from algorithms and lower bounds to practical implementations.

Before IST, Dan was affiliated with ETH Zurich and Microsoft Research, Cambridge, UK. Prior to that, he was a Postdoctoral Associate at MIT CSAIL, working with Prof. Nir Shavit. He received his PhD from the EPFL, under the guidance of Prof. Rachid Guerraoui.