Lenka Zdeborova (Seminar Speaker)

Title: Phase transition in learning with neural networks 

Abstract: Statistical physics has studied exactly solvable models of neural networks for more than four decades. In this talk, we will put this line of work in perspective of recent questions stemming from deep learning. We will describe several types of phase transition that appear in the high-dimensional limit as a function of the amount of data. Discontinuous phase transitions are linked to adjacent algorithmic hardness. This so-called hard phase influences the behaviour of gradient-descent-like algorithms. We show a case where the hardness is mitigated by overparametrization, proposing that the benefits of overparametrization may be linked to the usage of a specific type of algorithm. We will also discuss recent progress in identifying phase transitions and their consequences in networks with attention layers and in sampling with generative diffusion-based networks.