Speaker: Pratik Chaudhari ARNI WG Multi-resource-cost optimization of neural network models
Date: September 17, 2026
Location: ZI L7-119
Time: 2:00pm
Title: How Has Learning Shaped Our World?
Abstract:
Almost every sub-field of science, engineering and business is rapidly adopting deep learning technology. I will argue that it is really the properties of natural data derived from the physical world that makes deep networks so effective. Correlations of typical inputs (images, text, sounds, and even tabular data) are “sloppy”, i.e., they exhibit geometric decay. Sloppy systems are neither perfectly robust nor perfectly tuned but instead exhibit a wide range of sensitivities that allows them to strike a balance between robustness and adaptability. This is what enables deep networks to generalize to new data.
I will show that the training process in deep learning explores a remarkably low dimensional manifold, as low as three. Networks with a wide variety of architectures, sizes, optimization and regularization methods lie on the same manifold. Networks being trained on different tasks (e.g., different subsets of ImageNet) using different methods (e.g., supervised, transfer, meta, semi and self-supervised learning) also lie on the same low-dimensional manifolds. This is proof that there exist neural architectures with extremely few parameters—as few as 50—that can reproduce the predictions of networks with millions of weights. Architectures that exploit this phenomenon can reduce the energy required to build deep learning systems by multiple orders of magnitude.
I will show that typical tasks that we perform with deep learning are highly redundant functions of their inputs. Many perception tasks, from visual recognition, semantic segmentation, optical flow, depth estimation, to vocalization discrimination, can be predicted extremely well regardless of whether data is projected in the principal subspace where it varies the most, some intermediate subspace with moderate variability—or the bottom subspace where data varies the least. Any feature set is near-sufficient for most tasks. Ontogenetic learning refines neural circuitry to improve this feature set. I will argue that this redundancy could be—the serendipitous—consequence of the sensory apparatus of an organism adapting to a changing stream of ecological tasks.
Principles that inform learning in artificial agents can also shed light upon learning in biological organisms. To that end, I will describe results on developing visual perception for autonomous robots (flying platforms and quadrupeds) that operate in different environments (harsh daylight as well as night-time, indoors and outdoors), at very high speeds (without motion blur) and with very little energy.
Zoom Link: Upon request @ [email protected]