ARNI Emerging Researchers Talk Series #1: Rahul Ramesh

Deep networks are increasingly trained on data from multiple tasks with the goal of sharing synergistic information across related tasks. A language model, for example, is trained on 10 trillion tokens on tasks ranging from programming, finance, trivia to translation and a vision model is trained on over a billion images for tasks like object recognition, depth prediction and semantic segmentation. With this motivation, in this talk, I will present the principles behind how to optimally train on multiple tasks and attempt to answer why we are able to learn on these tasks. In the first part of the talk we develop a theory that shows that dissimilar tasks fight for model capacity when trained together. We use this insight to design Model Zoo — a learner that splits its capacity to train many small models on related subsets of tasks — which is state-of-the-art for task-incremental continual learning. In the second half of this talk, we show that typical tasks are highly redundant functions of the input, i.e., the subspaces that vary the most and ones that vary the least are both highly predictive of typical tasks. This result suggests that there are many subspaces that can be used to solve typical tasks, which allows us to learn a shared representation for these tasks. We believe that organisms choose to solve redundant tasks because they are the only ones that agents with bounded resources can readily learn.

Rahul Ramesh is a 6th year PhD student at the University of Pennsylvania in the department of computer and information science and is advised by Pratik Chaudhari. He previously received his B.Tech from the Indian Institute of Technology Madras in Computer science and Engineering. Rahul is interested in using perspectives from statistical learning theory, information theory and neuroscience to study self-supervised and multitask learning.