Internal Summer 2025 Talk Series
Title: "Building artificial intelligence models to understand the human brain and mind"
Abstract:
Presentation:
Title: Foundation Models for Robotic Manipulation: Opportunities and Challenges
Bio:
Yunzhu Li is an Assistant Professor of Computer Science at Columbia University. Before joining Columbia, he was an Assistant Professor at UIUC CS and spent time as a Postdoc at Stanford, collaborating with Fei-Fei Li and Jiajun Wu. Yunzhu earned his PhD from MIT under the guidance of Antonio Torralba and Russ Tedrake. Yunzhu’s work has been recognized with the Best Paper Award at ICRA, the Best Systems Paper Award, and as a Finalist for the Best Paper Award at CoRL. Yunzhu is also the recipient of the AAAI New Faculty Highlights, the Sony Faculty Innovation Award, the Amazon Research Award, the Adobe Research Fellowship, and was selected as the First Place Recipient of the Ernst A. Guillemin Master’s Thesis Award in AI and Decision Making at MIT. His research has been published in top journals and conferences, including Nature and Science, and featured by major media outlets such as CNN, BBC, and The Wall Street Journal.
Abstract:
Foundation models, such as GPT, have marked significant achievements in the fields of natural language and vision, demonstrating exceptional abilities to adapt to new tasks and scenarios. However, physical interaction—such as cooking, cleaning, or caregiving—remains a frontier where foundation models and robotic systems have yet to achieve the desired level of adaptability and generalization. In this talk, I will discuss the opportunities for incorporating foundation models into classic robotic pipelines to endow robots with capabilities beyond those achievable with traditional robotic tools. The talk will focus on three key improvements in (1) task specification, (2) low-level, and (3) high-level scene modeling. The central idea behind this research is to translate the commonsense knowledge embedded in foundation models into structural priors that can be integrated into robot learning systems. This approach leverages the strengths of different modules (e.g., VLM for task interpretation and constrained optimization for motion planning), achieving the best of both worlds. I will demonstrate how such integration enables robots to interpret instructions provided in free-form natural language, and how foundation models can be augmented with additional memory mechanisms, such as an action-conditioned scene graph, to handle a wide range of real-world manipulation tasks. Toward the end of the talk, I will discuss the limitations of the current foundation models, challenges that still lie ahead, and potential avenues to address these challenges.
Presentation: Link and for external access email [email protected]
Title: The first half: " I know it when I see it: creativity with vibe" and The second half: "brain decodes deep nets"
Abstract:
Presentation: Link and for external access email [email protected]
Slides: "I know it when I see it" - Link and "Brain Decodes Deep Nets" - Link
Title: Research in Natural Language Processing at Columbia
Presentation: Link and for external access email [email protected]
Title: How to generalize
Abstract:
Brains and AI have some principles in common that allow them to draw good conclusions from their training. Going beyond that training data is a key challenge called generalization. We’ll talk about how circuits and learning rules at the large and small scales help both brains and AI generalize.
Presentation: Link and for external access email [email protected]