Speaker: Mia Dai – ARNI Frontier Models for Neuroscience and Behavior Working Group

Speakers: Mia Dai, PhD student in the Paninski Group at Columbia University

Title: Sparse-View Interpretable 3D Animal Behavior Representations for Neural Encoding and Decoding

Abstract: A deeper understanding of brain function requires a precise, structured characterization of behavior. Yet, capturing behavior from video in a form suitable for scientific analysis remains a fundamental challenge. Many prior studies represent behavior via pose estimation or nonlinear video embeddings. However, pose tracking discards rich information beyond predefined keypoints, while nonlinear video embeddings lack interpretability. We address this limitation with SABLE (Sparse-view Animal Behavior Latent Embeddings), a self-supervised framework that leverages geometric inductive biases to learn behavior representations. By augmenting a multi-view transformer with priors from monocular depth and pose estimation, SABLE reconstructs 3D animal behavior from extremely sparse views while learning explicit 3D latent structure. Without ground-truth 3D labels, it reliably recovers 3D behavior from two-view videos, whereas state-of-the-art methods fail or yield degenerate solutions. Through comprehensive evaluation, we demonstrate that SABLE learns 3D representations that match or exceed prior SOTA performance in neural encoding and decoding. Our method establishes 3D-aware video embeddings that capture complex behavior, opening new avenues for studying brain-behavior relationships.

Zoom: Upon request @ [email protected]