Konstantin Willeke 

Date: October 8, 2025
Time: 2:00pm
Zoom: Upon request @ [email protected]

Title:
OmniMouse: Scaling properties of multi-modal, multi-task Brain Models on 150B Neural Tokens

Abstract:
Scaling data and artificial neural networks has transformed AI, driving breakthroughs in language and vision. Whether similar principles apply to modeling brain activity remains unclear. Here we leveraged a dataset of 3.3 million neurons from the visual cortex of 78 mice across 323 sessions, totaling more than 150 billion neural tokens recorded during natural movies, images and parametric stimuli, and behavior. We train multi-modal, multi-task transformer models (1M–300M parameters) that support three regimes flexibly at test time: neural prediction (predicting neuronal responses from sensory input and behavior), behavioral decoding (predicting behavior from neural activity), neural forecasting (predicting future activity from current neural dynamics), or any combination of the three. We find that performance scales reliably with more data, but gains from increasing model size saturate — suggesting that current brain models are limited by data rather than compute. This inverts the standard AI scaling story: in language and computer vision, massive datasets make parameter scaling the primary driver of progress, whereas in brain modeling — even in the mouse visual cortex, a relatively simple and low-resolution system — models remain data-limited despite vast recordings. These findings highlight the need for richer stimuli, tasks, and larger-scale recordings to build brain foundation models. The observation of systematic scaling raises the possibility of phase transitions in neural modeling, where larger and richer datasets might unlock qualitatively new capabilities, paralleling the emergent properties seen in large language models.

Vinam Arora and Ji Xia

Date: August 27, 2025
Location: JLGSC-L3-079
Time: 2:00pm
Zoom: Upon request @ [email protected]

Title and Abstracts:

1st Speaker: Vinam Arora
Title: Know Thyself by Knowing Others: Learning Neuron Identity from Population Context

Abstract: Identifying the functional identity of individual neurons is essential for interpreting circuit dynamics, yet remains a major challenge in large-scale in vivo recordings where anatomical and molecular labels are often unavailable. Here we introduce NuCLR, a self-supervised framework that learns context-aware representations of neuron identity by modeling each neuron's role within the broader population. NuCLR employs a spatiotemporal transformer that captures both within-neuron dynamics and across-neuron interactions, and is trained with a sample-wise contrastive objective that encourages stable, discriminative embeddings across time. Across multiple open-access datasets, NuCLR outperforms prior methods in both cell type and brain region classification. It enables zero-shot generalization to entirely new populations—without retraining or access to stimulus labels—offering a scalable approach for real-time, functional decoding of neuron identity across diverse experimental settings.

2nd Speaker:
Title: In painting the neural picture: Inferring Unrecorded Brain Area Dynamics from Multi-Animal Datasets.

Abstract: Understanding how the brain drives memory-guided movements requires recording neural activity from the motor cortex and interconnected subcortical areas. Neuropixels probes now allow simultaneous recordings from subsets of these areas, but no single session captures all areas of interest, and different neurons are sampled from each area across sessions. This poses a key challenge: how to integrate neural data across sessions to reconstruct the complete multi-area picture. We address this with a transformer-based autoencoder that aligns neural activity into a shared latent space across sessions and animals, separately for each brain area, including those not recorded in a given session. This approach enables single-trial analysis of multi-area neural dynamics from all areas of interest. I am now working on improving this method, and will discuss both its present challenges and promising directions for future work.

Memming Park

Date: July 30th, 2025
Location: JLGSC-L3-079
Time: 2:00pm
Zoom: Upon request @ [email protected]

Title: Meta-dynamical state space modeling for integrative neural data analysis

Abstract:
Uncovering the organizing principles of neural systems requires integrating information across diverse datasets—each alone offering a limited view and signal-to-noise ratio, but together revealing coherent dynamical structures. We present a meta-dynamical state-space modeling framework that learns a shared solution space of neural dynamics from heterogeneous recordings across sessions, animals, and tasks. By capturing cross-dataset similarity and variability on a low-dimensional manifold that spans a space of dynamical systems, our approach enables few-shot inference, rapid adaptation to new recordings, and discovery of latent dynamical motifs that underlie behavior. We demonstrate its utility in modeling motor cortex activity, revealing dynamics that generalize across individuals and track the change in dynamics during learning. We argue that for understanding neural computation and real-time neuroscience applications, our approach is well-suited as a foundation model for integrative neuroscience.

Guillaume Lajoie

Title: POSSM: Generalizable, real-time neural decoding with hybrid state-space models

Abstract:
Real-time decoding of neural spiking data is a core aspect of neurotechnology applications such as brain-computer interfaces, where models are subject to strict latency constraints. Traditional methods, including simple recurrent neural networks, are fast and lightweight but are less equipped for generalization to unseen data. In contrast, recent Transformer-based approaches leverage large-scale neural datasets to attain strong generalization performance. However, these models typically have much larger computational requirements and are not suitable for settings requiring low latency or limited memory. To address these shortcomings, we present POSSM, a novel architecture that combines individual spike tokenization and an input cross-attention module with a recurrent state-space model (SSM) backbone, thereby enabling (1) fast and causal online prediction on neural activity and (2) efficient generalization to new sessions, individuals, and tasks through multi-dataset pre-training. We evaluate our model’s performance in terms of decoding accuracy and inference speed on monkey reaching datasets, and show that it extends to clinical applications, namely handwriting and speech decoding. Notably, we demonstrate that pre-training on monkey motor-cortical recordings improves decoding performance on the human handwriting task, highlighting the exciting potential for cross-species transfer. In all of these tasks, we find that POSSM achieves comparable decoding accuracy with state-of-the-art Transformers, at a fraction of the inference cost. These results suggest that hybrid SSMs may be the key to bridging the gap between accuracy, inference speed, and generalization when training neural decoders for real-time, closed-loop applications.

Frontier Models for Neuroscience and Behavior Working Group Priorly Known as Animal Behavior Video Analysis Working Group

Srini Turaga

Title: Whole-body simulation of realistic fruit fly locomotion with deep reinforcement learning

Abstract:

The body of an animal determines how the nervous system produces behavior. Therefore, detailed modeling of the neural control of sensorimotor behavior requires a detailed model of the body. Here we contribute an anatomically-detailed biomechanical whole-body model of the fruit fly {\em Drosophila melanogaster} in the \mujoco physics engine. Our model is general-purpose, enabling the simulation of diverse fly behaviors, both on land and in the air. We demonstrate the generality of our model by simulating realistic locomotion, both flight and walking. To support these behaviors, we have extended \mbox{MuJoCo} with phenomenological models of fluid forces and adhesion forces. Through data-driven end-to-end reinforcement learning, we demonstrate that these advances enable the training of neural network controllers capable of realistic locomotion along complex trajectories based on high-level steering control signals. With a visually guided flight task, we demonstrate a neural controller that can use the vision sensors of the body model to control and steer flight. Our project is an open-source platform for modeling neural control of sensorimotor behavior in an embodied context.

Ugne Klibaite

Title: Mapping the landscape of social of social behavior using high-resolution 3D tracking of freely interacting animals

Abstract:
Social interaction is a fundamental component of animal behavior. However, we lack tools to describe it with quantitative rigor, limiting our understanding of its principles and the neuropsychiatric disorders, like autism, that perturb it. To address these limitations, I and collaborators have developed a technique for high-resolution 3D tracking of freely interacting animals and their body-wide social touch patterns, solving the challenging subject occlusion and part assignment problems using 3D geometric reasoning, graph neural networks, and semi-supervised learning. Using this technology, I have collected and annotated over 34 million 3D postures in interacting rats, featuring five new monogenic autism models lacking reports of social behavioral phenotypes. I will introduce a novel multi-scale approach which I have used to identify a rich landscape of stereotyped interactions, synchrony, and body contact across strains. This deep phenotyping approach revealed a spectrum of changes in rat autism models and in response to amphetamine, and this framework has the potential to facilitate quantitative studies of social behaviors and their neurobiological underpinnings.

Carl Vondrick

Title: Multimodal Learning from Pixels to People

Abstract:
People experience the world through modalities of sight, sound, words, touch, and more. By leveraging their natural relationships and developing multimodal learning methods, my research creates artificial perception systems with diverse skills, including spatial, physical, logical, and cognitive abilities, for flexibly analyzing visual data. This multimodal approach provides versatile representations for tasks like 3D reconstruction, visual question answering, and object recognition, while offering inherent explainability and excellent zero-shot generalization across tasks. By closely integrating diverse modalities, we can overcome key challenges in machine learning and enable new capabilities for computer vision, especially for the many upcoming applications where trust is required.