Multi-resource-cost Optimization for Neural Networks Models Working Group (NNMS)

Title: Scope of the working group, example project, and literature

Short Description: From Nikolaus Kriegeskorte’s (Professor of Psychology and of Neuroscience (in the Mortimer B. Zuckerman Mind Brain Behavior Institute) lab, Eivinas Butkus (grad student) will show an example of a modeling project optimizing energetic demands along with accuracy in a vision task, and Josh Ying (grad student) will give a sense of the literature

More about NNMS:
Neural network models are typically set up with a fixed architecture that defines the number of nodes and the connectivity, and are unrolled for a fixed number of timesteps to obtain a computational graph for backpropagation. This amounts to fixing the resources that a physical implementation in a biological brain or dedicated engineered system would require in terms of space (to accommodate nodes and connections), time (to execute the steps), and energy. The fixed architecture of neural network models allows us to limit the resource requirements and discover what level of performance is possible through optimization. However, it makes it difficult to explore the tradeoffs between the multiple resources. For example, would a smaller network that runs for more timesteps give preferable results according to a joint cost of nodes, connections, time, energy, and error? It would be useful to be able to flexibly trade off resources against each other and against task performance as part of the optimization of a single model, rather than having to train many models (each with a fixed vector of costs) to explore the space of solutions. We will develop (1) ways to quantify space, time, and energy costs of neural network models and (2) differentiable objectives that enable efficient joint minimization of the costs of multiple resources. Such methods could help us understand biological neural mechanisms that emerge from particular profiles of resource costs and behavioral affordances and also to engineer more efficient AI for resource-limited devices.

Zoom Link: https://columbiauniversity.zoom.us/j/97052575063?pwd=SllDVFd4VlA2TnN4RDV3VVJ3b2lldz09