ARNI Continual Learning Working Group Spring Opening Meeting

As a potential topic for this project, we think we might consider the problem of long and short term memory in LLMs.  There has been recent interest from industry labs, e.g. Google (paper link) and Meta (paper link), in fitting an LLM with a long-term neural memory module to complement the short-term memory given by the context window.  Several threads relevant to ARNI could extend from this research direction.  For instance, we might consider cognitively-inspired benchmarks for LLM memory systems for lifelong learning, e.g., based on human-like tasks that might be difficult for autoregressive models.  Also, we could explore methodological work in LLM memory mechanisms based on our understanding of natural intelligence.  We are particularly interested in learning about relevant studies in neuroscience and cognitive science that could help constrain and inspire the methodological approaches.  Beyond these, one could imagine many other related directions of interest to ARNI.