ARNI Emerging Researchers Talk Series #3: Matteo Alleman

Title: Discovery of categorical concepts

Abstract: We seem to like reasoning in terms of discrete, logical categories, even in the face of continuous variation. A generous interpretation of this phenomenon is that we create abstractions about the world which enable powerful generalization to new situations — once I know what “hot-and-sour” is, I can instantly know if I will like any dish. But state of the art learning systems, and as far as we can tell our own brains, use highly distributed, continuous representations. A way that we reconcile these two ideas — abstract symbols and continuous representations — is to imagine that there are certain “component-level” directions in the vector space which are re-used across all instances which contain the component. Like the famous “king:queen :: man:woman” square from Miklov et al (2013). This provides a clear way of encoding abstract categories in continuous vector space, but is it possible to go the other way, and infer the categories from the vectors?
The first thing that may come to mind when hearing about “inferring categories” is clustering. But this has a limitation — you cannot put together “king” and “man” into a “male” category while also clustering “king” and “queen” into a “monarch” one. It also doesn’t quite seem right to say that “king” is 0.5 “male” and 0.5 “monarch” (a soft clustering), or that one cluster is a subset of the other (a hierarchical clustering). Instead, I want a way to group together items into multiple clusters based on their vector representations. When data are free to belong to multiple clusters (not in a sum-to-one probabilistic way) it becomes most natural to think of the problem as a matrix factorization, where we are trying to express our data as the product of an otherwise unconstrained binary matrix of item-cluster assignments (call it S), and a real-valued matrix of cluster means (call it W), i.e. X = SW. That is the model I will present on.