Language & Vision

This project seeks to build an assistive technology that gives BLV people an audio-haptic sense of vision that is similarly interactive and flexible as the human visual system, enabling interactive shifts of spatial focus to “zoom in” or “pan” to different areas as needed, and providing information that ranges from the sensory (shape, texture, color), to the semantic (objects and their parts) to the relational (arrows of the scene graph), and on to the contextual level (past, future, knowledge base). Computer vision (including panoptic segmentation and scene understanding) and large language models are now sufficiently advanced and well-integrated to provide the information from cameras and learned contextual knowledge. The challenge is to make this information interactively accessible to BLV people to help them experience the visual world and, in particular, complex forms of art. Our system will be designed based on cognitive and neuroscientific insights (led by cognitive psychologist Emmanouil and computational neuroscientist Kriegeskorte) and, specifically, studies on scene gist, saliency, attention, and eye movements.