The Conversational Refreshable Tactile Display: Multimodal AI for people who are blind or have low vision

For decades, the primary way people who are blind or have low vision (BLV) interact with computers has been through screen readers. These linearise an application or document interface, presenting content as synthetic speech or on a single-line braille display, which the user navigates using keyboard shortcuts or touch gestures on a smartphone. Screen readers have a major limitation: they cannot show graphics. Instead, they rely on alternative text descriptions, which are often incomplete or inadequate.

This project will explore a transformative approach to making graphical information accessible to blind people by integrating two emerging technologies. The first is refreshable tactile displays (RTDs), which use arrays of electromechanically actuated pins to render tactile graphics that can be explored through touch. The RTD is the blind person's equivalent of the computer visual display. The second is genAI-powered conversational agents. These enable the blind user to control the RTD and obtain additional information about the graphic. Together, they form what we call a conversational RTD, a new multimodal interaction paradigm that addresses current barriers to graphical access.

This project will investigate this new paradigm and explore its potential across a choice of applications, including accessible spreadsheets, concept mapping, a world atlas, or an illustrated book of famous buildings.