AI-based classification of user behavioural responses in VR systems

Presence is the subjective feeling of "being there" that differentiates virtual reality (VR) systems from other forms of computer interfaces, and has long been thought to be beneficial in applications of VR such as training and psychotherapy. The quantification of presence has largely revolved around the use of subjective questionnaires, despite longstanding problems with existing questionnaires. Behavioural responses (i.e., watching users of VR systems for behaviours such as attempting to interact with virtual objects, or avoiding virtual obstacles as if they were real) are a good alternative offering objective evidence of a presence response to a VR system. Yet measuring behavioural responses is time-consuming and labour-intensive, requiring researchers to record video, synchronize it with streams from the VR viewer's point of view, and manually coding VR events and the participant behavioural response. 

This project focuses on automating this process through the training of AI models to detect the aforementioned virtual events and participant behavioural responses from recorded video. The AI models would classify behavioural responses (e.g., attempts to interact with virtual objects, facial expressions, utterances, etc.) and develop a tool to assist researchers quantifying presence in virtual reality experiments.