Multimodal Chatbot for Mental Health

Primary supervisor

Lizhen Qu

Chatbots for mental health are shown to be helpful for preventing mental health issues and improving the wellbeing of individuals, and to ease the burden on health, community and school systems.  However, the current chatbots in this area cannot interact naturally with humans and the types of interactions are limited to short text, predefined buttons etc. In contrast, psychologists in real-world interact with patients with multiple modalities, including accustic and visual information. Non-textual information is also essential for health observation and treatments of patients.

In this project, you will contribute to data collection and implementation of the chatbot for mental health in a multi-disciplinary team. Your task will focus on the multimodal perspective of the chatbot by allowing visual and accusitic responses apart from text messages. 

Student cohort

Single Semester
Double Semester

Aim/outline

The goal of this thesis is to collect multimodal data, construct a multimodal knowledge base, as well as generate multimodal responses for the chatbot. This work includes i) designing and implementating the multimodal components of the chatbot, ii) assisting in the construction of multimodal dialogue datasets; iii) conducting experiments to show the effectiveness of the multimodal perspective of the chatbot.

URLs/references

Cameron, Gillian, David Cameron, Gavin Megaw, Raymond Bond, Maurice Mulvenna, Siobhan O’Neill, Cherie Armour, and Michael McTear. "Towards a chatbot for digital counselling." In Proceedings of the 31st International BCS Human Computer Interaction Conference (HCI 2017) 31, pp. 1-7. 2017.Cameron, Gillian, David Cameron, Gavin Megaw, Raymond Bond, Maurice Mulvenna, Siobhan O’Neill, Cherie Armour, and Michael McTear. "Towards a chatbot for digital counselling." In Proceedings of the 31st International BCS Human Computer Interaction Conference (HCI 2017) 31, pp. 1-7. 2017.

Saxena, Ashutosh, Ashesh Jain, Ozan Sener, Aditya Jami, Dipendra K. Misra, and Hema S. Koppula. "Robobrain: Large-scale knowledge engine for robots." arXiv preprint arXiv:1412.0691 (2014). (https://cs.stanford.edu/people/asaxena/papers/robobrain-isrr2015.pdf) 

Required knowledge

Students should have excellent grades in machine learning and relevant math courses. Preference will be given to students who have strong written and oral communication skills, as well as strong programming skills. Students with an interest in pursuing PhD research or careers in research are especially encouraged to apply.