Evaluating Generative AI Chatbots for Student Learning Support

Primary supervisor

Sadia Nawaz

Co-supervisors

Generative AI chatbots are increasingly being adopted in higher education to support student learning and provide timely responses to questions. Within the Faculty of Information Technology at Monash University, AI-powered chatbots have been integrated into Ed Discussion (Ed Forum) for several units to assist teaching teams in managing online discussions and supporting student engagement.

 

While these tools offer significant potential, there is limited understanding of how students use AI chatbot services, the quality of responses they generate, how teaching teams interact with AI-suggested replies, and the practices used to train and maintain these systems over time.

 

This research project will investigate the effectiveness of generative AI chatbots in online learning environments and examine whether they can augment and support teaching teams in providing high-quality, timely assistance to students.

 

 

Aim/outline

The aim of this project is to analyse the use and effectiveness of generative AI chatbots in supporting communication and learning within university online discussion forums.

Depending on the student's interests, the project may involve:

  • analysing patterns of student interaction with AI-powered chatbots in Ed Discussion
  • evaluating the quality and relevance of AI-generated responses across different units and contexts
  • investigating how teaching staff use, modify, and adopt AI-generated reply suggestions
  • examining approaches used by teaching teams to train, configure, and update chatbot systems
  • analysing quantitative and qualitative data from online forum interactions
  • identifying opportunities and challenges associated with AI-supported learning support
  • developing evidence-based recommendations for future deployment of generative AI chatbots in higher education

The project combines research in Human-Computer Interaction (HCI), Artificial Intelligence, Learning Analytics, and Computing Education.

Required knowledge

Students should have an interest in one or more of the following areas:

  • Artificial Intelligence and Generative AI
  • Human-Computer Interaction (HCI)
  • Learning Analytics
  • Computing Education
  • Educational Technologies
  • Data Science

Experience with one or more of the following would be beneficial:

  • Python
  • Data analysis and visualisation
  • Qualitative research methods
  • Statistics
  • Natural Language Processing
  • User experience research
  • Expected outcomes

Students will gain experience conducting research on emerging applications of generative AI in higher education. The project will contribute to understanding how AI-powered chatbots can support both students and teaching staff, and will inform future design and deployment of AI-assisted learning support systems at Monash University.