[Malaysia] A Study on Multimodal Sentiment Analysis and Emotion Recognition

In our current day and age, there is an exponential growth in multimodal data, especially the transition of social media from text-based communications to video formats which can be observed with the rise of TikTok, Youtube, and Instagram Reels. This shift requires a shift in how we analyze multimodal data as we will have to move away from traditional text sentiment analysis such as TextCNN. Multimodal data presents us the opportunity to improve on text-based analysis given the new information that is coded in speech and visuals that can provide additional context for a sentiment.

This project has its real-world applications across different industries. Social media is one aspect which multimodal sentiment analysis can be applied to. VADER (Valence Aware Dictionary for sEntiment Reasoning) for sentiment analysis and JAMMIN for emotion analysis successfully detect hate speech on Facebook. Sentiment analysis in product reviews can greatly benefit companies trying to understand their customers sentiment for their products. In an era of generative AI hype, AI customer service models built on generative AI could benefit from understanding sentiment from a multimodal input to provide the appropriate responses.