Enhancing the Performance of Machine Learning Algorithm for Intent Sentiment Analysis on Village Fund Topic

Abstract

This study explores the implementation of Intent Sentiment Analysis on Twitter data related to the Village Fund program, leveraging Multinomial Naïve Bayes (MNB) and enhancing it with Synthetic Minority Over-sampling Technique (SMOTE) and XGBoost (XGB). The analysis categorizes tweets into six labels: Optimistic, Pessimistic, Advice, Satire, Appreciation, and No Intent. Initially, the MNB model achieved an accuracy of 67% on a 90:10 data split. By applying SMOTE, accuracy improved by 12%, reaching 89%. However, adding Chi-Square feature selection did not increase accuracy further. Incorporating XGB into the MNB+SMOTE model led to a 6% improvement, achieving a final accuracy of 95%. Comprehensive model evaluation revealed that the MNB+SMOTE+XGB model achieved 96% accuracy, 96% precision, 96% recall, and a 96% F1-score, with an AUC of 99%, categorizing it as excellent. These findings demonstrate that the combination of SMOTE for addressing class imbalance and XGBoost for boosting performance significantly enhances the MNB model's classification capabilities. The novelty lies in the integration of these techniques to improve intent sentiment classification for public opinion analysis on the Village Fund program. The results indicate that the majority of tweets labeled as "No Intent" reflect a lack of specific sentiment or actionable intent, providing valuable insights into public perception of the program.