Evaluasi Sentimen Pengguna ChatGPT Menggunakan Naive Bayes: Tinjauan dari Confusion Matrix dan Classification Report

Dianda Rifaldi; Tri Stiyo Famuji; Bella Okta Sari Miranda; Fauzan Purma Ramadhan; Iriene Putri Mulyadi; Vanji Saputra6; Galih Pramuja Inngam Fanani

doi:10.30787/restia.v3i2.1990

Authors

Dianda Rifaldi Universitas Riau Indonesia
Tri Stiyo Famuji Universitas Al–Irsyad Cilacap
Bella Okta Sari Miranda Universitas Ahmad Dahlan
Fauzan Purma Ramadhan Universitas Riau Indonesia
Iriene Putri Mulyadi Universitas Riau Indonesia
Vanji Saputra6 Universitas Riau Indonesia
Galih Pramuja Inngam Fanani Universitas 'Aisyah Surakarta

DOI:

https://doi.org/10.30787/restia.v3i2.1990

Keywords:

AI, ChatGPT, Kaggle, NLP, Naive Bayes

Abstract

The development of artificial intelligence (AI) technology, particularly in natural language processing (NLP), has led to various innovations, including ChatGPT. Its growing popularity highlights the need for user sentiment analysis. This study evaluates user sentiment toward ChatGPT using the Naive Bayes algorithm. The dataset, obtained from Kaggle, consists of 500 labeled English tweets categorized as positive, neutral, or negative. The process involved text preprocessing, TF-IDF feature extraction, data splitting (80% training, 20% testing), and model training. The results show an accuracy of 56%, with the highest f1-score in the negative class (0.67) and the lowest in the neutral class (0.38). The model exhibits classification imbalance, with high precision but low recall in the neutral class, and high recall but low precision in the positive class. The confusion matrix further confirms frequent misclassifications between classes. These findings reflect the limitations of Naive Bayes in handling contextual relationships in text data. Improvements can be achieved through data balancing, enhanced NLP-based feature representation, and the application of more complex classification algorithms.

References

S. Aggrawal dan A. J. Magana, “Teamwork Conflict Management Training and Conflict Resolution Practice via Large Language Models,” Futur. Internet, vol. 16, no. 5, hal. 1–25, 2024, doi: 10.3390/fi16050177.

M. Nanda Fahriza dan N. Riza, “Analisis Sentimen Pada Ulasan Aplikasi Chat Generative Pre-Trained Transformer Gpt Menggunakan Metode Klasifikasi K-Nearest Neighbor(Knn),” JATI (Jurnal Mhs. Tek. Inform., vol. 7, no. 2, hal. 1351–1358, 2023, doi: 10.36040/jati.v7i2.6767.

M. S. Adhi, M. Z. Nafan, dan E. Usada, “Pengaruh Semantic Expansion pada Naïve Bayes Classifier untuk Analisis Sentimen Tokoh Masyarakat,” J. RESTI (Rekayasa Sist. dan Teknol. Informasi), vol. 3, no. 2, hal. 141–147, 2019, doi: 10.29207/resti.v3i2.901.

Merinda Lestandy, Abdurrahim Abdurrahim, dan Lailis Syafa’ah, “Analisis Sentimen Tweet Vaksin COVID-19 Menggunakan Recurrent Neural Network dan Naïve Bayes,” J. RESTI (Rekayasa Sist. dan Teknol. Informasi), vol. 5, hal. 802–808, 2021, doi: 10.29207/resti.v5i4.3308.

K. Aulia dan L. Amelia, “Analisis Sentimen Twitter Pada Isu Mental Health Dengan Algoritma Klasifikasi Naive Bayes,” Siliwangi J. (Seri Sains Teknol., vol. 6, no. 2, hal. 60–65, 2020.

S. O. S. Dinauni, Herlawati, dan Rasim, “Analisis Sentimen Mengenai Gangguan Bipolar Pada Twitter Menggunakan Algoritma Naïve Bayes,” vol. 6, no. 2, hal. 63–73, 2022.

K. Li, C. Zhou, X. (Robert) Luo, J. Benitez, dan Q. Liao, “Impact of information timeliness and richness on public engagement on social media during COVID-19 pandemic: An empirical investigation based on NLP and machine learning,” Decis. Support Syst., vol. 162, no. February, hal. 113752, 2022, doi: 10.1016/j.dss.2022.113752.

J. Frei, L. Frei-Stuber, dan F. Kramer, “GERNERMED++: Semantic annotation in German medical NLP through transfer-learning, translation and word alignment,” J. Biomed. Inform., vol. 147, hal. 104513, 2023, doi: 10.1016/j.jbi.2023.104513.

A. F. Rahma, A. R. W. Wibowo, N. N. Nidya, dan Agussalim, “Analisis Sentimen Hashtag ‘Dirumahaja’ Saat Pandemi Covid-19 Di Indonesia Menggunakan Nlp,” J. Inform. dan Sist. Inf., vol. 1, hal. 343–353, 2020, [Daring]. Tersedia pada: http://jifosi.upnjatim.ac.id/index.php/jifosi/article/view/239

N. L. P. M. Putu, Ahmad Zuli Amrullah, dan Ismarmiaty, “Analisis Sentimen dan Pemodelan Topik Pariwisata Lombok Menggunakan Algoritma Naive Bayes dan Latent Dirichlet Allocation,” J. RESTI (Rekayasa Sist. dan Teknol. Informasi), vol. 5, no. 1, hal. 123–131, 2021, doi: 10.29207/resti.v5i1.2587.

A. I. Kadhim, Y. N. Cheah, dan N. H. Ahamed, “Text Document Preprocessing and Dimension Reduction Techniques for Text Document Clustering,” Proc. - 2014 4th Int. Conf. Artif. Intell. with Appl. Eng. Technol. ICAIET 2014, hal. 69–73, 2015, doi: 10.1109/ICAIET.2014.21.

H. T. Duong dan T. A. Nguyen-Thi, “A review: preprocessing techniques and data augmentation for sentiment analysis,” Comput. Soc. Networks, vol. 8, no. 1, hal. 1–16, 2021, doi: 10.1186/s40649-020-00080-x.

M. K. Sohrabi dan F. Hemmatian, “An efficient preprocessing method for supervised sentiment analysis by converting sentences to numerical vectors: a twitter case study,” Multimed. Tools Appl., vol. 78, no. 17, hal. 24863–24882, 2019, doi: 10.1007/s11042-019-7586-4.

D. Rifaldi, A. Fadlil, dan Herman, “Teknik Preprocessing Pada Text Mining Menggunakan Data Tweet Mental Health,” J. Pendidik. Teknol. Inf., vol. 3, no. 2, hal. 161–171, 2023.

R. Ahuja, A. Chug, S. Kohli, S. Gupta, dan P. Ahuja, “ScienceDirect ScienceDirect The Impact of Features Extraction on the Sentiment Analysis,” Procedia Comput. Sci., vol. 152, hal. 341–348, 2019, doi: 10.1016/j.procs.2019.05.008.

D. Sunitha, R. K. Patra, N. V. Babu, A. Suresh, dan S. C. Gupta, “Twitter sentiment analysis using ensemble based deep learning model towards COVID-19 in India and European countries,” Pattern Recognit. Lett., vol. 158, hal. 164–170, 2022, doi: 10.1016/j.patrec.2022.04.027.

D. Rifaldi, A. Fadlil, dan Herman, “Implementation of Word Trends Using a Machine Learning Approach with TF-IDF and Latent Dirichlet Allocation,” Int. J. Informatics Vis., vol. 8, no. 4, hal. 2297–2304, 2024, doi: 10.62527/joiv.8.4.2452.

A. S. Neogi, K. A. Garg, R. K. Mishra, dan Y. K. Dwivedi, “Sentiment analysis and classification of Indian farmers’ protest using twitter data,” Int. J. Inf. Manag. Data Insights, vol. 1, no. 2, hal. 100019, 2021, doi: 10.1016/j.jjimei.2021.100019.