1. Home
  2. Vol 4, No 2 (2023)
  3. Sayeed

BERT: A Review of Applications in Sentiment Analysis

Md Shohel Sayeed, Varsha Mohan, Kalaiarasi Sonai Muthu

Abstract


E-commerce reviews are becoming more valued by both customers and companies. The high demand for sentiment analysis is driven by businesses relying on it as a crucial tool to improve product quality and make informed decisions in a fiercely competitive business environment. The purpose of this review paper is to explore and evaluate the applications of the BERT model, a Natural Language Processing (NLP) technique, in sentiment analysis across various fields. The model has been utilized in certain studies for various languages, restaurant businesses, agriculture, Automated Essay Scoring (AES), Twitter, and Google Play. The BERT model's fine-tuning steps involve using pre-trained BERT to perform various language understanding tasks. Text pre-processing is conducted to clean up the data and convert it to numbers before feeding it into BERT, which generates vectors for each input token. We found that BERT outperformed the norm on a range of general language understanding tasks, including sentiment analysis, paraphrase recognition, question-answering, and linguistic acceptability. The detection of neutral reviews and the presence of false reviews in the dataset are two problems that have an impact on the model's accuracy. Training is also slow because it is huge and there are many weights to update. Additional research could be conducted to improve the BERT model's accuracy by constructing a false review categorization model and providing more training to the model in recognizing neutral reviews.

 

Doi: 10.28991/HIJ-2023-04-02-015

Full Text: PDF


Keywords


Natural Language Processing; BERT; Fine-Tuning; Machine learning; Sentiment Analysis.

References


Haque, T. U., Saber, N. N., & Shah, F. M. (2018). Sentiment analysis on large scale Amazon product reviews. 2018 IEEE International Conference on Innovative Research and Development (ICIRD). doi:10.1109/icird.2018.8376299.

Fan & Fuel. No online customer reviews mean BIG problems in 2017. Available online: https://fanandfuel.com/no-online-customer-reviews-means-big-problems-2017/ (accessed on April 2023).

Gumelar, A. B., Yogatama, A., Adi, D. P., Frismanda, & Sugiarto, I. (2022). Forward feature selection for toxic speech classification using support vector machine and random forest. IAES International Journal of Artificial Intelligence, 11(2), 717–726. doi:10.11591/ijai.v11.i2.pp717-726.

Noor, F., Bakhtyar, M., & Baber, J. (2019). Sentiment Analysis in E-commerce Using SVM on Roman Urdu Text. Emerging Technologies in Computing. iCETiC 2019, Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, 285, Springer, Cham, Switzerland. doi:10.1007/978-3-030-23943-5_16.

Ying, O. J., Zabidi, M. M. A., Ramli, N., & Sheikh, U. U. (2020). Sentiment analysis of informal Malay tweets with deep learning. IAES International Journal of Artificial Intelligence, 9(2), 212–220. doi:10.11591/ijai.v9.i2.pp212-220.

Sun, C., Qiu, X., Xu, Y., & Huang, X. (2019). How to Fine-Tune BERT for Text Classification?. Chinese Computational Linguistics. CCL 2019. Lecture Notes in Computer Science, 11856, Springer, Cham, Switzerland. doi:10.1007/978-3-030-32381-3_16.

Nguyen, Q. T., Nguyen, T. L., Luong, N. H., & Ngo, Q. H. (2020). Fine-Tuning BERT for Sentiment Analysis of Vietnamese Reviews. 2020 7th NAFOSTED Conference on Information and Computer Science (NICS). doi:10.1109/nics51282.2020.9335899.

Chakkarwar, V., Tamane, S., & Thombre, A. (2023). A Review on BERT and Its Implementation in Various NLP Tasks. Proceedings of the International Conference on Applications of Machine Intelligence and Data Analytics (ICAMIDA 2022), 112–121. doi:10.2991/978-94-6463-136-4_12.

Nugroho, K. S., Sukmadewa, A. Y., Wuswilahaken DW, H., Bachtiar, F. A., & Yudistira, N. (2021). BERT Fine-Tuning for Sentiment Analysis on Indonesian Mobile Apps Reviews. 6th International Conference on Sustainable Information Engineering and Technology 2021. doi:10.1145/3479645.3479679.

Azhar, A. N., & Khodra, M. L. (2020). Fine-tuning Pretrained Multilingual BERT Model for Indonesian Aspect-based Sentiment Analysis. 2020 7th International Conference on Advance Informatics: Concepts, Theory and Applications (ICAICTA). doi:10.1109/icaicta49861.2020.9428882.

Prottasha, N. J., Sami, A. A., Kowsher, M., Murad, S. A., Bairagi, A. K., Masud, M., & Baz, M. (2022). Transfer Learning for Sentiment Analysis Using BERT Based Supervised Fine-Tuning. Sensors, 22(11), 4157. doi:10.3390/s22114157.

Li, M., Chen, L., Zhao, J., & Li, Q. (2021). Sentiment analysis of Chinese stock reviews based on BERT model. Applied Intelligence, 51(7), 5016–5024. doi:10.1007/s10489-020-02101-8.

M.Abdelgwad, M., A Soliman, T. H., I.Taloba, A., & Farghaly, M. F. (2022). Arabic aspect based sentiment analysis using bidirectional GRU based models. Journal of King Saud University - Computer and Information Sciences, 34(9), 6652–6662. doi:10.1016/j.jksuci.2021.08.030.

Khan, L., Amjad, A., Ashraf, N., & Chang, H. T. (2022). Multi-class sentiment analysis of Urdu text using multilingual BERT. Scientific Reports, 12(1). doi:10.1038/s41598-022-09381-9.

Chong, W. K., Ng, H., Yap, T. T. V., Soo, W. K., Goh, V. T., & Cher, D. T. (2022). Objectivity and Subjectivity Classification with BERT for Bahasa Melayu. Proceedings of the International Conference on Computer, Information Technology and Intelligent Computing (CITIC 2022), 246–257. doi:10.2991/978-94-6463-094-7_20.

Martin, L., Muller, B., Ortiz Suárez, P. J., Dupont, Y., Romary, L., de la Clergerie, É., Seddah, D., & Sagot, B. (2020). CamemBERT: a Tasty French Language Model. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. doi:10.18653/v1/2020.acl-main.645.

Chakravarthi, B. R., Jose, N., Suryawanshi, S., Sherly, E., & McCrae, J. P. (2020). A sentiment analysis dataset for code-mixed Malayalam-English. arXiv preprint arXiv:2006.00210. doi:10.48550/arXiv.2006.00210.

Salnikova, S., & Kyrychenko, R. (2021). Sentiment Analysis Based on the BERT Model: Attitudes Towards Politicians Using Media Data. Advances in Social Science, Education and Humanities Research. doi:10.2991/assehr.k.211218.007.

Rani, P., Suryawanshi, S., Goswami, K., Chakravarthi, B. R., Fransen, T., & McCrae, J. P. (2020). A comparative study of different state-of-the-art hate speech detection methods in Hindi-English code-mixed data. Proceedings of the second workshop on trolling, aggression and cyberbullying, May 2020, Marseille, France.

Bhargava, N., & Johari, A. (2023). Enhancing Deep Learning Approach for Tamil English Mixed Text Classification. Proceedings of the International Conference on Applications of Machine Intelligence and Data Analytics (ICAMIDA 2022), 829–837. doi:10.2991/978-94-6463-136-4_73.

Pota, M., Ventura, M., Catelli, R., & Esposito, M. (2021). An effective BERT-based pipeline for twitter sentiment analysis: A case study in Italian. Sensors (Switzerland), 21(1), 1–21. doi:10.3390/s21010133.

Singh, M., Jakhar, A. K., & Pandey, S. (2021). Sentiment analysis on the impact of coronavirus in social life using the BERT model. Social Network Analysis and Mining, 11(1). doi:10.1007/s13278-021-00737-z.

Gao, Z., Feng, A., Song, X., & Wu, X. (2019). Target-dependent sentiment classification with BERT. IEEE Access, 7, 154290–154299. doi:10.1109/ACCESS.2019.2946594.

Nguyen, D. Q., Vu, T., & Tuan Nguyen, A. (2020). BERTweet: A pre-trained language model for English Tweets. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 9-14. doi:10.18653/v1/2020.emnlp-demos.2.

Benitez-Andrades, J. A., Alija-Perez, J. M., Garcia-Rodriguez, I., Benavides, C., Alaiz-Moreton, H., Vargas, R. P., & Garcia-Ordas, M. T. (2021). BERT Model-Based Approach for Detecting Categories of Tweets in the Field of Eating Disorders (ED). 2021 IEEE 34th International Symposium on Computer-Based Medical Systems (CBMS). doi:10.1109/cbms52027.2021.00105.

Zhao, X., & Sun, Y. (2022). Amazon Fine Food Reviews with BERT Model. Procedia Computer Science, 208, 401–406. doi:10.1016/j.procs.2022.10.056.

Beseiso, M., & Alzahrani, S. (2020). An empirical analysis of BERT embedding for automated essay scoring. International Journal of Advanced Computer Science and Applications, 11(10), 204–210. doi:10.14569/IJACSA.2020.0111027.

Kusnadi, R., Yusuf, Y., Andriantony, A., Ardian Yaputra, R., & Caintan, M. (2021). Sentiment Analysis of the Game Genshin Impact Using Bert. Rabit: Jurnal Teknologi Dan Sistem Informasi Univrab, 6(2), 122–129. doi:10.36341/rabit.v6i2.1765. (In Indonesian).

Lee, S. (2022). Sentiment Analysis Using Bert on Yelp Restaurant Reviews. Ph.D. Thesis, Purdue University Graduate School, West Lafayette, United States.

Cao, Y., Sun, Z., Li, L., & Mo, W. (2022). A Study of Sentiment Analysis Algorithms for Agricultural Product Reviews Based on Improved BERT Model. Symmetry, 14(8). doi:10.3390/sym14081604.

Durairaj, A. K., & Chinnalagu, A. (2021). Transformer based Contextual Model for Sentiment Analysis of Customer Reviews: A Fine-tuned BERT. International Journal of Advanced Computer Science and Applications, 12(11), 474-480. doi:10.14569/ijacsa.2021.0121153.

Sousa, M. G., Sakiyama, K., Rodrigues, L. de S., Moraes, P. H., Fernandes, E. R., & Matsubara, E. T. (2019). BERT for Stock Market Sentiment Analysis. 2019 IEEE 31st International Conference on Tools with Artificial Intelligence (ICTAI). doi:10.1109/ictai.2019.00231.

Lee, X. J., Yap, T. T. V., Ng, H., & Goh, V. T. (2022). Comparison of Word Embeddings for Sentiment Classification with Preconceived Subjectivity. Proceedings of the International Conference on Computer, Information Technology and Intelligent Computing (CITIC 2022), 488–502. doi:10.2991/978-94-6463-094-7_39.

Su, Y. (2022). Emotion Analysis of Microblog Epidemic Coexistence Based on BERT. Proceedings of the 2022 2nd International Conference on Modern Educational Technology and Social Sciences (ICMETSS 2022), 275–280. doi:10.2991/978-2-494069-45-9_33.

Abbas, M., Ali Memon, K., & Aleem Jamali, A. (2019). Multinomial Naive Bayes Classification Model for Sentiment Analysis. IJCSNS International Journal of Computer Science and Network Security, 19(3), 62.

Shamrat, F. M. J. M., Chakraborty, S., Imran, M. M., Muna, J. N., Billah, M. M., Das, P., & Rahman, M. O. (2021). Sentiment analysis on twitter tweets about COVID-19 vaccines using NLP and supervised KNN classification algorithm. Indonesian Journal of Electrical Engineering and Computer Science, 23(1), 463–470. doi:10.11591/ijeecs.v23.i1.pp463-470.

Navér, N. (2021). The past, present or future?: A comparative NLP study of Naive Bayes, LSTM and BERT for classifying Swedish sentences based on their tense. Master of Science Programme in Information Technology Engineering, Uppsala Universitet, Uppsala, Sweden.

Geetha, M. P., & Karthika Renuka, D. (2021). Improving the performance of aspect based sentiment analysis using fine-tuned Bert Base Uncased model. International Journal of Intelligent Networks, 2, 64–69. doi:10.1016/j.ijin.2021.06.005.

Hussain, M. A., Chen, Z., Wang, R., Shah, S. U., Shoaib, M., Ali, N., ... & Ma, C. (2022). Landslide Susceptibility Mapping using Machine Learning Algorithm. Civil Engineering Journal, 8(2), 209-224. doi:10.28991/CEJ-2022-08-02-02.

Nurhidayat, I., Pimpunchat, B., Noeiaghdam, S., & Fernández-Gámiz, U. (2022). Comparisons of SVM kernels for insurance data clustering. Emerging Science Journal, 6(4), 866-880. doi:10.28991/ESJ-2022-06-04-014.

Singh, J., & Tripathi, P. (2021). Sentiment analysis of Twitter data by making use of SVM, Random Forest and Decision Tree algorithm. 2021 10th IEEE International Conference on Communication Systems and Network Technologies (CSNT). doi:10.1109/csnt51715.2021.9509679.

Akhtar, M. B. (2022). The use of a convolutional neural network in detecting soldering faults from a printed circuit board assembly. HighTech and Innovation Journal, 3(1), 1-14. doi:10.28991/HIJ-2022-03-01-01.

Kang, H. W., Chye, K. K., Ong, Z. Y., & Tan, C. W. (2021). The Science of Emotion: Malaysian Airlines Sentiment Analysis using BERT Approach. Conference Proceedings: International Conference on Digital Transformation and Applications (ICDXA 2021). doi:10.56453/icdxa.2021.1013.

Huang, X., Zhang, W., Tang, X., Zhang, M., Surbiryala, J., Iosifidis, V., Liu, Z., & Zhang, J. (2021). LSTM Based Sentiment Analysis for Cryptocurrency Prediction. Database Systems for Advanced Applications. DASFAA 2021, Lecture Notes in Computer Science, 12683, Springer, Cham, Switzerland. doi:10.1007/978-3-030-73200-4_47.

Kumar, P., & Singh, A. K. (2022). A comparison between MLR, MARS, SVR and RF techniques: hydrological time-series modeling. Journal of Human, Earth, and Future, 3(1), 90-98. doi:10.28991/HEF-2022-03-01-07.

Minaee, S., Azimi, E., & Abdolrashidi, A. (2019). Deep-sentiment: Sentiment analysis using ensemble of CNN and Bi-LSTM models. arXiv preprint arXiv:1904.04206. doi:10.48550/ARXIV.1904.04206.

Wen, S., Wei, H., Yang, Y., Guo, Z., Zeng, Z., Huang, T., & Chen, Y. (2021). Memristive LSTM Network for Sentiment Analysis. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 51(3), 1794–1804. doi:10.1109/TSMC.2019.2906098.

Zhou, J., Lu, Y., Dai, H. N., Wang, H., & Xiao, H. (2019). Sentiment analysis of Chinese microblog based on stacked bidirectional LSTM. IEEE Access, 7, 38856–38866. doi:10.1109/ACCESS.2019.2905048.

Yang, G., & Guo, X. (2023). Application of Decision Tree Mining Algorithm in Data Model Collection of Hydraulic Engineering Equipment. Proceedings of the 4th International Conference on Big Data Analytics for Cyber-Physical System in Smart City - Volume 1. BDCPS 2022, Lecture Notes on Data Engineering and Communications Technologies, 167, Springer, Singapore. doi:10.1007/978-981-99-0880-6_76.

Mookdarsanit, P., & Mookdarsanit, L. (2021). The covid-19 fake news detection in Thai social texts. Bulletin of Electrical Engineering and Informatics, 10(2), 988–998. doi:10.11591/eei.v10i2.2745.

Phan, H. T., Tran, V. C., Nguyen, N. T., & Hwang, D. (2019). Decision-Making Support Method Based on Sentiment Analysis of Objects and Binary Decision Tree Mining. Advances and Trends in Artificial Intelligence. From Theory to Practice. IEA/AIE 2019. Lecture Notes in Computer Science, 11606, Springer, Cham, Switzerland. doi:10.1007/978-3-030-22999-3_64.

Rathi, M., Malik, A., Varshney, D., Sharma, R., & Mendiratta, S. (2018). Sentiment Analysis of Tweets Using Machine Learning Approach. 2018 Eleventh International Conference on Contemporary Computing (IC3). doi:10.1109/ic3.2018.8530517.

Hutto, C., & Gilbert, E. (2014). VADER: A Parsimonious Rule-Based Model for Sentiment Analysis of Social Media Text. Proceedings of the International AAAI Conference on Web and Social Media, 8(1), 216–225. doi:10.1609/icwsm.v8i1.14550.

Bajaj, A. Can Python understand human feelings through words? – A brief intro to NLP and VADER Sentiment Analysis. Analytics Vidhya. Available online: https://www.analyticsvidhya.com/blog/2021/06/vader-for-sentiment-analysis/ (accessed on May 2023).

Holmer, D. (2020). Context matters: Classifying Swedish texts using BERT's deep bidirectional word embeddings. Department of Computer and Information Science, Linköping University, Linköping, Sweden.

Wu, K. (2023). BERT Transformers: How Do They Work?. Available online: https://dzone.com/articles/bert-transformers-how-do-they-work (accessed on May 2023).


Full Text: PDF

DOI: 10.28991/HIJ-2023-04-02-015

Refbacks

  • There are currently no refbacks.


Copyright (c) 2023 Md Shohel Sayeed, Varsha Mohan, Kalaiarasi Sonai Muthu