PERFORMANCE AND ACCURACY OF CHATGPT IN GENERATING MALAY ACADEMIC TEXTS: A COMPARATIVE STUDY WITH EXPERT CORRECTIONS

Lalu Nurul Yaqin; Hasmidar Hassan; Badriyah Yusof

doi:10.24071/llt.v28i1.11698

PERFORMANCE AND ACCURACY OF CHATGPT IN GENERATING MALAY ACADEMIC TEXTS: A COMPARATIVE STUDY WITH EXPERT CORRECTIONS

Lalu Nurul Yaqin^(1*), Hasmidar Hassan⁽²⁾, Badriyah Yusof⁽³⁾,

(1) Universiti Brunei Darussalam, Brunei Darussalam
(2) Universiti Brunei Darussalam, Brunei Darussalam
(3) Universiti Brunei Darussalam, Brunei Darussalam
(*) Corresponding Author

Abstract

The increasing use of artificial intelligence in academic writing has raised concerns about the accuracy and coherence of AI-generated texts, particularly in underrepresented languages like Malay. This study evaluates the performance of ChatGPT in generating Malay academic texts by comparing AI-generated outputs with expert corrected versions, focusing on grammatical errors, structural inconsistencies, and lexical inaccuracies. A comparative analysis was conducted on two datasets: Trained Dataset (TD), where prompts included detailed context, and Untrained Dataset (UTD). ChatGPT-generated texts were reviewed by Malay linguistics and translation experts, who identified and corrected grammatical errors. A quantitative and qualitative analysis assessed error frequency and categorized linguistic challenges. Findings reveal that UTD contained significantly more grammatical errors (87 errors) than TD (18 errors), demonstrating the role of structured prompts in enhancing text quality. Common errors in UTD included incorrect sentence structure (27.59%), omission of names (14.94%), and inappropriate word choices (11.49%). While TD showed improved grammatical accuracy, errors in phrase structure, conjunction usage, and affixation persisted. The study concludes that AI-generated Malay texts lack syntactic stability, requiring expert intervention and model refinement. These findings highlight the need for linguistic adaptation, expanded training datasets and the integration of expert to enhance AI-generated Malay academic writing. Ultimately, this study presents a case study that provides empirical evidence that context-aware prompt engineering and expert-in-the-loop approaches are essential for enhancing the quality of AI outputs, especially in non-English settings. It also advocates for the development of AI models that can capture nuances and linguistic diversity, vital for inclusive education for all.

Keywords

AI-generated text, ChatGPT, expert correction, linguistic accuracy Malay academic writing

Full Text:

PDF

References

Adıgüzel, T., Kaya, M. H., & Cansu, F. K. (2023). Revolutionizing education with AI: Exploring the transformative potential of ChatGPT. Contemporary Educational Technology, 15(3), ep429. https://doi.org/10.30935/cedtech/13152

Daungsupawong, H., & Wiwanitkit, V. (2023). Comment on the use of artificial intelligence in writing scientific papers. Brain Communications, 6(1), fcad354. https://doi.org/10.1093/braincomms/fcad354

Dergaa, I., Chamari, K., Żmijewski, P., & Saad, H. B. (2023). From expert writing to artificial intelligence generated text: Examining the prospects and potential threats of ChatGPT in academic writing. Biology of Sport, 40(2), 615–622. https://doi.org/10.5114/biolsport.2023.125623

Dubey, S. (2024). Redefining cognitive domains in the era of ChatGPT: A comprehensive analysis of artificial intelligence’s influence and future implications. Medical Research Archives, 12(6), 1-7. https://doi.org/10.18103/mra.v12i6.5383

Ebrahimian, M., Behnam, B., Ghayebi, N., & Sobhrakhshankhah, E. (2023). ChatGPT in Iranian medical licensing examination: Evaluating the diagnostic accuracy and decision-making capabilities of an AI-based model. BMJ Health & Care Informatics, 30(1), 1-6. http://dx.doi.org/10.1136/bmjhci-2023-100815

Gao, Y., Wang, R., & Hou, F. (2024). How to design translation prompts for ChatGPT: An empirical study. Proceedings of the 6th ACM International Conference on Multimedia in Asia Workshops, 1–7. http://dx.doi.org/10.1145/3700410.3702123

Glahn, K. (2024). Using ChatGPT to teach English for academic purposes. Jeicom, 5(2), 63–75. https://doi.org/10.34097/jeicom-5-2-december-23-5

Hirobumi, S. R. (2011). Kata pemeri: Satu takrifan nahuan dalam kerangka ayat bahasa Melayu. Jurnal Bahasa, 11(2), 163-195.

Hu, X., Liu, A., & Dai, Y. (2024). Combining ChatGPT and knowledge graph for explainable machine learning-driven design: A case study. Journal of Engineering Design, 1–23. https://doi.org/10.1080/09544828.2024.2355758

Kostka, I., & Toncelli, R. (2023). Exploring applications of ChatGPT to English language teaching: Opportunities, challenges, and recommendations. Teaching English as a Second or Foreign Language—TESL-EJ, 27(3). https://doi.org/10.55593/ej.27107int

Lechien, J. R., Gorton, A., Robertson, J., & Vaira, L. A. (2024). Is ChatGPT‐4 accurate in proofreading a manuscript in otolaryngology–head and neck surgery? Otolaryngology–Head and Neck Surgery, 170(6), 1527–1530. https://doi.org/10.1002/ohn.526

Li, J. (2024). Exploring the potential of artificial intelligence to enhance the writing of English academic papers by non-native English-speaking medical students - The educational application of ChatGPT. BMC Medical Education, 24(1), 736. https://doi.org/10.1186/s12909-024-05738-y

Li, R. (2020). Using artificial intelligence in learning English as a foreign language: An examination of IELTS LIULISHUO as an online platform. Journal of Higher Education Research, 1(2), 85-89. https://doi.org/10.32629/jher.v1i2.178

Li, X. (2023). Empowering Chinese language learners from low-income families to improve their Chinese writing with ChatGPT’s assistance afterschool. Languages, 8(4), 238. https://doi.org/10.3390/languages8040238

Mills, M., Collier, D., Mahon, J., Ebbinghaus, B., Eggan, F., Gravlee, C., & Maxwell, J. (2008). Comparative research. In The SAGE encyclopedia of qualitative research methods (Vol. 0, pp. -). SAGE Publications, Inc., https://doi.org/10.4135/9781412963909.n55

Nhan, L. (2024). Vietnamese university students’ perceptions in learning English using ChatGPT. International Journal of Science and Management Studies (IJSMS), 7(1), 142–148. https://doi.org/10.51386/25815946/ijsms-v7i1p121

Niloy, A. C., Akter, S., Sultana, N., Sultana, J., & Rahman, S. I. U. (2024). Is ChatGPT a menace for creative writing ability? An experiment. Journal of Computer Assisted Learning, 40(2), 919–930. https://doi.org/10.1111/jcal.12929

Nomoto, H. (2023). Issues surrounding the use of ChatGPT in similar languages: The case of Malay and Indonesian. Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics, 76-82. Bali: Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.ijcnlp-short.9

Nomoto, H., Moeljadi, D., & Razak, F. A. A. (2024). Masalah teknologi dan isu sosial berkaitan penggunaan ChatGPT dalam Bahasa Melayu: Technological and social issues related to using ChatGPT in Malay. RENTAS: Jurnal Bahasa, Sastera Dan Budaya, 3(1), 1–22. https://doi.org/10.32890/rentas2024.3.1

Omar, A. H. (1993). Nahu Melayu mutakhir (4th ed.). Kuala Lumpur: Dewan Bahasa dan Pustaka.

Rojas, A. J. (2024). An investigation into ChatGPT’s application for a scientific writing assignment. Journal of Chemical Education, 101(5), 1959–1965. https://doi.org/10.1021/acs.jchemed.4c00034

Ruksakulpiwat, S. (2024). Assessing the efficacy of ChatGPT versus expert researchers in identifying relevant studies on mHealth interventions for improving medication adherence in patients with ischemic stroke when conducting systematic reviews: Comparative analysis. JMIR MHealth and UHealth, 12, e51526. https://doi.org/10.2196/51526

Safiah, K. N., Farid, M. O., Hashim, M., & Hamid, M. A. (2010). Tatabahasa dewan (3rd ed.). Kuala Lumpur: Dewan Bahasa dan Pustaka.

Safrai, M., & Orwig, K. E. (2024). Utilizing artificial intelligence in academic writing: An in-depth evaluation of a scientific review on fertility preservation written by ChatGPT-4. Journal of Assisted Reproduction and Genetics, 41(7), 1871-1880. https://doi.org/10.1007/s10815-024-03089-7

Seong, Y. P. (2014). Strategic competence and L2 speaking assessment. Teachers College, Columbia University Working Papers in TESOL & Applied Linguistics, 14(1), 13-24

Singh, M. K. M. (2015). International graduate students' academic writing practices in Malaysia: Challenges and solutions. Journal of International Students, 5(1), 12-22. https://doi.org/10.32674/jis.v5i1.439

Singh, M. K. M., & Kaur, M. (2016). An emic perspective on academic writing difficulties among international graduate students in Malaysia. GEMA Online® Journal of Language Studies, 16(3), 83-97. http://dx.doi.org/10.17576/gema-2016-1603-06

Sr, H. İ. S. (2024). Evaluation of ChatGPT’s responses to Google searches about dry eye (preprint). https://doi.org/10.2196/preprints.60357

Syafiee, A. S. E. E. A., & Yaqin, L. N. (2025). A comparative analysis of Google Translate and ChatGPT in translating Borneo Bulletin News into standard Malay. Journal of Research on English and Language Learning (J-REaLL), 6(1), 115-126. https://doi.org/10.33474/j-reall.v6i1.23252

Xu, X., Chen, Y., & Miao, J. (2024). Opportunities, challenges, and future directions of large language models, including ChatGPT in medical education: A systematic scoping review. Journal of Educational Evaluation for Health Professions, 21, 6. https://doi.org/10.3352/jeehp.2024.21.6

Yaqin, L. N., Yusof, B., Yusof, N., & Damit, A. R. (2025). Students' perception of using ChatGPT as an AI-integrated tool in the Malay Language. Jurnal Penelitian dan Pengkajian Ilmu Pendidikan: E-Saintika, 9(1), 13-31. https://doi.org/10.36312/e-saintika.v9i1.2584

DOI: https://doi.org/10.24071/llt.v28i1.11698