Sign Language Detection Models using Resnet-34 and Augmentation Techniques

Rizki Ramdhan Hilal(1), Aradea Aradea(2), Vega Purwayoga(3*),

(1) Universitas Siliwangi
(2) Universitas Siliwangi
(3) Universitas Siliwangi
(*) Corresponding Author

Abstract


For deaf or hard of hearing people, sign language is a primary means of communication, but low public understanding makes social engagement difficult. Researchers now use computer vision technology and Convolutional Neural Network (CNN) to detect sign language movements. Problems such as overfitting and missing gradients still exist. Using CNN and ResNet-34 architecture, as well as image augmentation to overcome this problem, this research builds a deep learning-based sign language detection model. The Indonesian Sign Language System (SIBI) dataset was used to test the model. The test results show that the model with image augmentation trained for more than 50 epochs obtained an accuracy of 99.4%, precision of 99.5%, recall of 99.5%, and an F1 score of 99.5%. The model without image augmentation produced an accuracy of 99.4%, recall of 99.3%, F1 score of 99.3%, and precision of 99.4%. ResNet-34 architecture overcomes the problem of missing gradients, while image augmentation avoids overfitting and improves model accuracy.

Full Text:

PDF

References


[1] Saleha and M. R. Yuwita, “Saleha & MR Yuwita ANALISIS SEMIOTIKA CHARLES SANDERS PEIRCE PADA SIMBOL RAMBU LALU LINTAS DEAD END.”

[2] F. S. Pandiangan and M. Rosadi, “Analisis Dialek Dalam Bentuk Bahasa Percakapan Dalam Film ‘Imperfect’ Karya Meira Anastasia,” Journal of Educational Research and Humaniora (JERH), vol. 1, no. September, pp. 47–58, 2023.

[3] Nasha Hikmatia A.E. and M. I. Zul, “Aplikasi Penerjemah Bahasa Isyarat Indonesia menjadi Suara berbasis Android menggunakan Tensorflow,” Jurnal Komputer Terapan, vol. 7, no. 1, pp. 74–83, 2021, doi: 10.35143/jkt.v7i1.4629.

[4] I. Sari, Fivrenodi, E. Altiarika, and Sarwindah, “Sistem Pengembangan Bahasa Isyarat Untuk Berkomunikasi dengan Penyandang Disabilitas (Tunarungu),” Journal of Information Technology and society, vol. 1, no. 1, pp. 20–25, 2023, doi: 10.35438/jits.v1i1.21.

[5] Nofal Anam, “Sistem Deteksi Simbol Pada Sibi (Sistem Isyarat Bahasa Indonesia) Menggunakan Mediapipe Dan ResNet-50,” 2022.

[6] L. Arisandi and B. Satya, “Sistem Klarifikasi Bahasa Isyarat Indonesia (Bisindo) Dengan Menggunakan Algoritma Convolutional Neural Network,” Jurnal Sistem Cerdas, vol. 5, no. 3, pp. 135–146, 2022, doi: 10.37396/jsc.v5i3.262.

[7] G. Latif, D. A. Alghmgham, R. Maheswar, J. Alghazo, F. Sibai, and M. H. Aly, “Deep learning in Transportation: Optimized driven deep residual networks for Arabic traffic sign recognition,” Alexandria Engineering Journal, vol. 80, no. July 2022, pp. 134–143, 2023, doi: 10.1016/j.aej.2023.08.047.

[8] S. Mekruksavanich, N. Hnoohom, and A. Jitpattanakul, “A Hybrid Deep Residual Network for Efficient Transitional Activity Recognition Based on Wearable Sensors,” Applied Sciences (Switzerland), vol. 12, no. 10, 2022, doi: 10.3390/app12104988.

[9] P. A. Pattanaik, M. Mittal, M. Z. Khan, and S. N. Panda, “Malaria detection using deep residual networks with mobile microscopy,” Journal of King Saud University - Computer and Information Sciences, vol. 34, no. 5, pp. 1700–1705, 2022, doi: 10.1016/j.jksuci.2020.07.003.

[10] B. Tasci, M. R. Acharya, M. Baygin, S. Dogan, T. Tuncer, and S. B. Belhaouari, “InCR: Inception and concatenation residual block-based deep learning network for damaged building detection using remote sensing images,” International Journal of Applied Earth Observation and Geoinformation, vol. 123, no. August, p. 103483, 2023, doi: 10.1016/j.jag.2023.103483.

[11] Z. Niswati, R. Hardatin, M. N. Muslimah, and S. N. Hasanah, “Perbandingan Arsitektur ResNet50 dan ResNet101 dalam Klasifikasi Kanker Serviks pada Citra Pap Smear,” Faktor Exacta, vol. 14, no. 3, p. 160, 2021, doi: 10.30998/faktorexacta.v14i3.10010.

[12] H. Imaduddin, F. Y. A’la, A. Fatmawati, and B. A. Hermansyah, “Comparison of transfer learning method for COVID-19 detection using convolution neural network,” Bulletin of Electrical Engineering and Informatics, vol. 11, no. 2, pp. 1091–1099, Apr. 2022, doi: 10.11591/eei.v11i2.3525.

[13] X. Ma, W. Chen, and Y. Xu, “ERCP-Net: a channel extension residual structure and adaptive channel attention mechanism for plant leaf disease classification network,” Sci Rep, vol. 14, no. 1, pp. 1–14, 2024, doi: 10.1038/s41598-024-54287-3.

[14] Z. Bin Niu, S. Y. Jia, and H. H. Xu, “Automated graptolite identification at high taxonomic resolution using residual networks,” iScience, vol. 27, no. 1, p. 108549, 2024, doi: 10.1016/j.isci.2023.108549.

[15] D. Sarwinda, R. H. Paradisa, A. Bustamam, and P. Anggia, “Deep Learning in Image Classification using Residual Network (ResNet) Variants for Detection of Colorectal Cancer,” Procedia Comput Sci, vol. 179, no. 2019, pp. 423–431, 2021, doi: 10.1016/j.procs.2021.01.025.

[16] L. Ali and S. A. C. Bukhari, “An Approach Based on Mutually Informed Neural Networks to Optimize the Generalization Capabilities of Decision Support Systems Developed for Heart Failure Prediction,” IRBM, vol. 42, no. 5, pp. 345–352, 2021, doi: https://doi.org/10.1016/j.irbm.2020.04.003.

[17] M. A. A. Fawwaz, K. N. Ramadhani, and F. Sthevani, “Klasifikasi Ras pada hewan peliharaan menggunakan Algoritma Convolutional Neural Network (CNN),” vol. 8, no. 1, pp. 715–730, 2020.

[18] Rima Dias Ramadhani, A. Nur Aziz Thohari, C. Kartiko, A. Junaidi, T. Ginanjar Laksana, and N. Alim Setya Nugraha, “Optimasi Akurasi Metode Convolutional Neural Network untuk Identifikasi Jenis Sampah,” Jurnal RESTI (Rekayasa Sistem dan Teknologi Informasi), vol. 5, no. 2, pp. 312–318, Apr. 2021, doi: 10.29207/resti.v5i2.2754.

[19] J. Sanjaya and M. Ayub, “Augmentasi Data Pengenalan Citra Mobil Menggunakan Pendekatan Random Crop , Rotate , dan Mixup,” vol. 6, pp. 311–323, 2020.

[20] Y. Vita Via, I. Yuniar Purbasari, and A. Putra Pratama, “Analisa Algoritma Convolution Neural Network (Cnn) Pada Klasifikasi Genre Musik Berdasar Durasi Waktu,” SCAN Jurnal Teknologi dan Informasi, vol. 17, no. 1, pp. 35–41, 2022, [Online]. Available: http://ejournal.upnjatim.ac.id/index.php/scan/article/view/3251/2003

[21] R. Z. Fadillah, A. Irawan, M. Susanty, and I. Artikel, “Data Augmentasi Untuk Mengatasi Keterbatasan Data Pada Model Penerjemah Bahasa Isyarat Indonesia (BISINDO),” Jurnal Informatika, vol. 8, no. 2, pp. 208–214, 2021, [Online]. Available: https://ejournal.bsi.ac.id/ejurnal/index.php/ji/article/view/10768

[22] N. E. Khalifa, M. Loey, and S. Mirjalili, “A comprehensive survey of recent trends in deep learning for digital images augmentation,” Artif Intell Rev, vol. 55, no. 3, pp. 2351–2377, 2022, doi: 10.1007/s10462-021-10066-4.

[23] W. M. Pradnya D and A. P. Kusumaningtyas, “Analisis Pengaruh Data Augmentasi Pada Klasifikasi Bumbu Dapur Menggunakan Convolutional Neural Network,” Jurnal Media Informatika Budidarma, vol. 6, no. 4, p. 2022, 2022, doi: 10.30865/mib.v6i4.4201.

[24] D. Putri Ayuni, Jasril, M. Irsyad, F. Yanto, and S. Sanjaya, “Augmentasi Data Pada Implementasi Convolutional Neural Network Arsitektur Efficientnet-B3 Untuk Klasifikasi Penyakit Daun Padi,” ZONAsi: Jurnal Sistem Informasi, vol. 5, no. 2, pp. 239–249, 2023, doi: 10.31849/zn.v5i2.13874.

[25] T. B. Sasongko, H. Haryoko, and A. Amrullah, “Analisis Efek Augmentasi Dataset dan Fine Tune pada Algoritma Pre-Trained Convolutional Neural Network (CNN),” Jurnal Teknologi Informasi dan Ilmu Komputer, vol. 10, no. 4, pp. 763–768, 2023, doi: 10.25126/jtiik.20241046583.

[26] Hendri Candra Mayana and Desmarita Leni, “Deteksi Kerusakan Ban Mobil Menggunakan Convolutional Neural Network dengan Arsitektur ResNet-34,” Jurnal Surya Teknika, vol. 10, no. 2, pp. 842–851, 2023, doi: 10.37859/jst.v10i2.6336.

[27] N. I. Sanusi, S. Ramadhani, and M. Irsyad, “Analisa Gambar X-Ray Mammography dengan Convolution Neural Network pada Deep Learning dengan Arsitektur Resnet,” Jurnal Sistem Komputer dan Informatika (JSON), vol. 4, no. 4, p. 604, 2023, doi: 10.30865/json.v4i4.6365.

[28] M. F. Gunardi, “Implementasi Augmentasi Citra pada Suatu Dataset,” Jurnal Informatika, vol. 9, no. 1, pp. 1–5, 2023.

[29] W. Maulana Baihaqi, C. Raras, A. Widiawati, D. P. Sabila, and A. Wati, “Analisis Gambar Sel Darah Berbasis Convolution Neural Network untuk Mendiagnosis Penyakit Demam Berdarah Convolution Neural Network-Based Image Analysis of Blood Cells to Diagnose Dengue Fever,” Cogito Smart Journal |, vol. 7, no. 1, 2021.

[30] R. Cendekia Vandara, S. A. Wibowo, and K. Usman, “PERFORMANCE ANALYSIS OF FACE ALIGNMENT ON 3-DIMENSIONAL (3D) FACE RECONSTRUCTION USING MODIFIED POSITION MAP REGRESSION NETWORK).”

[31] A. Ridhovan and A. Suharso, “Penerapan Metode Residual Network (Resnet) Dalam Klasifikasi Penyakit Pada Daun Gandum,” JIPI (Jurnal Ilmiah Penelitian dan Pembelajaran Informatika), vol. 7, no. 1, pp. 58–65, 2022, doi: 10.29100/jipi.v7i1.2410.

[32] I. Ariawan et al., “Extraction of Morphometric Features the shape of mangrove leaves based on digital images and classification using the Support Vector Machine,” Karbala International Journal of Modern Science, vol. 10, no. 2, May 2024, doi: 10.33640/2405-609X.3349.

[33] L. N. Smith, “A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay,” Mar. 2018, [Online]. Available: http://arxiv.org/abs/1803.09820

[34] J. Xu, Y. Zhang, and D. Miao, “Three-way confusion matrix for classification: A measure driven view,” Inf Sci (N Y), vol. 507, pp. 772–794, 2020, doi: https://doi.org/10.1016/j.ins.2019.06.064.




DOI: https://doi.org/10.24071/ijasst.v7i2.12888

Refbacks

  • There are currently no refbacks.









Publisher : Faculty of Science and Technology

Society/Institution : Sanata Dharma University

 

 

 

Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 International License.