Convolutional Implementation Neural Network (CNN) on the System and Introduction Sign Language to Support Communication with People with Disabilities

Abstract

The results of this study are based on verifying communication solutions for deaf and hard of hearing participants in Indonesia, the majority of whom use the Indonesian Sign Language System (SIBI). The goal is to design an accurate and efficient system for detecting and recognizing sign language using Convolutional Neural Network (CNN) and accessible in real-time on mobile devices to support more inclusive communication. This study uses a quantitative approach and uses deep learning techniques by utilizing a SIBI image dataset totaling 5,280 images (24 classes without the letters J and Z) with a division of 80% for training data and 20% for validation data. This research process starts from image preprocessing, designing a CNN architecture consisting of convolution, pooling, and fully connected layers, implementing it systematically with Python–TensorFlow, and integrating it into the Flutter application through APIs. Evaluation was conducted using Intersection over Union (IoU) metrics and measured algorithm performance with precision, recall, and mean Average Precision (mAP). The model is capable of achieving a precision of 0.97, a recall of 1.00 and a high mAP in a wide range of backgrounds and lighting.