AI Model Recognizes American Sign Language Hand Gestures

Florida Atlantic University researchers combined MediaPipe hand tracking and a YOLOv8 model to recognize static American Sign Language (ASL) alphabet gestures. In the reported study, accuracy and recall reached 98%, the F1 score reached 99%, mean average precision reached 98%, and mAP50–95 reached 93%. These are results for the tested gestures, not a complete translation of sign language.

Sign languages communicate through hand gestures, facial expression and body movement. ASL has its own grammar and syntax, and sign languages differ around the world in grammar, syntax and vocabulary. A system that recognizes hand gestures alone does not capture all of that linguistic information. Researchers are nevertheless exploring real-time gesture recognition as one possible aid to more accessible communication.

Using technology to recognize signs

Researchers at Florida Atlantic University's College of Engineering and Computer Science focused on computer-vision recognition of ASL alphabet gestures. They created a custom data set of 29,820 still images. MediaPipe annotated each image with 21 hand landmarks, providing spatial information on hand structure and position. The goal was to detect small differences in hand shape and movement more reliably.

Combining landmark tracking and deep learning

The researchers used those annotations to train a YOLOv8 object-detection model. Their work was published in Franklin Open. First author Bader Alsharif, a doctoral student in FAU's Department of Electrical Engineering and Computer Science, described the combination of MediaPipe, YOLOv8 and hyperparameter tuning as a promising approach that had not been explored in the earlier work he cited.

The team reported 98% accuracy, 98% recall, a 99% F1 score, 98% mean average precision and 93% mAP50–95. Alsharif said the results showed low error rates in detecting and classifying the tested gestures and suggested potential for intuitive real-time interaction. The figures concern the tested gestures; they do not demonstrate performance in unrestricted conversations.

Understanding hand position and gesture

Integrating MediaPipe's landmarks into YOLOv8 training improved the model's reported bounding-box and gesture-classification performance. The two-step approach combined landmark tracking with object detection so the model could capture subtle variations in hand pose. The researchers said recognition remained strong across different hand positions and gestures.

Co-author Mohammad Ilyas, a professor in FAU's Department of Electrical Engineering and Computer Science, credited transfer learning, careful data-set construction and hyperparameter tuning for the results. He described real-time gesture recognition as a potential basis for assistive technology, while future work will expand the data set to more hand shapes and gestures, improve discrimination between visually similar signs, and optimize the model for resource-limited edge devices.

Stella Batalama, dean of FAU's College of Engineering and Computer Science, said the work could contribute to tools that support communication by deaf and hard-of-hearing people in education, health care and social settings. Those uses remain potential applications of the research, not demonstrated full ASL translation.

Original report: The Hearing Review.