The ASL Machine is a machine learning program taught to recognize the American Sign Language alphabet. Most ASL recognition programs use raw pixel input, meaning they compare the pixels that make up the users' hands to their known data set of images in order to make a classification decision. The ASL Machine, however, uses palm detection and finger skeleton tracking to first recognize the pose of the hand (ml5.js Handpose). The program then feeds those skeleton coordinates into another machine learning algorithm to classify them as a letter of the ASL alphabet! This means that as long as the Handpose model can see the right hand of the user, the pose can be classified. Since the Handpose model has been extensively trained with a variety of skin tones, angles, and lighting conditions, the ASL Machine has the potential to be more accurate than pixel based classification programs. At the time of writing, however, the data set is still small and the program still has trouble with several letters (J, M, N, and Z).
Use your right hand to form the letters of the ASL alphabet! As mentioned several letters are still being trained (J, M, N, and Z), and H is currently flipped. Try memorizing a few letters at a time!
I hope to make the training data public very soon! This project is intended to be entirely open source, which should include an open data set. Right now, the data set is incredibly limited, using only training data from 4 people. In the future, I also hope to be able to accept training submitted data from users.
Have an idea on how to make this better? Want to collaborate? Want to just say hi? DM me @jayborgwardt or email me at jayborgwardt@gmail.com
Thanks to Claire Wong for help with visual design, Daniel Shiffman for his amazing coding tutorials, and Qianqian Ye for her guidance!