Speech to ISL Converter

This was our final year project in the undergraduate term. Me and my team wanted to build a product that could have a high social impact. So we came up with the idea of building a Speech to Indian Sign Language translation system. This project helped us reach the finals of DJ ASCII, a state-level project competition.

First we created our own database of ISL gestures. We used Microsoft Kinect and Unity to make video motion files of these gestures. We then modeled these VMD files onto a 3D animated character to be a medium for the gesture outputs. The text output from speech-to-text module was then processed to be converted into the proper grammar for ISL. The text output was then mapped to the gesture outputs to get the desired outcome.

The research paper on this project was presented at the 2021 International Conference on Computing, Communication, and Intelligent Systems (ICCCIS) and was subsequently published in IEEE. The paper can be found here.