Application description
Goal of the project is to familiarize people with the Polish Sign Language as it's the means of communication for the 50 thousand Deaf people in Poland. The application allows users to learn the alphabet of PSL by watching videos of signs and practicing showing the gestures themselves.
1. Data used to train the model
The main part of the application is the sign recognition module, which uses a neural network to recognize the signs shown by the user. The training data was collected over half a year and consists of over 5000 arrays of hand keypoints (detected using MediaPipe library).
Examples of the data can be seen below.
The data was additionally augmented 11 times (the original data was randomly rotated, translated, and scaled 10 times) to increase the size of the dataset. It was later divided into training, validation, and test sets in the ratio of 80:10:10 and stratified to ensure that all gestures are included in the same ratio in each part of data.
2. Model to recognize signs of the Polish Sign Language
A sequential neural network was created with Tensorflow library to recognize the signs. It consists of an input layer, batch normalization, 2 LSTM layers, 3 dense layers and an output layer. Between each LSTM and dense layer, a dropout layer was added.
For every layer, the ReLU activation function was used, except for the output layer, where the softmax function was used, and for optimization, the Adam optimizer was used.
Diagram of the model can be seen below.
During the training, both Learning Rate Reduction and Early Stopping callbacks were used to prevent model overfitting to training data. The metric of the model was categorical accuracy and the loss function was categorical crossentropy.
Plot of change of the model accuracy over epochs and final confusion matrix can be seen below.
3. Game to practise signs
To make the process of learning the signs more engaging for user, the application includes a game where the user is supposed to spell a word (displayed as an image) using the Polish Sign Language alphabet. For each correctly shown sign, the user receives a point and there is a bonus for completing the whole word. Currently there are 50 different words and each graphic was generated by an AI model from Canva.
In settings, user can change which hand is used, how long the game lasts, and the difficulty level.
Statistics of each game are saved and can be accessed in the Django admin panel. There is also a short tutorial available from the main menu that explains how to use the application.
4. Signs tutorial
Apart from the sign recognition module, the application also allows user to learn the alphabet of the Polish Sign Language by watching videos and graphics. After selecting a letter in the application, proper video is displayed.
