SimplyMime: A Control at Our Fingertips

The majority of existing methods can be divided into two categories, with sensor-based systems constituting the first. Utilizing specialised sensors to capture the movement and orientation of the hand, sensor-based methods for hand gesture recognition capture the hand’s motion and the orientation. These sensors may include accelerometers, gyroscopes, magnetometers, and other types of sensors that can measure the hand’s movement and position [10]. Sensor-based methods have a number of advantages over other methods, including the ability to capture precise information about the hand’s movement and position and the capacity to function in environments with poor lighting or visibility. In a recent study, the authors employed the utilization of inertial sensors, specifically accelerometers and gyroscopes, in conjunction with a multilayered perceptron classifier to recognize hand gestures [11]. Adopting a similar methodology, other researchers have developed a finger-worn ring device that captures acceleration data, paired with an LSTM model for the classification of hand gestures [12, 13, 14]. Likewise, Inviz device incorporates textile-based flexible capacitive sensor arrays for motion detection [15]. This approach has been utilized to develop systems that cater to individuals with paralysis, paresis, weakness, and restricted range of motion. Other notable developments include the WristCam [16], a wrist-worn camera sensor that estimates and recognizes hand trajectories, as well as the EchoFlex [17], a wearable ultrasonic device that tracks hand movements through a novel method. The research makes use of the muscle flexor data collected by the proposed device to develop an algorithm for gesture recognition that combines image processing and neural networks.

Sensor-based systems provide precise information about the hand’s movement and position and can function in environments with poor lighting or visibility. However, the requirement for the user to wear additional devices restricts the versatility and scalability of these systems. In many instances, faulty sensor calibration can result in errors in gesture recognition, resulting in abysmal performance.

The integration of computer vision and human-computer interaction has enabled the development of innovative systems aimed at addressing limitations and providing users with a more immersive and efficient experience when interacting with machines. One of the initial approaches in the development of vision-based sensors was the utilization of a colored glove [18]. This method required the user to wear a colored glove, which enabled the system to employ a nearest-neighbor approach for tracking hand movements at interactive rates. Several studies that employed the use of the Kinect sensor utilized color and image depth data to establish a hand model for the analysis of tracked hand gestures. The hand gesture tracking was accomplished through the implementation of a Kalman filter [19]. However, it was observed that the gesture recognition accuracy was relatively low. Similar research utilized RGB-D cameras to extract hand location data through the use of in-depth skeleton-joint information from images [20, 21]. Furthermore, similar to the architecture of SimplyMime, a few works employed neural networks to infer real-time hand landmarks, which were used to establish a skeletal structure of the hand gesture [22, 23].

Computer vision-based systems are non-intrusive and more versatile than sensor-based systems, but they can be affected by poor lighting or visibility. Moreover, recognizing gestures from complex backgrounds is still a challenging task. As a reason, SimplyMime focuses on developing robust and reliable hand gesture recognition systems that can overcome these challenges and enable natural and intuitive interactions with electronic devices.

The foundation of an effective hand gesture identification model is the accurate detection and isolation of the hand from the given image. To achieve this, SimplyMime employs state-of-the-art neural network technology for localizing the coordinates of the palm. Our model specifically utilizes the Single Shot Detector (SSD) architecture [24]. While conventional RCNN models have been widely used for object localization, they require significant computational power, making them less reliable for real-time applications [25]. In contrast, our SSD algorithm achieves superior real-time performance by eliminating the need for bounding box proposals and the subsequent feature re-sampling stage. This allows for a more efficient and effective approach to hand detection, which is a crucial component of an accurate gesture identification system.

The hand detection model employed in SimplyMime utilizes a unique approach in which it is initially trained on human palms rather than the entire hand. This approach allows the network to more easily perceive patterns and generate bounding boxes, as the palms and fists are more rigid objects when compared to a human hand with articulated fingers. In the task of detecting human hands, the last layer of the model creates substantially smaller anchor boxes, as human hands are smaller objects and thus require smaller anchor boxes. The architectural network of SimplyMime’s detection model is illustrated in Figure 3. The feature extraction network takes an input RGB image of 224x224px and is followed by a series of convolutional layers, referred to as ConvBlocks, which serve as the bottleneck for the higher abstraction level layers. Essentially, the image is passed through 5 single and 6 double ConvBlocks. The introduced ConvBlocks consist of a series of convolutional layers, followed by a depth-wise and a point-wise convolutional layer. The detailed architecture of our Convblocks is depicted in the Figure 4. Furthermore, our network outperformed a popular light-weight model, MobileNetV2-SSD [26], in terms of both accuracy and inference speed.

The detection architecture of SimplyMime is designed to be highly robust and accurate, and this is achieved through the use of a diverse set of training data. To ensure that the model is able to generalize well to a wide range of scenarios, it is initially trained on three different sources of data. The first of these is an in-the-wild dataset, which comprises a diverse set of images captured in various geographical locations and under different lighting conditions, allowing the model to learn to detect hands under a wide variety of conditions. The second data source is an in-house dataset, which is specifically designed to cover all possible hand angles in a controlled environment. This dataset allows the model to learn to detect hands under consistent conditions, and the combination of these two datasets allows for a well-represented and diversified dataset. Finally, the model is further trained on a synthetic dataset to ensure that it is able to detect hands under a wide range of angles and in a variety of environments, further improving its robustness and accuracy.

Having successfully localized the hand in the image, the next step in SimplyMime’s pipeline is to extract key-points from the isolated hand region. For this purpose, we employ a modified version of the Convolutional Pose Machines (CPM) network, which has been extensively used in the field of human pose estimation [27]. The CPMs provide a confidence map for each keypoint, represented as a Gaussian centered at the true position. These maps are generated based on the size of the input image patch, and the final location of each keypoint is determined by identifying the peak in the corresponding confidence map. By assigning 21 landmarks across different points across the palm, the model achieves to generate a skeletal structure of the palm.

The protection of security is of paramount importance in hand gesture-based systems utilized for controlling consumer electronics, as these systems are susceptible to unauthorized access and control [28]. Without robust security mechanisms, these systems can be easily compromised, leading to data breaches, unauthorized access to sensitive information, and potential damage to devices. Therefore, it is essential to incorporate advanced security measures during the design and development of hand gesture-based systems to mitigate these risks. The integration of strong security measures is critical for ensuring the reliability and integrity of hand gesture-based systems for consumer electronics.

SimplyMime employs advanced biometric authentication methods, specifically, PalmPrint identification. To achieve this, we utilize a Siamese Network architecture, a common approach used in similarity recognition tasks like facial recognition [29]. Our network includes two parallel feature extraction blocks that process two distinct palm images, producing a 4096-dimensional tensor for each image. By measuring the dissimilarity between the two sets of embeddings utilizing the euclidean distance function, the similarity between the two palms is verified. A pre-determined threshold is established to establish the authenticity of the user. Access to the system is restricted if the dissimilarity measure exceeds the threshold, ensuring that only authorized individuals have access to the system. This multi-factor approach ensures that only authorized individuals have access to the system, protecting against unauthorized attempts to control the consumer electronics. Furthermore, implementing these security measures guarantees the protection of personal and sensitive information, enhancing the system’s reliability and integrity.

During the training process, the network is presented with three input images, including an anchor image, a positive image, and a negative image. The anchor and positive images correspond to the palm print of the same individual, while the negative image represents the palm print of a different individual. The network is trained for 100 epochs using the Triplet Loss function, which is optimized using Adam optimizer [30]. The XceptionNet [31] serves as the base model, acting as the encoder. The output of the training process is a set of two distances, which are input into the Triplet Loss function [32], as shown in Equation 1:

Figure 7 illustrates the architectural functioning of the SimplyMime’s authentication module.

In the Equation 1, the function $f(x)$ maps each input image to a 4096-dimensional embedding, represented by a tensor. The input images, denoted as $a$, $p$, and $n$, respectively correspond to the anchor, positive, and negative samples used in the triplet loss function. The margin parameter $\alpha$ is used to control the relative distance between the positive and negative pairs, with the goal of maximizing the separation between them.

The hardware setup of SimplyMime is a crucial aspect of its overall design and functionality. The camera is mounted on a cuboidal cardboard structure that is equipped with a motor on its right side, which enables control over the Y-axis movement. The cardboard structure is further mounted on a CD disk which is connected to another motor, allowing for control over the X-axis movement. These motors are connected to an On-board microcontroller, which facilitates communication between the hardware components and the software algorithms. Figure 9 provide a visual representation of the hardware setup, while the circuit diagram in the Figure 8 illustrates the connections between the various components. The hardware setup is designed to be compact, portable, and easy to set up, enabling users to easily control consumer electronics with hand gestures.

The HANDS dataset comprises a substantial corpus of samples collected from 5 diverse participants, consisting of 3 male and 2 female individuals [33]. The participants were instructed to perform 16 pre-defined gestures, comprising of 12 single-handed gestures performed with both arms, and 4 double-handed gestures. Each gesture was captured in 150 RGB frames, resulting in a total of 11,250 images across all participants. Additionally, the gestures were captured from varying distances, resulting in a diverse range of depths and variations in the hand poses.

Upon evaluation of the dataset, our proposed model achieved an overall accuracy of 96.16%. The evaluation metric, the recall in particular, is extremely crucial for evaluating SimplyMime as they provide an understanding of how well the model is able to correctly identify hand gestures among the input data, and balance the trade-off between false positives and false negatives. These metrics aid in evaluating the overall performance of the model and the ability of the model to generalize to unseen data. The Recall is used compute the proportion of true positive predictions among all actual positive instances in the data. In the context of SimplyMime, a high recall score indicates that the model is able to identify a high proportion of true hand gestures among all gestures present in the input data. These performances are further highlighted in Figure 11 and Table II, where the gesture-specific results are also illustrated.

Furthermore, in order to evaluate the effectiveness of our proposed system, we conducted a comparative analysis against other notable models in the field. One such study utilized Generative Adversarial Networks for hand gesture detection [36], while another employed a similar pipeline to ours and leveraged the popular MobileNetV2 architecture as the baseline model [37]. Despite the high accuracy results achieved by these models, they required significant computational power. In contrast, our proposed system, SimplyMime, achieved comparable performance and precision while requiring significantly less computational resources as shown in Table III.

A robust hand gesture-based control system requires a strong hand detector as its backbone. The hand detector’s primary function is to accurately localize the hand region within an image, which is crucial for the subsequent gesture recognition stage. Therefore, to evaluate the performance of our hand gesture recognition model, SimplyMime, we also conducted evaluations to measure the capacity of our hand detector. The results of these evaluations provide insight into the model’s ability to accurately detect and localize the hand region, which is crucial for the overall performance of the system. Additionally, by comparing the results of our hand detector with those of other models, we can gain a better understanding of the performance of our system in relation to the state-of-the-art.

The FreiHand dataset is a benchmark dataset specifically designed to evaluate the performance of hand detection and hand pose estimation models [34]. The dataset contains over 32,560 frames of synchronized RGB and depth data, captured using Microsoft Kinect v2 sensors, collected by 32 people. This dataset is considered to be one of the most challenging datasets for hand detection and pose estimation, as it contains a wide range of hand poses and motion, captured under various lighting conditions and backgrounds. To evaluate the performance of SimplyMime, we used the Freihand dataset to test our model’s ability to detect hands and estimate hand poses. The dataset was particularly useful in evaluating the model’s performance under challenging conditions such as low resolution, low contrast, and occlusions [34]. Our model was able to achieve a high level of accuracy in detecting hands and estimating hand poses, even under these challenging conditions. Our system’s results on the dataset are depicted in Table IV.

In addition to evaluating the performance of our hand gesture recognition model, we also sought to assess the accuracy of our palmprint identification component. To do so, we utilized the CASIA Dataset [35], which comprises a large and diverse collection of palmprint images. The dataset includes images from over 100 individuals, captured under various lighting conditions. However, we utilised a subset of the data, particularly the images taken in lowest wavelength sample and white light specifically. As a result, we acquired a dataset with 12000 sample in total. Further on, we pre-processed the dataset leveraging a custom data loader that generated test triplets from the CASIA dataset, where a triplet consists of an anchor image, a positive image, and a negative image. By training our model on these triplets, it is able to learn to differentiate between the palmprints of different individuals and accurately identify a user based on their palmprint. We trained our model using a triplet loss function [29] and Adam optimizer [30], which allows the model to optimize its error by comparing the similarity between the anchor and positive images to that of the anchor and negative images. This approach allows the model to learn the underlying features of a palmprint that are unique to an individual, which enables it to accurately identify a user based on their palmprint. The proposed model’s training metrics are depicted in the Figure 12.

One of the key factors for optimized performance of our model is the Triplet Loss [29]. We utilised Triplet loss to compare the relative similarity of three inputs: an anchor image, a positive image, and a negative image. The anchor image is taken to be the ”neutral” image, while the positive image is a image from the same subject, in our case, and the negative image is from a different subject. The functions was utilised to minimize the distance between the anchor and positive images, while maximizing the distance between the anchor and negative images. This is done by adjusting the model’s weights and biases to better discriminate between the three inputs. As a result, the model becomes better at recognizing the similarities and differences between the anchor and positive images, and can more effectively differentiate between the anchor and negative images. The Figure 13 illustrates the confusion matrix acquired from the test set of the data.

Finally, the evaluation results of our proposed SimplyMime model on multiple benchmark datasets indicate its effectiveness in both hand gesture recognition and palmprint identification. Compared to existing solutions, our model has demonstrated superior performance. Further, by incorporating palmprint identification as an additional security measure, the model’s practicality and usability in real-world applications have been further enhanced, positioning it as a viable alternative to conventional remote control devices. The novel combination of hand gesture recognition and palmprint identification in one system presents an exciting advancement in the field of human-computer interaction.