Real-Time Mask Detection Based on SSD-MobileNetV2

With the development of deep convolutional neural networks [7], object detection has gradually transitted from canonical methods to deep learning methods. Most of the current object detection technology is based on DCNN (Dynamic Convolution Neural Network), and capturing the position and classification of objects are its essence. With the continuous improvement of hardware computing power and the increasing number of excellent datasets, DCNN-based object detection has flourished. First, AlexNet[30] was proposed by Krizhevsky et al. and won the ILSVRC-2012 competition. This is the first time DCNN appeared in history. Then Girshick et al. proposed the RCNN(Regions with CNN) model[31] based on this model, marking that DCNN has entered a new stage in the field of target detection. The object detection algorithm based on deep learning has ushered in rapid development, new network models are constantly being proposed [8, 9], and parameters such as recognition accuracy are continuously refreshed.

Generally, object detection is divided into two categories, namely one-stage object detection and two-stage object detection. Two-stage object detection consists of two steps: the first step is to locate the target and find the position of the target to be detected in the image or each frame of the video. The second step is to classify objects and classify the located objects according to their various characteristic processes. The representative algorithm of Two-stage is the aforementioned RCNN series, including the original RCNN model and other improved models such as Fast RCNN [10] and Faster RCNN [11]. Although the accuracy of these algorithms can reach a relatively high level, the overall efficiency is relatively low due to the need for two processing. In order to solve this problem, Joseph et al. proposed Yolo (You Only Look Once), which is a model based on a single network [12], and the one-stage algorithm officially entered the stage of history. This type of method omits the step of generating candidate regions, and directly detects the final result in a single time, so the speed is very fast, but the accuracy of positioning is sacrificed. Subsequently, the Yolo family continued to expand, and gradually developed YoloV2 and YoloV3, etc., and the accuracy and speed of detection were continuously improved. Although the speed of the Yolo series is greatly improved than that of RCNN, the accuracy rate is not high enough compared to the RCNN series. So in order to solve this problem, Liu et al. proposed SSD (Single Shot MultiBox Detector) [13], which borrowed some ideas from both Faster R-CNN and Yolo, and is more efficient than Yolo in terms of dense multi-targets and large objects. It is effective and can achieve high accuracy while meeting the speed requirements of real-time detection.

MobileNetV2 is an image classifier that provides lightweight and accurate predictions. In MobileNetV1, a different method from traditional convolution is depthwise separable convolution [14], which can greatly reduce the parameters of the model. MobileNetV2 adds Linear Bottleneck and Inverted Residual to the v1 version. As one of the core networks of target detection, MobileNet is often used to reduce the size of the model due to its lightweight characteristics. For example, Huang R et al. utilize MobileNet to mitigate YoloV3 [15], thus achieving a balance between speed and accuracy.

With the continuous development of object detection, it is also gradually applied in many important fields, such as: face detection, pedestrian detection [16], automatic driving [17], medical decision-making [18], video surveillance [19], etc.

After the outbreak of Covid-19, masks have become an indispensable part of people’s daily life, and mask detection has gradually become a popular research direction in computer vision[29]. In general, mask detection still belongs to the category of object detection. In recent years, many scholars have done a series of studies in the field of mask research. Loey et al. adopted a hybrid model [20], in which one part uses the popular ResNet-50 in deep transfer learning for feature extraction, and the other part uses traditional machine learning algorithms such as SVM, decision tree, and ensemble learning. A feature extraction model with an end-to-end structure is constructed. High detection accuracy is achieved, but time consumption is not considered too much. Nagrath et al. obtain better performance of classifying mask images by establishing the SSDMNV2 model leveraging a deep convolutional neural network.  [21]. Apart from that, the contribution of a good dataset to the model cannot be ignored. In view of the lack of the current mask dataset, Wang et al. [22] constructed three face datasets with masks, namely: MFDD, RMFRD, and SMFRD dataset. And a new algorithm was proposed, which made the face detection with masks reach 95$\%$ accuracy. Jignesh Chowdary et al. obtained very high accuracy on the Simulated Masked Face Dataset (SNFD) dataset by transfer learning from a pre-trained InceptionV3 model [23].

In general, the above models have improved mask detection in some aspects, and our method will draw on their experience and focus on real-time mask detection.

Mask detection has always been considered a challenging task, and real-time mask detection brings us even greater challenges. As shown in Figure 1, real-time mask detection is generally divided into two types: active detection and passive detection. The former is mainly for large-scale detection in places with high traffic and high density. Each image used for processing contains multiple face-mask objects and often accounts for a small proportion. While active detection has only one target and collects images at close range, similar to the scene of face detection, only one result is returned per image. This paper mainly introduces more challenging passive detection techniques which are suitable for applications in public places.

The real-time mask detection model proposed in this paper is based on the Python language. We set up the development environment and completed all the work of model training, validation as well as testing on the Windows PC.

We list our experiment environment in table I.

We get the dataset from Kaggle. The first part is ”Face Mask Detection  12K Images Dataset”.¹¹1https://www.kaggle.com/datasets/ashishjangra27/face-mask-12k-images-dataset This dataset includes nearly 12K images, and the dataset divides the images into different folders according to whether people wear masks or not. All images with masks are crawled from Google, and other images without masks are preprocessed from Jessica Liby Jessica Li. However, the proportion of faces in the entire dataset is too high, so the Face Mask Detection Dataset ²²2https://www.kaggle.com/datasets/wobotintelligence/face-mask-detection-dataset is used. This dataset contains a large number of face mask images in complex scenes, and the proportion of faces is small. After simple filtering of the dataset, we obtained the original dataset of the system. This dataset consists of about 10,000 face mask pictures, 50$\%$ of which are faces with masks and 50$\%$ without masks. Each picture contains a variable amount of faces with or without a mask.

First of all: According to the Vicinal Risk Minimization (VRM) principle, in order to prevent overfitting, and also to maximize the use of limited data sets, we have adopted some methods of Data Augmentation. In addition to traditional techniques (eg: translation, rotation and mirroring), we also adopted MixUp method [26].

Mixup is a simple Data Augmentation method that is not related to the dataset itself. It can effectively improve the generalization ability of the model while being simple enough and not bringing too much additional burden to the model. In our system, the two sample-label data pairs to be processed are fused together(as shown in Figure 5) according to a certain ratio, and then a new image is obtained by linear interpolation, as shown in Equation (1) and Equation (2):

Where $x_{i}$,$x_{j}$ are raw input vectors, and $y_{i}$,$y_{j}$ are one-hot label encodings. ($x_{i}$,$y_{i}$) and ($x_{j}$,$y_{j}$) are two samples randomly selected from the training data, and $\widetilde{x}$ is a new sample after mixing $x_{i}$ and $x_{j}$. $\lambda\in[0,1]$ is the probability value, and $\lambda\sim Beta(\alpha,\alpha)$ obeys the Beta distribution whose parameters are all $\alpha$. In our experiments, we set the value of $\alpha$ to 0.4.

LabelImage is an image annotator that can be downloaded from GitHub and is easy to install. Markers are saved as XML files in PASCAL VOC format. The data set just obtained has no labels. We use the LabelImg tool ³³3https://github.com/heartexlabs/labelImg to label each image with Mask or NoMask, and use a rectangular box to mark the location of the mask. Make each picture have an XML file to record its information. In each XML file, the location of the corresponding picture and all the marked mask or nomask labels and their coordinates are recorded. In addition, all images and their corresponding XML files are divided into training part and test part, and the ratio of training and testing datasets is 9:1.

After getting the preprocessed dataset, we create a LabelMap file in pbtxt format, with a ”Mask” value of 1 and a ”NoMsak” value of 2. This file is used to provide auxiliary information for the subsequent marking and identification stages. Then convert all XML files and labelMaps into TFRecords files to prepare for subsequent training with TensorFlow FrameWork.

Our experiments are based on the TensorFlow Object Detection API. This framework can detect specific objects in images based on a given model, while users do not need to pay too much attention to internal details. We just need to write the $pipeline.config$ file as needed. This file contains all the configuration needed for training with the TensorFlow Object Detection API.

Since our system only recognizes whether the target is wearing a mask, $num\_classes$ is set to 2, corresponding to the labels of ”Mask” and ”NoMask”. The transfer learning method can build our model faster, so we specify to use the pre-trained model checkpoint0 as the training starting point instead of starting from scratch. The package pyCOCOtools (python API tools of COCO) is specified as an evaluation model to evaluate our model during training. After editing all other configuration items, we use the training file provided by the TensorFlow Object Detection API and specify the previously written configuration file to start training. Each stage in the training process is saved in a folder in the form of checkpoints to facilitate building the system later.

After the model training is complete, we can start building the real-time detection system. The construction of the real-time detection system mainly relies on the OpenCV library. We load a checkpoint from the model that we trained earlier. And use OpenCv to open the local camera, get the video stream, and capture frames from the video stream. Then we can use the loaded model to perform the following processing on the video frame: First, preprocess. Crop the image size to 320*320; then put it into the model for detection, and get the returned label and location; finally, postprocess. Draw a rectangular frame on the detected target area, and convert the detected labels into the text to mark around the rectangular frame. In this way, a real-time mask detection system is constructed using our trained model.

MS COCO (Microsoft Common Objects in Context) is a large-scale image dataset, usually used for object detection, segmentation, human keypoint detection, material segmentation, etc. COCO also provides a series of evaluation indicators, which is one of the most commonly used benchmarks for target detection. COCO provides more than ten indicators including average precision. We will use the COCO metric to evaluate our model. The metrics used in our model such as accuracy, precision, and recall are all based on True Positive (TP), False Positive (FP), True Negative (TN), and False Negative (FN). As shown in Equation(3. 4. 5). From Figure 6 we can clearly see that the mAP, AR, and Total Loss, denote that our model can get outstanding performance in mask detection tasks.

TensorBoard is a tool that can visualize the parameter changes during the training process. We use tensorboard to visualize the parameters during the model training process.

As shown in Figure 7 (a-e), under various conditions such as dark light, occlusion, and inadequate mask-wearing, our real-time mask detection system can still accurate positioning and achieve high accuracy.

However, for a few cases with severe occlusion and strange shapes of masks, there is still the possibility of misjudgment and omission.

Real-time detection faces challenges such as random occlusion, changing features, and subject blur. Therefore, the accuracy obtained in the actual real-time detection system is lower than that obtained in the model testing process. As shown in Figure 7 (f)

After a large number of tests in different scenarios, our real-time mask detection system can perform well in various environments. This has great significance for practical use. Our system takes up little resources and can run on embedded devices (such as Raspberry Pi). After actual use experience, our mask detection system can make a judgment in real-time which is so short that it can meet the detection needs of the crowd in a state of continuous movement.