WearMask: Fast In-browser Face Mask Detection with Serverless Edge Computing for COVID-19

Historically, most of the papers focused on performing face recognition while wearing a mask or with other obstructions, while few, if any, previous studies have been conducted to determine if a subject is wearing a mask. Since the COVID-19 pandemic, however, more efforts have been devoted to face mask detection due to the emergence of the need for reducing COVID-19 transmission.

To detect occluded faces, Shiming Ge et al. established the \acMAFA dataset, which contains 30,811 images and established the LLE-CNNs model, obtaining an Average Precision (AP) of 76.4% [9]. A. Nieto-Rodríguez and others developed an alarm system for the wearing of masks in the operating room, combining the face detector with the mask detector, and optimized for low False Positive rate and high Recall, and obtained 95% True Positive rate [10]. But its direction is limited to the surgical mask in the operating room. Bosheng Qin et al. established SRCNet and classified the mask-wearing situation into three categories: no facemask-wearing, incorrect facemask-wearing, and correct facemask-wearing, to classify images and obtain 98.70% accuracy [11]. However, it needs to crop the face area before face mask detection and concentrate on the face. Mohamed Loey et al. used ResNet-50 and SVM to obtain 99.49% accuracy on \acRMFD [12]. Later, they used YOLO-V2 based on ResNet-50 for object detection in another publication and achieved 81% \acAP on the Medical Masks Dataset (MMD) and Face Mask Dataset (FMD) [13]. G. Jignesh Chowdary et al. achieved 99.9% accuracy on \acSMFD with an InceptionV3 based model [14]. Nevertheless, since \acSMFD is a small dataset generated using mask images. Results are more difficult to be generalized in complicated scenarios.

As opposed to these studies, whose goals were to develop a more precise face mask detection algorithm, our work emphasizes leveraging lightweight serverless edge-computing based in-browser deployment of face mask detection with deep learning. To the best of our knowledge, no previous publications have investigated the serverless edge-computing based in-browser solution of face mask detection by aggregating deep learning models, serverless edge-computing frameworks, and stack-based virtual machines.

The typical way of designing in-browser deep learning model is to use TensorFlow.js [15] or ONNX.js [16]. These methods use a specialized JavaScript library to read the model file and completes the inference through calling computing operations written in JavaScript. However, JavaScript is not a typical programming language in the deep learning field, with less community support.

The data used to train the \acWearMask model consists of two types: (1) face with masks, and (2) face without masks. The normal face data (without masks) were collected from the WIDER FACE dataset [20]. The existing mask datasets were divided into two categories, including real faces with real masks (\acMAFA, \acRMFD [21], MMD) and real faces with generated masks (\acSMFD). Considering that the generated mask images influence model generalization ability, we used real images instead. Briefly, the \acMAFA is composed of face pictures with various faces with heterogeneous masks, background, and annotation information. The \acRMFD only contains the faces without background. Considering that the background images without faces would help the model improve precision, the \acMAFA dataset were finally selected as training data. However, the face in \acMAFA only contains the regions below the eyebrows, while the WIDER FACE dataset marks the whole face’s position. To reconcile the differences from the annotation protocols of two datasets, the partial annotations in WIDER FACE was extended to whole faces, following the protocol that the marked bounding box range was from below the hairline to above the lower edge of the chin, and the left and right borders do not include the ears. Moreover, we also collected some samples of incorrectly wearing masks and some edge conditions. As a result, the final training dataset contained 4065 images from \acMAFA, 3894 images from WIDER FACE, and 1138 additional images from the internet. In general, a total of 9,097 images with 17,532 labeled boxes were divided into 80% training and validation, and 20% testing.

To deploy deep learning models on edge devices without advanced GPUs, the efficiency and size of such models are essential. In general, the smaller the model is, the less computational resource is needed on edge devices. The YOLO-Fastest model is employed as the detection method in the \acWearMask solution, as a lightweight version of \acYOLO [4] object detection model. \acYOLO is one of the most widely used fast object detection algorithms. As opposed to the traditional anchor-based detection algorithms (e.g., Faster-RCNN [22]), \acYOLO divides the image into predefined grids to match the target objects. The grid definition alleviates repetitive anchor computation, thus improves the computational efficiency dramatically. To further improve the computational speed, YOLO-Fastest was proposed as an open-source object detection model. In the YOLO-Fastest implementation, the EfficientNet-lite [23] is employed as the encoder. The total model size is only 1.3 MB (MegaByte), compatible with low computing power scenarios such as edge devices.

To further speed up the convergence of the training process, the COCO dataset (Microsoft Common Objects in Context, 80 categories) [23] was employed to pretrain the detection backbone. The model was then further trained by the face mask datasets as a transfer learning practice [24], by adjusting the network to suit new object definitions.

Moreover, as opposed to current cloud-computing based implementations, we implemented the \acWearMask as a serverless edge-computing framework. For face mask detection, the cloud computing solution is limited by the availability of enough internet bandwidth of real-time camera video streams, with high costs for cloud services. By contrast, our serverless edge-computing design minimizes the hardware costs by utilizing users’ existing devices. Beyond the lower hardware costs, the edge-computing design has another critical advantage of low privacy risk. In \acWearMask framework, the video data are processed locally without uploading processes (e.g., upload to cloud servers), which is essential as a healthcare-related method.

We further implement the edge-computing strategy as an in-browser deployment by aggregating NCNN and \acWASM architectures ( Fig. 2). NCNN is an open-source optimized inference framework for mobile platforms developed by Tencent. It is implemented in pure C++ without third-party library dependencies. Herein, the size of the compiled model will be minimized, with decent computing efficiency on edge devices, especially on devices with ARM architecture chips. It supports custom layers, which means it works when there are custom operators in the model. Moreover, it has provided tools to convert from various frameworks to NCNN. \acWASM is a low-level language that run in the browser. It is in binary form, which is faster than JavaScript programs. Moreover, prevalent web browsers, such as Chrome, Edge, Firefox, and WebKit (Safari), have already supported \acWASM with minimal generalizability barriers.

Using the NCNN library, we first converted the PyTorch model into an NCNN model with a model size of 581 KB. Then we created a new C++ program to streamline the detection process from image preprocessing to finally output the category, confidence, and boxes position, using the NCNN library for inference. After compiling this C++ program into \acWASM format, the entire framework was executed as a function in JavaScript. To visualize the detection results, we used HTML and CSS to render the detected bounding boxes with original face images.

The training was performed on Google Colab (Tesla V100-SXM2-16GB). The PyTorch Version was 1.7.0 + cu 101. The training code was modified from the public code for YOLO-V3 [25] from Ultralytics [26].

The qualitative results of our proposed WearMask framework is presented in ( Fig. 3). It is able to correctly identify the cases when a subject is not wearing the mask properly, such as when the nostrils or mouth are exposed. It also has a higher probability of recognizing that it is not a mask even if the object uses the hands, elbows, or other things such as cell phones to cover the face. The model is able to work on multiple faces within the same frame.

From quantitative results, the trained YOLO-Fastest model achieved \acmAP@0.5 of 0.89 after 120 epochs with batch-size 16( Fig. 4). The detection FPS on representative edge devices are presented is Fig. 5.

After converting the PyTorch model to an NCNN model with \acWASM, we established a website and published it at facemask-detection.com, to demo the in-browser deployment.

Previous related works used various datasets with different tasks and evaluation strategies. For instance, some of the methods evaluated the performance of detection and classification using detection metrics, while other works only evaluated detection or classification, respectively. Therefore, most of the previous evaluation results cannot be directly compared with the results obtained in this paper. In this study, we compared our performance with the most similar settings among previous works, which (1) used real mask pictures instead of simulated mask pictures, (2) performed simultaneous object detection and classification.

Among them, the LLE-CNNs model used by Shiming Ge et al. obtained an AP of 76.4% [9]. Mohamed Loey et al. used YOLOV2 with ResNet-50 and obtained an AP of 81% [13]. Our model achieved an AP of 89% ( Fig. 4). Although the datasets are not completely same, it shows that the performance of this model is at a comparable level with prior arts. Moreover, this study aims to build a light-weight edge computing framework for face mask detection rather than to achieve the best detection accuracy. More testing examples are presented in ( Fig. 6)