COVID-Net Assistant: A Deep Learning-Driven Virtual Assistant for COVID-19 Symptom Prediction and Recommendation

The coronavirus 2019 (COVID-19) pandemic, caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), continues to impact human society worldwide significantly. Real-time reverse transcription polymerase chain reaction (RT-PCR) testing remains the most reliable standard screening tool for detecting COVID-19. While RT-PCR testing is the primary screening tool against COVID-19, it typically requires two to three days to complete and significantly burdens the healthcare system. Another widely used screening test is the antigen test. They are rapid tests that produce results in 15-30 minutes. While medical screening tests are becoming more accessible and widespread, many countries do not have widespread access, especially in some developing countries.

Earlier works show deep learning is successful on particular COVID-19 classification tasks [1, 2] with deep convolutional neural networks (CNNs). Deep learning algorithms involve using a dataset to train a tailored parameterized model to help detect symptoms and scientific evidence. The success of deep learning led researchers to explore new diagnosis methods via visual information, such as chest X-rays and CT images. However, a successful deep learning algorithm usually leverages a large dataset with high-quality labels, which is expensive in medical studies. Nevertheless, such visual information is challenging to retrieve in a short time. A more affordable and efficient pre-screening method is beneficial to the use of medical resources.

Coughing is one of the primary symptoms of COVID-19 and is also the consequence of other diseases. Due to the difference in infection sources, cough sounds may differ in patterns hardly detectable by humans. With signal processing techniques, imperceptible patterns can be learned and detected by deep learning techniques. One of the signal processing techniques is Mel-frequency cepstral coefficients (MFCCs), which work well with CNNs.

Motivated by providing an affordable and accessible tool to provide early recommendations and leverage the use of diagnostic tests in the fight against the COVID-19 pandemic, in this work, we present a design of a virtual assistant — COVID-Net Assistant. It can sit on a mobile device or website to provide early COVID-19 recommendations based on cough recordings due to the feasibility of retrieval. Figure 1 shows the workflow of COVID-Net Assistant: 1) An individual records or uploads cough audio through the application; 2) The audio gets processed by an efficient deep convolutions neural network to predict whether the individual may have signs of COVID-19; 3) If the network predicts signs of COVID-19 based on the audio, it will give the user a recommendation that they may have signs and seek medical advice or further testing using COVID-19 tests such as PCR and antigen tests.

We hope the open-source nature of the COVID-Net Assistant ¹¹1https://github.com/fshipy/COVIDNet-Assistant encourages further innovation as part of the COVID-Net global open initiative ²²2http://www.covid-net.ml/

We found three public cough datasets that frequently used in relevant research: the COUGHVID [3] dataset, Coswara [4] dataset, and Covid19-Cough [5] dataset. The positive samples in COUGHVID and Coswara datasets are all self-reported, which does not guarantee the correctness of labels. In our study, we require strong labels to indicate the status of COVID-19 infection; therefore, we chose Covid19-Cough as the main dataset to move forward.

This Covid19-Cough dataset consists of 1324 raw audio samples with 682 cases labeled positive, and 382 of them are confirmed by PCR test. For the audio pieces labeled negative, no verification information is provided. The audio was collected through a call center or telegram. We excluded two malformed audio files in our study, leaving us with 1322 pieces of audio, with 681 labeled positive and 381 positive samples verified.

In our study, we trained deep learning models with two dataset splits. The first split, namely Verified-Only, excludes all unverified positive samples as we aimed to experiment with high-quality labels. The other split, namely All-Data, contains a total of 1322 valid samples from the Covid19-Cough dataset. We used 60%, 20%, and 20% of the audio for each dataset split as the training, validation, and testing data, respectively. Table 1 shows the exact label distribution in each dataset split.

We set up experiments to compare the representation power and efficiency of COVID-Net Assistant models generated via generative synthesis [9], based on different seed designs introduced in section 3.3. Specifically, we evaluated the Area under the Receiver Operating Characteristic Curve (AUC) as the primary performance metric. To express the theoretical capacity of models, we also calculated the total number of parameters and floating point operations (FLOPs) of the final architectures of COVID-Net Assistant models.

Table 3 shows the performance and capacity of each proposed COVID-Net Assistant deep neural network generated via generative synthesis differentiated by seed designs. From Table 3, we noticed that COVID-Net Assistant CNN has dominant performance on the Verified-Only split, with the second smallest model in terms of the number of parameters. In addition, there is a slight increase in AUC by increasing the complexity of a residual block architected seed design (Res-CNN). In contrast, increasing the complexity of a depth-wise separable convolution architected seed design (DW-CNN) will cause a slight drop in the AUC score. The variants of COVID-Net Assistant DW-CNN have relatively small capacities, especially for models generated with seed design DW-CNN-S/M, whose numbers of FLOPs are less than 1/8 of other models. We also noticed a difference in the capacity levels of final architectures while the models were trained on different dataset splits with the same seed design. For example, the generated COVID-Net Res-CNN-S trained on All-Data Split has FLOPs fewer than half of the one trained on Verified-Only Split, while the number of parameters is only $3\%$ fewer. From our investigation, the difference is mainly in the number of filters in the first two stages described in Table 2, which are the most computationally expensive stages. We also noticed that a more complex seed design could result in a model with fewer FLOPs via generative synthesis. For instance, the final architecture of COVID-Net Assistant Res-CNN-M has $7\%$ fewer FLOPs than COVID-Net Assistant Res-CNN-S, trained on the Verified-Only split, while Res-CNN-S has much fewer parameters.

We noticed a considerable performance difference (over 0.1 AUC) on the test sets between the model trained on All-Data split and Verified-Only split. We hypothesized that the root cause is the more extensive noise of the unverified positive samples in All-Data split. To verify our hypothesis, we tested the models on the same test set in All-Data split but filtered — the unverified positive samples were rejected, left with 63 verified positive samples and 146 negative samples in the filtered test set. Table 4 shows the AUC scores on the filtered test set. As a result, the scores are close to the ones evaluated on Verfied-Only split, which is more than 10 points higher than the scores on the unfiltered test set, referred to Table 3. The observation implies that the unverified positive samples are noisy and may negatively impact the models trained via standard supervised learning.

Motivated by deploying the models on different platforms, such as mobile and web applications, we evaluated the model’s forward pass latency on different platforms, reported in Table 5. Particularly, we computed the trimmed mean of the latency in 1000 forward propagation on 1) a workstation (WORKSTN) with x86_64 architecture and Intel Xeon Gold 6230 CPU, 80 cores, 3.9 GHz, 400GB RAM; 2) a single-board computer (SBC) with aarch64 architecture and Cortex-A72 CPU, 4 cores, 2.0 GHz, 4GB RAM. According to Table 5, a model’s latency on the WORKSTN is not necessarily proportional to the latency on the SBC. For instance, the COVID-Net Assistant CNN trained on Verified-Only split can perform the fastest inference on the WORKSTN, but on the SBC, the latency of COVID-Net Assistant DW-CNN-S is almost half of its. The models infer fastest on the SBC are COVID-Net Assistant DW-CNN-S/M, whose FLOPs are strictly fewer than other models. However, for lightweight models, The impact of fewer Flops on a workstation with abundant computational resources is not significant because of the large amount of parallel computation that can be performed in a deep convolutional neural network.

In this study, we proposed the initiative of COVID-Net Assistant, a deep learning-driven system that performs pre-screening of COVID-19 conditions by processing cough audio. The goal of COVID-Net Assistant is not to make a diagnosis but to provide an early recommendation to users on whether they may have signs of COVID-19 and to seek medical advice or further COVID-19 tests. The study is purely research and should not be leveraged for medical advice or diagnosis. However, With effective pre-screening techniques and recommendation systems, limited medical sources like RT-PCR tests will gain more usability in the fight against the COVID-19 pandemic.

Our results show that deep learning algorithms can do well on a dataset with verified positive samples. We proposed multiple deep convolutional neural network architectures generated via generative synthesis  [9] with different seed architecture designs. We found the standard convolution based architecture, namely COVID-Net Assistant CNN, can achieve dominant representation power on the dataset with less noise; the lightweight depth-wise separable convolution based architecture, namely COVID-Net Assistant DW-CNN-S, can achieve the highest efficiency on low-end devices such as a single-board computer; and the residual block based architecture, namely COVID-Net Assistant Res-CNN-M, can achieve a great balance between representation power and efficiency, while the neural networks are not deep enough to fully leverage the power of residual connections.

One of the challenges is that the audio samples in the dataset are scarce. Due to the limited numbers of test data, the AUC scores with a slight difference are hardly comparable. More training data is necessary to generalize the model’s target missions and deploy them in practice. In fact, deep learning algorithms have shown promising results in many diagnostic tasks with a massive amount of labeled training data. However, high-quality labeled data in medical diagnosis is expensive to obtain and is always a challenge in related literature. Fortunately, recent studies have used semi-supervised learning approaches to leverage the unlabeled data or weak labels (e.g., [15]). As one of our future research directions, we will leverage transfer learning and semi-supervised learning algorithms to train the models on larger, crowdsourced datasets and leverage the unverified labels, such as combining the samples in COUGHVID [3] dataset and Coswara [4] dataset with Covid19-Cough dataset [5] currently in use.

The promising AUC scores on the Verified-Only dataset split also imply a potential pattern in coughing sounds by COVID-19 infection differentiated from other diseases. Hence, another research direction is leveraging explainable AI techniques to study the region of interest used in neural networks’ prediction. This could further help us study and understand the performance of generative synthesis. We hope our research can motivate further study and applications or even research in biological reasoning.

We would like to thank DarwinAI Corporation, Vision and Image Processing Lab, and Saeejith Muralidharan Nair (University of Waterloo) for preparing the computing resources and platforms. We would also like to thank the Natural Sciences and Engineering Research Council of Canada and the Canada Research Chairs program.