Generating and Weighting Semantically Consistent Sample Pairs for Ultrasound Contrastive Learning

For boosting the performance of DNNs on small datasets, representation learning in a supervised fashion is the earliest-developed method, among which ImageNet [7] pre-training is the most popular one. The effect of ImageNet pre-training is verified by various medical tasks such as dermatologist-level classification of skin cancer [26], recognition of Alzheimer’s disease [27], and abnormalities with X-ray [28].

Though the power of ImageNet transfer learning is robust for various medical applications [29], the transfer performance will be inevitably harmed by the large domain gap between natural images and medical images [5]. For example, transferring the ImageNet representations to glioma subtype identification will be less helpful since gliomas are highly heterogeneous tumors with variable 3D imaging phenotypes [9], which is hardly related to any images in ImageNet. When large-scale annotated medical datasets are available, supervised pre-training directly on medical domains becomes a preferred method to bridge the domain gaps. For example, Tajbakhsh et al.[30] designed a 2-channel image representation of emboli trained with voxel-level labeled pulmonary embolism data to better detect pulmonary embolism. Riasatian et al.[31] proposed to use weakly labeled whole slide images of formalin-fixed paraffin-embedded human pathology samples to learn histopathology image representations. Though frequently used and adopted by our previous conference paper [25], traditional label supervision is abandoned in this work for its apparent drawback: the high requirement of domain-specific annotation prohibits its further scaling.

In the medical field, unlabeled image data are far richer than labeled ones. Pre-training medical models with artificial labels, namely self-supervised representation learning, was verified to be an effective way to train with unlabeled data [32]. Self-supervised learning algorithms [33, 15, 34] surged recently in mainstream computer vision to learn generic visual representations since they show great potential to exploit the natural supervision inherent in the images themselves. Models Genesis [12], a perturbed auto-encoder framework, is the milestone of the medical self-supervised learning for its comparative transfer performance to ImageNet strongly supervised counterparts. Since then, from the technologies of frame order prediction [19], generative learning with Restricted Boltzmann Machine [35], Jigsaw game [36], image context restoration [33, 12] to more advanced contrastive learning [10], self-supervised representation learning boosted the medical models’ transferabilities to small datasets significantly.

Recently, contrastive learning [14] has been a dominating self-supervised method for its outstanding performance. It aims to pull semantically consistent positive samples together in the representation space, while pushing those dissimilar negative samples away. With the fast development of contrastive learning frameworks [16, 37], more and more works [13, 25] seek to combine properties inherent in medical data for learning better image representations. Since the US video clips are much different from individual images that contrastive learning was initially designed for, previous methods [19, 10] are not effective enough to learn powerful representations from them. We regard a key factor to the success of US contrastive learning to be mining semantic consistency from video clips to generate reasonable positive pairs. Previous works [21, 38] showed that an ideal positive pair should contain moderate appearance differences while keeping their semantics consistent. For this reason, this work considers the similarity between US frames as semantic continuity, and leverages this property to generate semantically similar samples from US videos.

Sample weighting is an effective approach to improve the training performance when the data are diverse, e.g., free-hand US videos. It increases the effect of “good” samples’ gradients on the parameter updating and decreases the effect of those not good enough. In contrastive learning, “good” positive sample pairs denote those with higher appearance differences and lower semantic differences. Meta-weighting [23, 24], a meta-learning [39] technique, can achieve this target without any handcrafted weighting schemes. It is regarded as a bi-level optimization process [22], and aims at optimizing the weighting scheme (upper-level objective) for conducting a better learning process (lower-level objective). To learn a better weighting scheme, there are two main meta-learning frameworks. The first one [23] is assigning weights for each sample, and updating the weights based on the generalization error given by this weighting scheme. This method is easy-to-implement but converges slowly. The second one [24] is weighting the samples with a weighting module, which is usually a shallow network. The parameters of the weighting module are updated to minimize the generalization error with their validation gradients. The recent applications of meta-weighting contain the recognition of out-of-distribution samples [40], label noises, and training set biases [23]. In the medical field, similar problems are still common, but only a few pieces of literature use this method to boost training efficiency [41, 42]. Considering that some sample pairs in contrastive learning contribute more to learning desirable representations while some are not, we assign larger weights for those more helpful sample pairs. This work extends the idea of the meta-weighting module to the medical contrastive learning task, which is a novel application scenario.

The core idea of contrastive learning methods [43, 10, 44, 18] is instance-level discrimination, which generates sample pairs from the original dataset and trains the neural networks by recognizing similar positive pairs against dissimilar negative pairs. Meta-USCL extends the core idea of contrastive learning from two aspects: 1) generating positive sample pairs from US videos by PPI module; 2) weighting the generated positive pairs with CMW-Net to make the training more efficient. The whole framework containing Meta-USCL pre-training and parameter transferring to downstream tasks is shown in Fig. 2.

The first part of Meta-USCL is generating positive sample pairs from videos. Given a raw video $V_{i}$, we construct a frame set $\mathbb{F}_{i}^{K}\!=\!\{\textbf{f}_{i}^{(k)}\}^{K}_{k=1}$ to discard overly redundant information, where $K$ is the number of extracted images. A simple way to do this is to sample several frames per second in a video. Then, the PPI module is used to generate a positive sample pair $\mathbb{S}_{i}^{2}\!=\!\{\textbf{x}^{(1)}_{i},\textbf{x}^{(2)}_{i}\}$ from the frame set.

After generating a sample pair $\mathbb{S}_{i}^{2}$, a DNN backbone $f(\cdot)$ encodes them to be a representation vector pair $h_{2i-1}=f(\textbf{x}^{(1)}_{i})$ and $h_{2i}=f(\textbf{x}^{(2)}_{i})$. In a video batch of size $N$, there are $2N$ samples in total, where $2i-1$ and $2i$ indicate two samples generated from the video $V_{i}$. Subsequently, the projection head (a two-layer MLP) $g(\cdot)$ maps the representations to be feature vectors $\{\textbf{z}_{2i-1},\textbf{z}_{2i}\}_{i=1}^{N}=\{g(\textbf{h}_{2i-1}),g(\textbf{h}_{2i})\}_{i=1}^{N}$ specialized for contrastive learning. In this feature space, samples can better be pulled together or push apart with contrastive loss without affecting the representation quality for downstream tasks. In addition, the CMW-Net compares the representation pairs of each positive sample pair, and outputs quality scores (weights) $\{\mathcal{W}(\textbf{h}_{2i-1},\textbf{h}_{2i})\}_{i=1}^{N}$ for reweighting their contrastive losses.

The InfoNCE [45] is the most widely-used loss function for contrastive learning. It aims at minimizing the distances between positive pairs $\{\textbf{x}_{i}^{(1)},\textbf{x}_{i}^{(2)}\}_{i=1}^{N}$, and maximizing the distances between negative pairs $\{\textbf{x}^{(1~{}or~{}2)}_{i},\textbf{x}^{(1~{}or~{}2)}_{j}\},~{}i\!\neq\!j~{},1\leq i,j\leq N$ in the mini-batch. This work modifies the InfoNCE to a weighted version, so that the loss for different positive sample pairs can update the model parameters differently:

	$\displaystyle\mathcal{L}$	$\displaystyle=\frac{1}{2N}\sum_{i=1}^{N}\mathcal{W}(\textbf{h}_{2i-1},\textbf{h}_{2i})L(\textbf{z}_{2i-1},\textbf{z}_{2i})$
		$\displaystyle=\frac{1}{2N}\sum_{i=1}^{N}\mathcal{W}(\textbf{h}_{2i-1},\textbf{h}_{2i})(l(2i,2i-1)+l(2i-1,2i)),$		(1)

where

$$l(i,j)=-\log\frac{\exp(s_{i,j}/\tau)}{\sum_{k=1}^{2N}\mathbbm{1}_{[i\neq k]}(k)\cdot exp(s_{i,k}/\tau)},$$

(2)

and

$$s_{i,j}=\textbf{z}_{i}\cdot\textbf{z}_{j}/(\|\textbf{z}_{i}\|\|\textbf{z}_{j}\|)$$

(3)

denotes the cosine similarity of a feature vector pair $\{\textbf{z}_{i},\textbf{z}_{j}\}$. The indicator functon $\mathbbm{1}_{A}(x)=1$ when $x\in A$, otherwise it equals to 0. And $\tau$ is a tuning temperature parameter.

Finally, the pre-training can be conducted with PPI-generated positive pairs and weighted InfoNCE loss. The contrastive training process learns the visual differences which would not affect the meaning of a US image. They act as the prior knowledge to assist the downstream classification or segmentation tasks.

The typical contrastive learning paradigm [14] generates positive pairs based on random augmentation and cannot perform well for US videos, because they were primarily designed for image datasets and will meet the three problems introduced in Sec. 1.

First, to prevent the negative-pair similarity conflict problem, the PPI module generates only one positive sample pair from one video. Secondly, it uses a mix-up [46] operation to generate semantically consistent positive pairs, which mitigates the positive-pair dissimilarity defect and similarity redundancy problems. The process of PPI is visualized in Fig. 3. In the first step, it randomly selects three frames $\widehat{\textbf{x}}_{i}^{(1)},\widehat{\textbf{x}}_{i}^{(2)},\widehat{\textbf{x}}_{i}^{(3)}$ in chronological order from the frame set $\mathbb{F}_{i}^{K}$ of video $i$. And then it regards the middle frame $\widehat{\textbf{x}}_{i}^{(2)}$ in temporal as an “anchor” frame, interpolates it with the other two perturbation frames $\widehat{\textbf{x}}_{i}^{(1)},\widehat{\textbf{x}}_{i}^{(3)}$ respectively:

$$\left\{\begin{array}[]{l}\textbf{x}_{i}^{(1)}=\xi_{1}\widehat{\textbf{x}}_{i}^{(2)}+(1-\xi_{1})\widehat{\textbf{x}}_{i}^{(1)}\\ \textbf{x}_{i}^{(2)}=\xi_{2}\widehat{\textbf{x}}_{i}^{(2)}+(1-\xi_{2})\widehat{\textbf{x}}_{i}^{(3)}\end{array},\right.$$

(4)

where $\xi_{1},\xi_{2}\sim Beta(\alpha,\beta)$, and $\alpha,\beta$ are parameters of $Beta$ distribution. On the one hand, positive pairs are random offsets from the anchor image to the perturbation images, ensuring that they share the common semantic information from the anchor image, which overcomes the positive-pair dissimilarity defect problem. On the other hand, the sampling interval for a frame set $\mathbb{F}_{i}^{K}$ results in low probability for PPI to sample three similar frames $\{\widehat{\textbf{x}}_{i}^{(k)}\}_{k=1}^{3}$ which are too close in appearance, which prevents the positive-pair similarity redundancy problem. The generated two positive samples $\textbf{x}_{i}^{(1)},\textbf{x}_{i}^{(2)}$ would be further randomly augmented to gain more abundant visual differences.

The PPI module generates positive pairs and regards samples from different videos as negative pairs. The benefits of the PPI can be summarized as follows: 1) Interpolation makes every point in the feature convex hull enclosed by the cluster boundary possible to be sampled, making the cluster cohesive as a whole; 2) Positive pairs generated with Eq. (4) have guaranteed semantic consistency and appearance discrepancies to make the training more stable and efficient.

For US videos that are too long, PPI may still generate dissimilar positive pairs due to the random mix-up coefficients $\xi_{1},\xi_{2}$ of the $Beta$ distribution. In addition, videos collected by different sonographers from different organs have different sequential characteristics, making them contribute to the representation learning differently in quality. So it is challenging for PPI to solve the three problems thoroughly. Sample weighting with CMW-Net is a promising method to make the model learn more with desirable sample pairs and further mitigate these problems.

The CMW-Net is mainly inspired by [24], which proposed a sample weighting net to discover the important samples for training. This work extends this method and proposes to obtain weights for each positive pair in contrastive learning. Let $\Theta_{m},\Theta_{c}$ denote the parameters of the main pre-training network and the CMW-Net, respectively. This weighting network takes the representation vectors $\textbf{h}_{2i-1},\textbf{h}_{2i}$ of a positive pair as the input, and outputs corresponding quality score $\mathcal{W}(\textbf{h}_{2i-1},\textbf{h}_{2i},\Theta_{c})$. The weighted training loss $\mathcal{L}^{train}(\Theta_{m},\Theta_{c})$ can be defined as Eq. 1 on the training set. The aim of representation learning is to minimize the weighted training loss w.r.t. the main network parameter $\Theta_{m}$ given CMW-Net parameter $\Theta_{c}$.

$$\Theta^{*}_{m}(\Theta_{c})=\mathop{\arg\min}_{\Theta_{m}}\mathcal{L}^{train}(x,\Theta_{m},\Theta_{c}).$$

(5)

Note that $\Theta^{*}_{m}$ can be regarded as a function of $\Theta_{c}$ since it acts as part of the learning scheme. The parameters $\Theta_{c}$ decides how the weight of each sample pair is calculated. We use the validation loss as a guide for the CMW-Net to find parameters with better generalization. The updating of $\Theta_{c}$ is based on the unweighted loss $\mathcal{L}^{valid}(\Theta^{*}_{m}(\Theta_{c}))$ calculated on the validation sample pairs with optimal main network parameter $\Theta^{*}_{m}$:

	$\displaystyle\Theta^{*}_{c}$	$\displaystyle=\mathop{\arg\min}_{\Theta_{c}}\mathcal{L}^{valid}(\Theta^{*}_{m}(\Theta_{c}))$
		$\displaystyle=\mathop{\arg\min}_{\Theta_{c}}\frac{1}{2N}\sum_{i=1}^{N}L^{valid}(\textbf{z}_{i}^{(1)},\textbf{z}_{i}^{(2)},\Theta^{*}_{m}(\Theta_{c})).$		(6)

The problem for deriving optimal parameters of the two networks can be understood as a nested optimization, where the lower-level objective is the minimization of training loss w.r.t. $\Theta_{m}$ (Eq. 5) while the upper-level objective is the minimization of validation loss w.r.t. $\Theta_{c}$ (Eq. 6).

Parameter Updating Procedure. To solve the above optimization problem, we design an online updating algorithm inspired by [24]. The core idea is to iteratively update the parameters of the two targets with the following three steps: (1) getting feedback about the current weights; (2) updating the weighting strategy; and (3) updating the network parameters. In the iteration $t$, the algorithm first updates the main model parameters

$\displaystyle\hat{\Theta}^{(t)}_{m}(\Theta^{(t)}_{c})=\Theta_{m}^{(t)}-\alpha_{1}\nabla_{\Theta_{m}}\mathcal{L}^{train}(\Theta_{m},\Theta_{c}^{(t)})\left.\right|_{\Theta_{m}^{(t)}},$

(7)

with one step of gradient descent, where $\alpha_{1}$ is the learning rate of the main model. This step is performed in a mini-batch of training data. Its function is to get feedback for the weights given by CMW-Net.

After taking the derivative of the meta (validation) loss w.r.t. the $\Theta_{c}$, we can take the second step to move the $\Theta_{c}^{(t)}$ along the meta objective gradient on the validation data

$\displaystyle\Theta_{c}^{(t+1)}=\Theta_{c}^{(t)}-\alpha_{2}\nabla_{\Theta_{c}}\mathcal{L}^{valid}(\hat{\Theta}_{m}^{(t)}(\Theta_{c}))\left.\right|_{\Theta_{c}^{(t)}},$

(8)

with $\alpha_{2}$ as the learning rate of the CMW-Net.

Finally, the updated $\Theta_{c}^{(t+1)}$ is used to generate the weights of this batch of sample pairs, and obtain the updated main model parameters with the third step of gradient descent:

$\displaystyle\Theta_{m}^{(t+1)}(\Theta_{c}^{(t+1)})=\Theta_{m}^{(t)}-\alpha_{1}\nabla_{\Theta_{m}}\mathcal{L}^{train}(\Theta_{m},\Theta_{c}^{(t+1)})\left.\right|_{\Theta_{m}^{(t)}}.$

(9)

All these three steps are conducted in an iterative fashion. They update the parameters of CMW-Net and main model alternatively.

In this work, we construct a US pre-training dataset named US-4. It is collected from four convex probes [1] US video datasets involving two scan regions (i.e., lung and liver). The attributes (e.g., depth and frame rate) of the US-4 dataset are described in Tab. 1. Among the four sub-datasets, Liver Fibrosis and COVID19-LUSMS datasets are collected by our local sonographers [34, 49] with Resona 7T ultrasound system. The frequency is FH 5.0, and the pixel size is 0.101mm - 0.127mm. Butterfly [47] and CLUST [48] are two public sub-datasets. In order to generate a diverse and sufficiently large dataset, images are selected from original videos with a suitable sampling interval. We extract three samples per second for each video, ensuring the sampling interval $I>5$ so that US-4 contains sufficient but not redundant information about videos. This results in 1366 videos and 39,021 images. US-4 is relatively balanced regarding the number of images in each video, where most videos contain tens of US images.

Meta-USCL proposed in this work is implemented with PyTorch 1.8.0. The model training can be split into two parts: pre-training and fine-tuning. In the pre-training phase, 80% of data are used for training, and 20% are used for validation. The batch size of 32 is adopted to conduct the training. We conduct contrastive learning with $224\times 224$ images for the POCUS pneumonia detection and the UDIAT-B breast tumor segmentation tasks. For BUI breast cancer diagnosis, we use $64\times 64$ images. The key reason for using a small image size for transferring to this classification task is that the BUI dataset mainly contains small images, which will be elaborated on in Sec. 5.2. For the experiments without meta-learning, the optimizer is Adam, with a learning rate $\alpha_{1}=10^{-3}$ and an $L2$ regularization (weight decay) strength of $10^{-4}$. The temperature parameter $\tau$ equals 0.5 as in [14]. For the experiments with meta-learning, the optimizer for the main model is replaced with SGD with a learning rate of $\alpha_{1}=0.1$. The learning rate for CMW-Net is $\alpha_{2}=6\times 10^{-5}$ as in the [24]. In each case, the images are augmented with random cropping followed by transformation to the initial size, random flipping, and random color jittering to make the model robust to different image appearances from different imaging systems. As important hyper-parameters, the batch size and the strengths for data augmentations are discussed in detail (see Sec. 5.3).

As for fine-tuning, we consider downstream classification and segmentation tasks. For classification tasks, we use a consistent batch size of 128 with the same image sizes as pre-training. The optimization method is set to be Adam with a learning rate of 0.01, training 30 epochs. Images are augmented with randomly cropping, resizing, and flipping. Please note that only the last three layers of pre-trained models are tuned in the fine-tuning phase of classification. For the segmentation task, we discard the final classification layer and use the pre-trained DNN backbone to construct a Mask-RCNN [50] model, and fine-tune it on the target dataset. The segmentation is trained with the batch size of 2 in the original image size. SGD method with learning rate $5\times 10^{-3}$, momentum 0.9, $L2$ regularization strength $10^{-4}$ is used for optimization. Only random flipping is used for data augmentation. All fine-tuning experiments are repeated 5 times. The mean and standard deviation of the results are reported to ensure the reliability of the results.

We first report the experimental results validating different components of Meta-USCL (Tab. 2). All pre-trained models are fine-tuned on the POCUS [1] pneumonia detection dataset. The POCUS is a widely used lung convex-probe US dataset for COVID-19 detection, consisting of 2116 frames from 140 videos for three classes (i.e., 655 frames from 58 videos for COVID-19, 349 frames from 24 videos for bacterial pneumonia, and 1112 frames from 58 frames for healthy controls). The video data were originally collected by Born et al. [1] from online websites and pre-processed for the subsequent algorithm development. We uniformly split POCUS into five folds w.r.t. videos in different classes and conduct a 5-fold cross-validation to ensure that the frames of a single video are present within a single fold only.

In this experiment, we test different DNN architectures (i.e., ResNet [51], ShuffleNet-v2 [52]) with different layers. First, ResNet is one of the most common network architectures for medical image analysis. Second, ShuffleNet-v2 is a practical choice for deploying DNN models to clinical US systems for its low memory footprint and high inference speed. All the architectures are chosen with increasing sizes (i.e., from 1M to 25M parameters). We also test 5 variants of Meta-USCL, namely SimCLR, $S_{1}$, $S_{2}$, $S_{3}$, and meta+$S_{3}$. “SimCLR” represents pure SimCLR trained with US-4 as an image dataset. Further, the $S_{1}$ variant only chooses one frame from a video and augments it twice to be a positive pair to avoid negative-pair similarity conflict, while $S_{2}$ chooses two frames from a video and augments them to gain richer visual features. Finally, $S_{3}$ (the USCL proposed by our preceding work [25]), is the method of using the PPI module to generate sample pairs with three frames in a video. Meta+$S_{3}$ adds extra CMW-Net to $S_{3}$ for reweighting. As shown in Tab. 2, there are a few noteworthy results: 1) Training from scratch gives undesirable classification accuracy. 2) Transfer from ImageNet demonstrates consistent and significant improvements for ShuffleNet v2, but negative transfer appears for ResNet. This result is consistent with [1], indicating that not all pre-trained models are beneficial. 3) With more and more sophisticated design for contrastive learning (from SimCLR to $S_{3}$), representation learned with the US-4 dataset becomes more and more powerful, finally outperforms ImageNet pre-training significantly. 4) Meta-weighting improves both SimCLR and USCL, implicating that positive pairs in contrastive learning usually do not contribute to the model training equally.

This experiment demonstrates that the pre-training on ImageNet does not bring consistent and stable improvement for the downstream performance. However, it is also hard for simple medical adaptation of self-supervised methods like SimCLR to learn satisfying US medical visual representations because the US videos have different properties from individual images, and US-4 is far smaller than regular pre-training datasets of traditional natural images. Therefore, more targeted designs for the US are needed. From SimCLR to $S_{1}$, the negative-pair similarity conflict problem is tackled. $S_{2}$ makes use of the semantic similarity between the video frames. It regards the appearance differences between frames as a perturbation, acting as a kind of natural data augmentation to promote the training process. $S_{3}$ uses the same idea as $S_{2}$ but further mitigates positive-pair similarity redundancy and positive-pair dissimilarity defect problems with PPI. The pre-training performances improve monotonically by solving negative-pair similarity conflict ($S_{1}$), enriching features ($S_{2}$), and solving positive-pair similarity redundancy and positive-pair dissimilarity defect ($S_{3}$) incrementally. In addition, the meta-learning module learns how important the positive sample pairs are by CMW-Net, and assigns different weights for different positive pairs in a batch. The experiment shows that the $S_{3}$ setting is still not perfect, and CMW-Net still improves $S_{3}$ by 0.3% to 3.0% with different DNN architectures. The reason why meta-weighting works will be elaborated in Sec. VI.

In comparative experiments, we select several supervised (ImageNet pre-training, supervised pre-training using the categorical labels of US-4) and self-supervised (SimCLR [14], MoCo v2 [53], Jiao et al. [19] ) methods to compare the transfer performance on CAD and segmentation tasks. All experiments are conducted with ResNet18. We use t-tests to evaluate the significance of performance improvement between our methods to ImageNet.

POCUS Pneumonia Detection. We show the sensitivity and specificity of the two abnormalities in POCUS. It can be observed from Tab. 3 that the Meta-USCL achieves the highest sensitivities for both COVID-19 and Pneumonia. It is also the only method to reach the normal/abnormal and overall accuracy above 90%. In addition, some interesting results can be noticed. Firstly, training DNNs from scratch only gets 84.3% total accuracy and low sensitivity for COVID-19, but some pre-trained models even perform worse (e.g., ImageNet). This result is consistent with the previous literature [1], reporting that pre-training is not necessarily helpful in some cases. For ImageNet, its domain has a large gap to the POCUS dataset, which may lead to negative transfer. With Meta-USCL, we can achieve performance up to about 95%, which is higher than the 89% reported in [1] and 94% in [25]. The result shows that our method appears to be a new SOTA.

BUI Breast Cancer Classification. BUI [54] is a breast cancer dataset collected with linear US probes. In this task, we need to classify whether the breast tumor that an image contains is malignant or not. This dataset contains 250 breast cancer images, where 100 samples are benign, and the rest 150 are malignant. We use the metrics of binary classification (e.g., AUC) to evaluate the transferability of pre-trained models. The 5-fold cross-validation results are shown in Tab. 4. In the BUI dataset, there are fewer images than in the POCUS, thus it needs better visual representations to train DNNs for gaining generalization on this task. In addition, videos in US-4 are derived with convex probes. The domain gap between different kinds of probes doesn’t weaken the performance of Meta-USCL. Our results show that Meta-USCL reaches a similar performance improvement on BUI as POCUS (e.g., its accuracy improvements compared with training from scratch are 10.3% and 10.2%). We hypothesize that the general US representations learned by Meta-USCL lead to a more powerful adaptation ability on a small US dataset than other methods. Therefore, it makes up for the performance loss brought by the domain gap. This hypothesis is validated in the following Sec. 5.3. It is noteworthy that the BUI contains images with an average size of $120\times 90$, making the pre-training and fine-tuning with smaller inputs even more stable and powerful (Tab. 5). This is because when pre-trained on small images of US-4, the learned basic pattern distribution is similar to the BUI dataset.

UDIAT-B Breast Tumor Segmentation. The UDIAT-B [55] consists of 163 linear US probe breast tumor images from different women with a mean image size of $760\times 570$ pixels. Each images presents one lesion with pixel-wise mask annotation. We choose this dataset to evaluate the model transferability for the downstream segmentation task. Of the 163 images, 53 are cancerous masses, and the other 110 are benign lesions. We randomly select 113 images for training and the other 50 images for validation. Since the segmentation task needs precise discrimination for pixels, we compare different methods with $224\times 224$ pre-training image size for better detail resolving ability.

For segmentation, our main concern is the true positive (TP) region, which is the overlapped region between the ground-truth mask and the prediction. The importance of TP in clinical scenarios is primarily due to the low tolerance of missing any important lesion. Therefore, the three metrics $Dice=2TP/(FP+2TP+FN)$, $PPV=TP/(TP+FP)$, and $Sensitivity=TP/(TP+FN)$ are chosen to evaluate the segmentation performance, where TN, FP, FN are the number of true negative, false positive ,and false negative pixels. The final results are shown in Tab. 6. We can see that the Meta-USCL outperforms all other counterparts in this pixel-level prediction task. Note that both domain (linear/convex probe) and the task (segmentation/instance discrimination) between the downstream task and pre-training task are different, and the number of samples is also limited. The medical pre-training methods with contrastive image operations, including Meta+SimCLR and Meta-USCL, show the best performance compared with other supervised or unsupervised methods. This phenomenon indicates that: 1) instance-wise discrimination learns more pixel-level prediction ability than class-wise discrimination due to the semantic-invariant appearance discrepancies; 2) Meta-USCL reaches a significantly higher performance (3.1%, $p<0.05$) compared to ImageNet, showing the potential for specialized representation learning methods to beat general pre-trained models on segmentation applications.

In addition, we visualize the segmentation results in Fig. 4. The first row illustrates that Meta-USCL pre-trained model can distinguish the edge of a large tumor well. The upper half of this lesion is quite clear, resulting in consistent segmentation results of all methods. However, the lower half (see green rectangles in Fig. 4) becomes blurred, so better semantic understandings of the tumors are needed, where Meta-USCL performs the best. The second row shows that the Meta-USCL pre-trained model can localize a small lesion region precisely. Given an inconspicuous tumor, its outline is hard to distinguish from the black background area, making the detection of small lesions even more challenging than segmenting the edges of large lesions. Only ImageNet pre-training and Meta-USCL pre-training work well in this case.

This subsection first explores the hyper-parameter tuning procedure. And then, how and why Meta-USCL works well is discussed through visualization of the knowledge that CMW-Net and the main model learn. Finally, we use quantitative analysis to reveal the performance w.r.t. the data amount of pre-training and fine-tuning.

Hyper-parameter Tuning. Previous works have shown that hyper-parameters are critical for contrastive learning to achieve desirable performance [14, 16]. Proper data augmentation and batch size are two of the most important factors that highly affect the transferabilities of learned image representations. Following SimCLR [14], we try image cropping, color jittering, and Gaussian blurring in our experiments. The results are shown in Tab. 7. For both image cropping and color distortion, there are optimal parameters of moderate strengths. However, the Gaussian blurring does not show any performance gain for ultrasound contrastive learning. We conjecture that the imaging sharpness of the US systems is lower than the regular optical lens. The blurring leads to more unsatisfying image quality for US images than natural images. Therefore, Gaussian blurring is less helpful for models to learn transformation invariants since it cannot guarantee semantic consistency. For SimCLR, it was proved that the larger batch size improves the performance to a larger extent. But this conclusion does not hold for Meta-USCL (see Tab. 8). The batch size of 32 is the best for Meta-USCL, and the larger batch size even brings an evident drop in transfer learning performance. This phenomenon may be caused by the high similarity of frames from different US videos. Compared with natural images, US videos are far more similar to each other. Larger batch size increases the probability of sampling similar videos from the pre-training dataset, which may lead to the negative-pair similarity conflict problem.

Visualization of CMW-Net Learned Weights. We visualize the sample pairs and the corresponding weights to verify that the meta-knowledge learned by CMW-Net can judge whether a positive pair is beneficial or not. As shown in Fig. 5, we use frames of a lung US video without any augmentation to obtain weights of sample pairs with CMW-Net. The leftmost image is the first frame of the video, and the following frames present descending cosine similarities to the first one. The resulting weights between the initial frame and the rest show a trend of rising and falling w.r.t. image similarities because the semantic marks are disappearing and the textural dissimilarities are becoming larger. The ideal sample pairs should have small semantic distances and appropriate texture information gaps between the two samples, which turns out to be moderately inconsistent (e.g., the third frame and the initial frame in Fig. 5).

In addition to different weights between positive pairs from a video, we also observe different weight distributions between different sub-datasets of US-4 (Fig. 6), indicating that different US domains do not contribute their medical visual representations equally. For example, about half the sample pairs of the COVID19-LUSMS sub-dataset have weights below 0.4, meaning they may be less important for pre-training. We infer that the different distributions are caused by the CMW-Net’s target, minimizing the overall validation loss. The imbalanced US videos of different organs may make some patterns (i.e., lesions) faster to be recognized, while the minority patterns are less focused. This results in the validation sample pairs of minority classes having higher loss values. For example, the COVID19-LUSMS sub-dataset has the most abundant videos among the four sub-datasets. Therefore, CMW-Net assigns lower weights to it so the model can learn more from smaller subsets to reduce the overall validation loss. It is also noticeable that the differences between sub-datasets do not exactly match the number of videos per subset. We conjecture that the meta-reweighting network does not only consider the number of videos of different subsets. It is also affected by other factors such as the organ and lesion distributions, video lengths, and frame similarities (e.g., the frame similarity of a CLUST video is higher than that of other sub-datasets, which leads to a lower probability of positive-pair dissimilarity defect and may be a reason for its higher overall weights).

All in all, CMW-Net makes contrastive learning work better by 1) emphasizing more on the sample pairs that help the model learn better semantic consistency; 2) balancing the sample pairs from different sample sub-domains to decrease the overall validation loss.

Visualization of Representations Learned by Meta-USCL. We use tSNE to visualize the feature space of the ImageNet, SimCLR, and Meta-USCL pre-training. Fig. 7 shows the feature distribution of randomly selected 15 US videos in the frame level. The ImageNet pre-trained DNNs are insensitive to US textures, resulting in scattered feature vectors. The reason may be that ImageNet pre-training does not involve sufficient US-related images, leading to the poor discriminative ability of US data. Using SimCLR to learn representations on US-4 can mitigate this issue. But the similarity defect problem makes the features from a video distributed on a broader range than Meta-USCL. This may be because SimCLR pushes two similar frames apart when they are sampled as a negative pair. However, Meta-USCL pre-training takes advantage of the characteristics of US videos to learn better feature representations, making all representations from a video distributed in a cluster. Overall, the performance advantage of Meta-USCL is due to two factors: (1) the pre-adaptation to the US domain and (2) its reasonable positive pair generating and weighting scheme.

The Relationship between Pre-training Data Amount and Transfer Learning Performance. Unlike ImageNet, US-4 is a video dataset of US containing only 1300 videos. Therefore, it is very likely that the increasing amount of pre-training data can improve the pretraining performance. As illustrated in Fig. 8, we only use one sub-dataset (i.e., COVID19-LUSMS or Liver Fibrosis, which both have above 600 videos) to investigate the pre-training performance w.r.t. data amount and data domain. The pre-trained models consistently perform better on the POCUS pneumonia detection task with increased video amount. Moreover, we find that the uptrend seems far from saturated when the video amount reaches 500, which means the larger US video dataset can further promote the pre-training performance. In addition, the performance curve of lung pre-training is nearly always higher than liver pre-training, which might suggest that: 1) pre-training with Meta-USCL on the same domain gains more benefits than on a different domain; 2) pre-training in a different domain can also boost the performance of downstream tasks, though it is not an ideal solution.

The Relationship between Fine-tuning Data Amount and Transfer Performance. It was shown in Sec. 5.2 that Meta-USCL pre-trained models gain comparative or even greater advantages when fine-tuned on smaller datasets than ImageNet pre-training. We demonstrate fine-tuning results with different video ratios of the POCUS dataset in Tab. 9. As the data ratio decreases from 100% to 10%, the ImageNet pre-trained model has a 17.5% performance drop, but Meta-USCL only shows a 12.7% difference. The result indicates that when fine-tuning on a small amount of data, Meta-USCL pre-trained models generalize the pattern of downstream tasks better since they are pre-trained on the US video domains themselves. The network parameters of these pre-trained networks can be regarded as closer to optimal points of different downstream tasks than ImageNet pre-training, leading to easier generalization.