Multi-Domain Few-Shot Learning and Dataset for Agricultural Applications

The embedding function is trainable and consists of two parts: feature extraction $\phi_{x}$ and transformation $\psi_{x}$. The feature extraction step $\phi_{x}$ extracts the features and projects them to a $d$-dimensional space. The feature extractor architecture consists of a ResNet-18 [10] network pre-trained on Imagenet [26]. The extracted features, however, are not ideal and do not capture important discriminative visual features for a specific episode
task [35]. In other words, simply using an embedding generated by $\phi_{x}$ does not incorporate any information about other support samples in the current episode. As a result, irrespective of the cohort, the embedding of a given image is always the same. Having such deterministic features is detrimental for discriminating the elements of the set. To overcome this, the embeddings need to be contextualized. Inspired by FEAT algorithm proposed in [35], the outputs of $\phi_{x}$ are transformed using a set-to-set function ($\psi_{x}$), which enriches the embedding of each image by considering those of all other images in the support set. By enabling interaction between embeddings of various images in the set it leads to richer representations and discriminative features. Another desired feature of the set-to-set function is that it should be permutation-invariant. Following [35], we implement $\psi_{x}$ using Transformer architecture [31], which fit the criteria of the required set-to-set function. Transformers rely on the self-attention mechanism which updates the representation of every input by weighing the relevance of all the other inputs. They map a query and a set of key-value pairs to an output.

where $f(\phi_{x_{i}},\phi_{x_{j}})$ is used to measure the similarity between $\phi_{x_{i}},\phi_{x_{j}}$.

where the functions Q and K are learned linear projections. Combined with V, which is also a learned linear projection, the functions Q and K project the inputs to a common representation space before applying the similarity measure.

The resultant transformed embeddings $(\psi_{x})$ act as inputs to the distance computation module.

In this step, we calculate the distance between the embeddings of the support samples with the query sample to finally classify the query. Following [2], we use Mahalanobis distance [8], a class-covariance based distance metric estimated at test-time, which has been shown to be better than using other metrics such as Euclidean or cosine. The final prediction is calculated as follows:

where $n\in N$ is one of the support classes in the current episode/task $t\in T$, $S^{t}$ is the support set for the current episode/task, $\mu_{n}$ is the mean transformed embedding ($\psi_{x}$) of all the $M$ samples corresponding to the class $n$ and $d_{n}$ is the squared Mahalanobis distance defined as below:

where $Q_{n}^{t}$ is the covariance matrix of class $n\in N$ for episode/task $t\in T$. We calculate the covariance matrix $Q_{n}^{t}$ using a regularized estimator

where $\Sigma^{t}$ is the covariance matrix of all the classes in the task $t$, $\Sigma_{n}^{t}$ is the covariance matrix of the class $n$ in task $t$ and $\lambda$ is a weigthing factor defined as follows:

where $|S_{k}^{t}|$ is the number of elements in the support set of the task $t$ corresponding to class $k$. For a theoretical explanation of the superiority of Mahalanobis distance to other metrics, we refer the reader to [2].

To ensure that the transformation function $\psi_{x}$ pulls the embeddings of same-class instances closer and drives different-class instances farther, we use a contrastive loss function. To achieve this, the transformation function is applied to instances of each of the $N$ classes present in the given episode/task, giving a transformed embedding $\psi_{x}^{\prime}$ and the mean class centers $\{c_{n}\}_{n=1}^{N}$. We then calculate the similarity of individual embedding to the class center corresponding to the individual. Apart from this we also use a standard cross-entropy loss function to calcualte the loss using the final prediction. Hence, the complete loss function is as follows:

where $sim$ is the similarity function calculated using the class-covariance-based distance metric, $l$ is the standard cross entropy loss function and $\lambda$ is the weighting factor.

The proposed model has been implemented using PyTorch [24]. We use stochastic gradient descent with Nesterov acceleration with an initial learning rate of 0.0002, weight decay of 5e-4, momentum of 0.9, and a learning rate scheduler with step size of 40 and gamma of 0.5 to optimize the model. We resize all the images to 84×84×3 before using the feature extractor. The value of $\lambda$ is set to 0.1.

We test our proposed architecture on two different challenging datasets and also provide baselines for the new dataset that we propose. We next describe the two external datasets that we use.

Plant and Pest (PP) [21] dataset contains 6000 images of both pests and plants. We use this dataset to test the mixed-domain and cross-domain performance of our proposed architecture. The dataset contains 20 classes, 10 each for plants and pests. During experimentation, the dataset is split into two: meta-train set and meta-test set. The labels of both sets do not overlap and the distribution of the sets may also differ. In mixed domain settings, both meta-train and meta-test sets contain instances of plants and pests. We show an example of this in Fig 2. On the other hand, there are two cross-domain settings. In the cross-domain 1 setting, the meta-train set contains examples of pests and the meta-test set is made up of plant instances. For example, Fig 4 can make up the meta-train set and the meta-test set contains images from Fig 3. Finally, the cross-domain 2 setting, refers to the meta-train set containing plants and the meta-test set containing pest instances. In this setting, the images in Fig 3 make up the meta-train set, whereas instances in Fig 4 constitute the meta-test set. While being challenging, the cross-domain setting is vital owing to its practical implications.

PlantVillage [13] is a public dataset containing 38 classes of healthy and diseased leaves of different crops. The dataset consists of 61,486 images. We split the dataset into 3 different and mutually exclusive meta-training and meta-testing sets, in the same manner followed in [1]. In split 1, the meta-test set consists of 10 different classes of Tomato (1 healthy, and 9 diseased) as shown in Fig 5, and the meta-training set consists of the rest of the 28 classes. In split 2, the meta-test set consists of 4 classes of Apple (3 diseased, 1 healthy): apple scab, black rot, cedar apple rust, and healthy; 4 classes of Grape (3 diseased, 1 healthy): black rot, esca, leaf blight, and healthy; 2 classes of Cherry (1 diseased, 1 healthy): powdery mildew, and healthy; and the meta-train consists of rest of the 28 classes. Similarly, in split 3, the meta-test set consists of 4 classes of Corn (1 healthy, 3 diseased), 4 classes of Grape (1 healthy, 3 diseased), and 2 classes of peach (1 healthy, 1 diseased).

Plants in Wild (PiW): In addition to these two datasets, we curated another dataset to validate our model’s performance across domains and under real-life settings. The images of leaves in the Plantvillage and PP dataset are in a laboratory setting such that each image contains only one leaf in a contrasting background. However, in a real-life setting, the images are less likely to be taken in a controlled setting and will be of varying backgrounds, lighting conditions, and angles. We used the publicly available images available on the internet (for instance, the Eden Library website¹¹1https://edenlibrary.ai/home) to collect images of diseased and healthy plants. All the images were taken using smartphone cameras in different fields and farms. We preprocessed all the images by resizing and center-cropping the images. Fig 6 shows some of the classes from our dataset. Comparing our dataset with the existing datasets, shown in Figures 3 and 5, highlights the differences between the datasets - lab controlled and real-life images.

We collected 20 categories of plants: 10 of which are healthy and 10 classes correspond to diseased plants. The dataset consists of 1980 images, which were taken by users using different smartphone cameras. Table 1 shows all the available classes in the dataset. We evaluate the model using three different splits of the dataset. In split 1, the meta-train set consists of Bell pepper healthy, Bell pepper leaf spot, Celery healthy, Chinese cabbage healthy, Chinese cabbage fusarium, Corn leaf blight, Corn rust leaf, Cucumber tetranychus, Cucumber thrips, and Red cabbage healthy. The meta-test set consists of the rest of the 10 classes. In split 2, the meta-test and meta-train sets are reversed. In split 3, the meta-train set consists of all the healthy plant leaf images, while the meta-test set consists of all the diseased plant leaf images.

We evaluate the performance of our proposed model on the PP dataset [21] and the PlantVillage dataset [13] to establish the superiority of our model compared to existing best-performing methods on these datasets. We then use our model and provide baselines for our newly proposed dataset. To evaluate the model, we generate 600 support-and-query sets from the meta-test set. To report the final accuracy, we use the average accuracy on all the 600 sets.

Plant and Pest (PP): On the PP dataset [13], we perform 36 sets of experiments, under three different settings: 2 cross-domain and mixed-domain following [21]. Please refer to the 5.1 for the details on different settings. For each of the domains we evaluate with different values of M and N. We also vary the value of N during the meta-train and meta-test phase, which we represent using $N_{meta\_train}$ and $N_{meta\_test}$. For example, a value of 3, 5 corresponding to $N_{meta\_train}$ and $N_{meta\_test}$ respectively means that the number of support classes in each episode is 3 during meta-training and 5 during the model evaluation (or the meta-testing) phase. We also experiment with 3 different values of M (1, 3, and 5). We observe significant performance gains across all the settings. Specifically, we observe around 3 to 6% gains in performance on the mixed-domain setting, approximately 7 to 11% improvement on cross-domain 1, and about 12 to 14 % improvement on cross-domain 2 setting. We observe that gains on cross-domain settings are much higher compared to the mixed domain setting. This shows that our architecture is much better at extracting discriminative features even on unseen domains. We also observe that the accuracy improvements when using a higher number of support classes during meta-testing are greater i.e. when using a value of 5 for $N_{meta\_test}$ compared to 3. Specifically, the average improvement across all the settings with a $N_{meta\_test}$ value of 5 is 9% while it is equal to 5% when $N_{meta\_test}$ is set to 3. This is due to the fact that, with lower values of $N_{meta\_test}$ even a random classifier can correctly predict the outcome with high probability (given just 3 three classes, a random classifier is correct 33% of the time). Increasing the value demands robust architectures for accurate predictions. In summary, we see the trend of the increasing performance gap with increasing difficulty of the task throughout our experiments: performance improvements on cross-domain settings are more pronounced compared to single-domain and the accuracy gap widens when we increase the number of support classes in meta-test set.

PlantVillage: On the PlantVillage dataset [13], we perform experiments under 6 different settings. We use three different splits (please refer to the Section 5.1 for details on splits) and use two different values of $M$ (1, 5) while keeping $N$ constant. Please refer to the row corresponding to Ours in Table 5 for the results. Experiments using our model outperform the current state-of-the-art architecture [1] on this dataset by 10 to 24%. Moreover, the performance gains are much higher in 1-shot settings with a mean improvement of 18% compared to an average improvement of 11% in 5-shot settings. This is in line with our observations on the PP dataset [21] that the performance gains are much higher as the tasks get tougher.

Plants in Wild (PiW): The PiW dataset, in contrast to the previous two datasets, consists of diseased and healthy images of plants taken in a natural setting. We perform six experiments using three different data splits (Please refer to the 5.1 for the details on different splits) to validate the performance of our model in a natural setting. As shown in Table 6, our model performs the same, if not better when compared to its performance on plant images in a controlled setting. These experiments show the effectiveness of our model even with real-life images taken with smartphone cameras.

Cross Dataset Performance: To further establish the superiority of our model in adapting to unseen domains, we only use the images of pests from the PP dataset [21] as our meta-train set and use plant images in the PlantVillage dataset [13] as meta-test set. We show that even under such a stringent cross-domain / cross-dataset setting, our algorithm still shows notable improvements (up to 17% gains). We record these values in the Ours - Cross Dataset row in Table 5.

In summary, results across multiple settings on various datasets indicate that using Transformers for generating episode-specific rich instance embeddings ($\psi_{x}$) provide for a useful calculation of episode and class specific covariance matrix ($Q_{n}^{t}$) in few-shot settings.