A Comprehensive Study on Torchvision Pre-trained Models for Fine-grained Inter-species Classification

Alex Krizhevsky introduced the novel convolutional neural network (CNN) named AlexNet in 2014. A year after competing in the ImageNet Large Scale Visual Recognition Challenge (ILSVRC)[25] in 2012. This challenge requires the classification of a subset of ImageNet with over one million images and a thousand classes. AlexNet performed very well as it was announced the winner of the best performance for that year with top-5 error of 15.3%, 10.8% lower than the previous top performer. AlexNet is considered to be the first CNN to be announced the winner for the competition.

AlexNet architecture consists of a total of 8 layers, 5 convolutional layers, and 3 max-pooling layers. This architecture had many characteristics that made it perform very well, one is the utilization of ReLU (Rectified Linear Unit)[26] activation function within the convolutional layers in replacement of the popular Tanh at the time. ReLU offered better performance and lower computation time. AlexNet benefited as well from the training on multiple graphic processors which allowed to distribute work on different GPUs enabling for the training of a bigger model and lowering the computation time. AlexNet suffered from overfitting which is a general issue that many models suffer from. Overfitting occurs when the model is fitted too well to the data and is not able to generalize well to new data which means that the model do not classify new images very good. To solve this issue, AlexNet utilized some common solutions to the problem, data augmentation, and dropout layers[27].

VGG (Visual Geometry Group) is a very popular CNN that was introduced in 2014 by Karen Simonyan and Andrew Zisserman from the University of Oxford. And similar to AlexNet, VGG participated in the ILSVRC challenge in 2014. VGG scored very well as it was announced the winner (2nd place) for that year with an accuracy of 97.7%. This architecture was able to surpass the performance of AlexNet as it had a deeper model design as well as replacing the large convolution kernels of AlexNet with multiple small convolution kernels of size 3x3.

Residual Neural Network abbreviated (ResNet) is a powerful network that was proposed in 2015 by Kaiming He et al. in their paper[12]. ResNet made a very big impact on deep learning as its unique architectural design enabled it to go deeper with layers and provide great performance. It was not an easy matter to get good performance with very deep models as there has been the problem of vanishing gradient. Vanishing gradient is a major problem in machine learning, it occurs as more layers are added to the network. Each layer has a certain activation function, and when the gradient is back-propagated to the earlier layers multiple times this causes the gradient to approach to zero (vanish) which makes it very hard to update the weights with very small values, and thus, the model do not converge and train well. ResNet offered a solution to the problem, in their paper, the authors described the use of a unique neural network layer named the ’Residual Block’. These layers utilize ’skip connections’ which are specific shortcuts to jump over some layers. These jumps usually skip over two or three layers, and these skip connections consists of a ReLU activation function and a batch normalization in between. The main intuition behind the utilization of this structure is to avoid vanishing gradients by the re-use of activation values from a previous layer to enable the neighboring layer to learn its weight.

SqueezeNet has three main goals, first is to get more efficient in distributed training. Second is to export models smaller in size that are more convenient for professional and industrial clients. Lastly is to provide a model that is better deployed on embedded systems. SqueezeNet has two varieties, SqueezeNet_v1.0 and SqueezeNet_v1.1.

DenseNet was introduced in 2017 by Gao Huang et al. It is a powerful CNN that is similar to ResNet in using shortcut connections in between layers to solve the gradient descent problem. DenseNet utilizes ’Dense Connections’ in between layers with the use of ’Dense Blocks’ which holds an n number of ’Dense Layers’. Each Dense Layer consists of a 1x1 convolution filter for feature extraction and a 3x3 convolution filter to decrease the number of channels. And in Dense Blocks, each layer receives feature maps from all previous layers, and then it passes the output which is concatenated as input to all subsequent layers. This special structure allows for the use of fewer layers and the re-use of the features learned by the network, and having a narrower model (fewer layers) is easier to train with having fewer parameters to learn.

Inception[15][16][28] is a family of CNNs developed by Christian Szegedy et al. and Google with four different versions, Inception v1 which is the GoogLeNet, Inception v2, a refined version of Inception v1 with the introduction of batch normalization, Inception v3, a more refined version with additional convolution factorization, and Inception v4.

GoogLeNet (Inception v1) was the first member of the Inception family. This architecture was introduced in 2015 and prior to that the architecture participated in the ILSVRC challenge and did very well as it won 1st place and achieved a new State-of-the-Art for ImageNet in that year. GoogLeNet was proposed for the main problem of overfitting and the expensive computation of stacking too many layers. The main solution provided by the authors is to have convolution filters of various sizes (1x1, 3x3, and 5x5) at the same level. Having these different-size filters helps in extracting features of various details, big and small, and adding more filters make the network wider rather than deeper. GoogLeNet architecture consists of ’Inception Modules’ that carries the essence of the Inception family, and it is what allows to have convolution filters of various sizes rather than one. The idea is to let the model decide on which filter to use rather than the manual selection of a filter that might not perform well. This works with the parallel use of different filters on the same input with the same padding, and then concatenating the feature map from each filter to create one-big feature map which to be passed as input to the next inception module.

Inception v3 was proposed in 2016. And similar to GoogLeNet it participated in the ILSVRC challenge in 2015 and scored 1st place for that year. Inception v3 holds a very similar structure to that of GoogLeNet, however, it does have some differences that enable it to generally perform better than GoogLeNet. Inception v3 architecture hold different changes, first is special convolution filter techniques (Factorized Convolutions, Smaller Convolutions, and Asymmetric Convolutions). second, a modified version of ’Auxiliary Classifiers’. Third is the use of ’Efficient Grid Size Reduction’. And lastly implementing ’Model Regularization via Label Smoothing. Factorized convolutions are utilized to reduce the number of parameters, whilst keeping the network efficient.

Ningning Ma et al. proposed ShuffleNet v2, a CNN based on the previous ShuffleNet v1 which aimed to build an efficient and effective network with lower computation needs than other networks. With ShuffleNet v2, the authors aimed to design a network that takes into consideration the direct metrics, such as speed and memory access cost, to measure the networks computational complexity rather than an indirect metric, such as FLOPs (Floating Point Operations). FLOPs are the default measure of the performance of a model, and specifically they are units to measure how many operations are needed to run a single instance of a model.

MobileNet is a family of CNNs designed to work effectively and efficiently on mobile systems. MobileNet v2 was introduced in 2018 and it is based on the previous MobileNet v1[29] which was introduced in 2017. MobileNet v1 had two main layers forming the network, first is a ’Depth-wise Convolution’ layer which applies a single convolution filter per input channel for lightweight filtering, and the other is a ’Point-wise Convolution’ layer which consists of a 1x1 convolution filter aimed for computing linear combinations of input channels to extract new features, each of these two layers has non-linearity (ReLU).

ResNeXt was proposed in a paper in 2017, a year after achieving 1st runner-up in the ILSVRC challenge. ResNeXt is based upon ResNet, and it differs by adding a ’NeXt’ structure. This structure is a ’Cardinality’ dimension which sticks on top of the width and depth of the ResNet architecture. This dimension refers to the size of the transformations set. ResNeXt is special as it replaces the use of regular linear functions in the neurons of the layers with using a non-linear function which is a part of a ’ResNeXt Block’, the number of transformations in this block is referred to by the Cardinality, and after applying the required number of transformations, the results are aggregated together.

Wide ResNet abbreviated (WRN) is a CNN proposed in 2016 by Sergey Zagoruyko and Nikos Komodakis. It is based on the concept of residual blocks introduced in the ResNet architecture. However, it aims to be shallower with fewer layers and less training time. Wide ResNet was proposed in efforts to solve certain limitations with the ResNet architecture. ResNet has an advantage in improving performance with a large number of layers, however, adding more layers causes the reuse of diminishing feature, this affects the network by making it slow to train. The authors of WRN proposed a wider architecture (has a wider ResNet block) and lower in-depth as well as having dropout layers in between layers of residual blocks. Wider and lower in-depth means increasing the number of channel dimensions, and decreasing the number of layers.

MNASNet is a novel architecture proposed in 2018 by Mingxing Tan et al. Authors aimed to introduce a network that performs well with mobile devices as they require models that are smaller in size, yet perform effectively and efficiently. In their paper, the authors proposed three main ideas for their model, first is the introduction of a multi-objective architecture based on reinforcement learning which is used to automatically find the most suitable CNN for high performance. Second is the introduction of a novel ’Factorized Hierarchical Search Space’ structure which is used as blocks in between layers and it is aimed for more efficient use of computational resources. lastly, the authors demonstrated the effectiveness of their model with regards to other architectures aimed for mobile-use, such as MobileNet, ShuffleNet, and SqueezeNet.

There are four different sets of fine-grained Inter-species data. The first one, named 10 Monkey Species[30], is a set which includes a total of 1400 images of size 400x300 distributed among 10 different species of monkeys. These species include Alouattapalliata, Erythrocebuspatas, Cacajaocalvus, Macacafuscata, Cebuellapygmea, Cebuscapucinus, Micoargentatus, Saimirisciureus, Aotusnigriceps and Trachypithecusjohnii. The collection of these images was done with the help of search engine web scrapers. 10 Monkey Species is available on the popular data platform, Kaggle.

The second set named Fruits 360[31]. It includes data of different fruits and vegetables. This set is much bigger than 10 Money Species with more than 90,000 images of fruits and vegetables of size 100x100 and ranging from watermelons, apples and oranges to potatoes and onions and much more with 103 classes.

225 Bird Species[32] is the third data set. It includes more than 30,000 training, testing, and validation images distributed among 225 classes of birds, all of these images are of size 224x224.

Oxford 102[33] is the fourth and last data set, and it consists of 102 classes of different flowers. This data set was created by the Visual Geometry Group of Oxford. The type of flowers available in the data set are commonly available in the United Kingdom. And each class holds between 40 and 258 images. Collection of images was done using web scrappers as well as manually capturing flower images.

One of the methods to increase the performance of models is to provide more training examples [34, 35]. this method can be accomplished with data augmentation where random examples are taken from the data set, and then apply different image processing methods on these examples, such as rotation, horizontal mirroring, and magnification. This allows for the addition of many examples to the training set. The main objective of utilizing image augmentation techniques is its proven effect on improving performance [36, 37]. Having a set with more images means that the model is faced with a bigger number of samples that can highly improve performance especially with low-volume data sets.

In this phase, we train each data set independently with 29 different pre-trained models available in the Torchvision package. These models include AlexNet, VGG (11, 13 16, 19, BN-11, BN-13, BN-16, BN-19), ResNet (18, 34, 50, 101, 152), SqueezeNet (1.0, 1.1), DenseNet (121, 169, 161, 201), Inception v3, GoogLeNet, ShuffleNet v2 (0.5, 1.0), MobileNet v2, ResNeXt (50, 101), Wide ResNet (50, 101) and MNASNet (1.0). For the training and development environment, we utilized Kaggle Kernels which are in essence Jupyter[38] Notebooks as they offer ease of importing required datasets.

This phase includes the extraction of results and performance of each model with each data set and form it all together in tables in order to understand and compare how each model performs with fine-grained Inter-species data. We investigate the results to find the best possible configurations for these data with Torchvision pre-trained models, and to suggest new information that can possibly lead for the implementation of more performant models.

This last phase is actually a derivative of the third phase. Here we implement each data set with pre-trained models with a small configuration. This change is to replace the last fully connected layer in each model with the SpinalNet[39] layer. SpinalNet is a novel network that is inspired from the human somatosensory system [40]. This network consists of input row, intermediate row and output row where the intermediate row contains a small number of neurons, and each hidden layer receives a part of the input and outputs of the previous layer with input segmentation.