Exploiting CNNs for Semantic Segmentation
with Pascal VOC

For our experiment we load datasets using PyTorch’s DataLoader and processes them using number of worker processes as 4 and a prefetch factor of 2 to improve data transfer speed. The batchsize for the dataloader is set to 16 and shuffle is set True to shuffle the data randomly at each epoch. The model is trained for 50 epochs using the Adam optimizer with a learning rate of 0.005. The FCN model is defined to have 21 classes, and its weights are initialized using a custom initialization function. The model is transferred to either the CPU or GPU device depending on availability. The loss function used is either cross-entropy or weighted cross-entropy depending on a flag variable(for baseline, normal cross-entropy is used), and early stopping is implemented to monitor model performance and prevent overfitting. These parameters would remain the same for all further experiments unless stated otherwise.

The weights are initialized using Xavier uniform distribution, which is a widely used method for initializing the weights of neural networks. The bias terms are initialized using a normal distribution because Xavier initialization is not applicable for biases. Xavier initialization is a method that tries to ensure that the scale of the gradients flowing through the network remains roughly the same across layers. This can help avoid vanishing or exploding gradients during training. The method initializes the weights of the layer with random numbers drawn from a uniform distribution between the range $-(1/sqrt(n))$ and $1/sqrt(n)$, where n is the number of inputs to the node

We use the early stopping algorithm that takes a specified number of rounds of patience, usually 5 to determine when to stop training. During training, the model is evaluated on both the training and validation datasets, and the loss, accuracy, and IOU (intersection over union) are computed and saved for each epoch. The training loop iterates over the DataLoader and updates the optimizer gradients using the backpropagation algorithm. The inputs and labels are transferred to the same device as the model’s, and the output is computed using the FCN model. The output is then softmaxed, and the loss is calculated using the cross-entropy or weighted cross-entropy loss function. The loss is used to update the optimizer weights using backpropagation. Finally, the training loop computes and saves the metrics and the model’s performance is monitored using the early stopping algorithm.

The model contains 5 convolutional blocks, each consisting of a convolutional layer, batch normalization, and ReLU activation function. The FCN module has an input layer that takes a 3-channel image as input. As shown in table 1, the convolutional layers have increasing number of filters, starting from 32 and doubling with each block. Each block also has a downsampling factor of 2, resulting in a feature map with reduced height and width.After the fifth convolutional block, there are five deconvolutional blocks with the same architecture as the convolutional blocks but with increasing number of filters in the opposite direction. Each block has an upsampling factor of 2, resulting in a feature map with increased height and width.

The final layer of the module is a convolutional layer with a kernel size of 1 that reduces the number of channels to the number of output classes. This layer produces a probability map for each pixel in the input image, indicating the probability of that pixel belonging to each of the output classes.The forward pass of the FCN module takes an input image and applies the convolutional layers followed by the transpose convolutional layers to upsample the feature maps. It then passes the result through the final convolutional layer to produce the probability map for each pixel in the input image.

In summary, this experiments takes an FCN model for image segmentation using PyTorch and trains it on a dataset using the Adam optimizer with cross-entropy or weighted cross-entropy loss functions. Early stopping is used to monitor performance, and metrics are computed and saved for each epoch. The model’s weights are updated using backpropagation, and the model’s device is determined based on availability. This implementation demonstrates an effective way of training and evaluating image segmentation models using PyTorch.

One of the most effective ways to improve the performance of a deep learning model is to tune its learning rate. A constant learning rate may cause the optimization process to get stuck in local minima. Instead, we can use a learning rate scheduling technique, which gradually reduces the learning rate over time, allowing the model to explore more promising regions of the loss surface. We use the cosine annealing learning rate scheduler, which gradually reduces the learning rate according to the cosine function of the current epoch and the total number of epochs. Historically, this scheduler has been shown to work well in many computer vision tasks. The equation for this scheduler is:

where $\eta_{t}$ is the learning rate at iteration $t$, $\eta_{min}$ and $\eta_{max}$ are the minimum and maximum learning rates, $T_{cur}$ is the current epoch, and $T_{max}$ is the total number of epochs. The learning rate starts at $\eta_{max}$ and decreases to $\eta_{min}$ over the course of $T_{max}$ epochs, following a cosine annealing schedule.

Across our experiments, we use different values for $T_{max}$, keep $\eta_{min}$ as 0.

Data augmentation is a powerful technique that can significantly increase the size of the training set, thereby improving the generalization ability of the model. We will apply various transformations to the input images, such as horizontal flipping, random cropping, and random rotation. Applying various transformations to the input images during training can have several advantages in improving the performance of the model. Horizontal flipping can increase the diversity of the dataset, which can help the model generalize better to new and unseen data. Random cropping can also increase the variability of the dataset and help the model learn robust features, while reducing the dependence on the exact positioning of objects in the image. Random rotation can also help the model learn features that are invariant to rotation, which is especially useful for object detection and recognition tasks. By introducing these transformations during training, the model can learn to recognize objects and features under a variety of conditions, leading to improved accuracy and robustness.

We apply the same transformations to the corresponding labels to ensure that the label information remains consistent. We perform the mirror flip with a probability of 0.5, and rotate the image by an angle randomly sampled from the uniform distribution [-5, 5] (degrees). Additionally, we also do a centre crop to size 180 from 224 and subsequently resize back to 224. Note that we use PyTorch transforms for this, and build upon it to build custom transforms that apply the same rotation, flipping and cropping. The dataset is augmented with these modified training examples and passed on to the training pipeline, which remains same as in the previous sub-section.

In many real-world classification problems, the data may be imbalanced, i.e., some classes may have very few samples compared to others. This can lead to poor performance on the minority classes. To address this, we can use a weighted loss function, which assigns higher weights to the minority classes during training. We will implement our own balanced cross-entropy loss function, which calculates the weights for each class based on its frequency in the training set. This will help the model to focus more on the minority classes and improve their classification accuracy. The weights computed using the formula below are passed into pytorch’s cross entropy loss function.

where $w_{i}$ denotes the weight for the $i^{th}$ class. and $n_{i}$ denotes the frequency of the $i^{th}$ class in the training set across all images and pixels. These weights were computed for the 21 classes on the VOC dataset and passed to the cross-entropy loss function. The weight for the background class was 0.2973, whereas it was between 0.98-0.995 for most other classes.

Our model, Advanced-FCN is an advanced variant of the Fully Convolutional Network (FCN) architecture, specifically designed for semantic segmentation tasks. Compared to the baseline FCN, Advanced-FCN incorporates several modifications and improvements to achieve higher accuracy and faster convergence.

One major change in Advanced-FCN is the addition of multiple convolutional blocks, each comprising multiple convolutional layers, batch normalization, and ReLU activation. These blocks allow the network to learn more complex and abstract features from the input images, resulting in improved segmentation performance. Another significant change is the introduction of residual connections, which facilitate the flow of gradient information during backpropagation, allowing deeper architectures to be trained effectively.

In addition to these modifications, Advanced-FCN also utilizes skip connections, similar to the U-Net architecture. These connections allow the network to capture both local and global contextual information, enabling more accurate segmentation of objects of different sizes and shapes. Finally, Advanced-FCN also includes a final deconvolution layer that upsamples the feature maps to the original image size, producing a dense pixel-level segmentation output. Overall, these changes and enhancements in Advanced-FCN make it a powerful and efficient architecture for semantic segmentation tasks.

Table 2 describes the various layers in our model. After each layer, a batch-norm was applied. Also, the skip connections have been denoted using the plus sign in the in-channels column. The output of the immediate previous deconvolution layer is passed along with output from a previous convolution layer as indicated in the column. For data-augmentation, we use random flipping(0.5 probability), random rotation (-5 to +5) and random cropping (180 size). The learning rate used is 0.005 along with Adam optimized, and a cosine annealing scheduler with a $T_{max}$ of 30. Xavier initialization was used for the weights and the class weights described in section 4.2.3 are used to handle class imbalance.

Transfer learning is a machine learning technique that involves taking a pre-trained model, typically on a large dataset, and using it as a starting point for a new task or problem with a smaller dataset. Instead of training a new model from scratch, transfer learning involves using the features learned by the pre-trained model to extract relevant information for the new task, which can speed up the training process and potentially lead to better performance. In transfer learning, the pre-trained model is typically a deep neural network that has been trained on a large dataset, such as ImageNet for image classification tasks, and then fine-tuned on a new task using a smaller dataset. The fine-tuning process involves adjusting the weights of the pre-trained model to better fit the new data while still retaining the important features learned from the large dataset. Transfer learning has been successfully applied in a wide range of domains, including computer vision, natural language processing, and speech recognition, and has led to state-of-the-art performance on many benchmarks.

In this experiment we used multiple pre-trained models such as VGG-16, ResNet12, ResNet34 etc and settled for ResNet34 based on the IOU score. For transfer learning we took the ResNet34 pre-trained model and used it as the convolution feed-forward step. We then removed the last fully connected layer and added multiple layers of deconvolution layers so as to map the pixel predictions back into the original image and generate the segmentation map. For training the new architecture, the weights of the ResNet34 model were freezed and the training on the VOC dataset was done so as to fine tune the model. One important to keep mind is to not initialize the ResNet34 layer weights. We see a great deal of model performance improvement in terms of both IOU and pixel accuracy. The resultant model has an accuracy of 71.33% and IOU score of 0.0926, on the test dataset.

When we use transfer learning to create a segmentation model using the pre-trained weights of a ResNet34 model, we are leveraging the knowledge learned by the ResNet34 model on a large dataset (such as ImageNet) to extract meaningful features from images that can help with the segmentation task. The ResNet34 model is trained on 1.2 million image in 1000 categories, which makes it a powerful feature extractor that has learned to identify and extract high-level features from images. Also, since the features learned by the ResNet34 model are more generalizable and robust to variations in the input data, we are able to adapt the features learned by the ResNet34 model to the specific segmentation task, which leads to better accuracy and higher IOU scores.

By using its pre-trained weights, we can avoid the need to train the initial layers of the model from scratch, which can save a lot of time and computational resources. We can benefit from the knowledge and experience that has been gained from training on large datasets and makes the model more generalizable, robust to variations and give us the ability to train with smaller datasets.

The table 3 gives in the details about the layer of the transfer learning architecture.The data preprocessing and augmentation is done as described in section $4.2.2$, imbalanced class labels are taken care using the weighted loss function (refer section $4.2.3$) and the learning rate scheduling is applied (refer $4.2.1$). We take the input image and pass it to the resnet34 architecture and gets 512 channel output. We then pass these through several deconvolution layers to perform prediction. We have applied batch normalization on every deconvolution layer after the ReLU activation. This will result in increased generalization and improved training speed. The learning rate used is 0.005 along with Adam optimized, and a cosine annealing scheduler with a $T_{max}$ of 40. Xavier initialization was used for the weights and the class weights described in section 4.2.3 are used to handle class imbalance.

The U-Net architecture was proposed in the paper ”The paper ”U-Net: Convolutional Networks for Biomedical Image Segmentation” and was specifically designed for the task of biomedical image segmentation, which involves separating different structures within an image. The U-Net architecture is based on an encoder-decoder structure, where the encoder is a series of convolutional and pooling layers that reduce the spatial resolution of the input image, while increasing the number of feature maps. The decoder is a series of upsampling and convolutional layers that reconstruct the segmentation mask from the feature maps produced by the encoder. One of the key innovations of the U-Net architecture is the use of skip connections that connect the encoder and decoder layers at the same spatial resolution. These skip connections allow the decoder to access high-resolution features from the encoder, which can help with the reconstruction of the segmentation mask.The architecture is called U-Net because it has a U-shaped design. The U-Net architecture achieved state-of-the-art performance on these tasks, with significantly higher accuracy and faster convergence compared to previous methods.

The table 4 provides the details for each layer in U-Net. The data pre-processing and augmentation is done as described in section $4.2.2$, imbalanced class labels are taken care using the weighted loss function (refer section $4.2.3$) and the learning rate scheduling is applied (refer $4.2.1$). We have applied batch normalization on every layer after the ReLu activation. This will result in increased generalization and improved training speed. The skip connections are mentioned after the deconv layer, denoted as n + n, where n one of the n comes from the deconv layer while the other comes from skip connections. The model had an IOU $0.0649$ and accuracy of $72.15\%$ on test dataset.

The U-Net architecture contains a repetitive pattern of layers. It has a contracting path and an expanding path, and the two paths are connected by skip connections. In particular, the contracting path typically consists of several repetitions of two convolutional layers followed by a max pooling layer. Each convolutional layer has a small receptive field and a large number of filters to capture local and global features. The max pooling layer reduces the spatial size of the feature maps and increases their receptive field, thus making the network more robust to small image translations. The contracting path downsample the input image and extract high-level features from it. The expanding path typically consists of several repetitions of an upsampling layer, a concatenation layer that combines the feature maps from the corresponding layer in the contracting path, and two convolutional layers. The upsampling layer increases the spatial resolution of the feature maps by a factor of 2, while the concatenation layer provides high-resolution features from the contracting path to the expanding path. The convolutional layers have a small receptive field and a small number of filters to generate a fine-grained segmentation mask. The expanding path restore the spatial resolution of the feature maps and generate a segmentation mask. Overall, the U-Net architecture has a symmetric shape and is designed to produce accurate and detailed segmentations of images with complex and irregular shapes. The repetitive pattern of layers allows the network to capture features at multiple scales and combine them to produce a robust and precise segmentation. Also, the learning rate used is 0.005 along with Adam optimized, and a cosine annealing scheduler with a $T_{max}$ of 40.

The model’s performance was evaluated on the test set with the following results after rounding off the values up to 4 decimal points: the Test Loss was 2.3946, Test IoU was 0.0529, and Test Pixel accuracy was 72.86%. During training, the model was stopped early at epoch 13, where the best loss achieved was 2.3726, best accuracy was 75.06%, and best IoU score was 0.0551. The best iteration was 8. These results suggest that the model has some room for improvement, especially in terms of IoU score, which indicates how well the model is able to predict the boundary of objects in the image.

The test loss is 2.4243, the test IoU is 0.0585, and the test pixel accuracy is 69.88%. The training was stopped early at epoch 37, where the best validation loss achieved was 2.4021, the best validation accuracy achieved was 72.11%, and the best validation IoU score achieved was 0.0640, with the best iteration being 32. These values indicate that the model did not perform very well on the test set, but still performed better than the previous models, as the test loss is relatively high, and the test IoU and pixel accuracy are relatively low.

The test performance of this model indicates that the Test Loss is 2.6869, the Test IoU is 0.0596, and the Test Pixel accuracy is 68.98%. The values have been rounded to 4 decimal places. The model was trained for 15 epochs, and the training was stopped early. The best loss achieved during training was 2.6576, the best accuracy was 70.74%, and the best IoU score was 0.0641, and these were achieved at the 10th iteration. Overall, the model’s performance suggests that it needs further improvement in order to achieve higher accuracy and better performance.

The test performance of the model is as follows: the test loss is 2.6843, the test IoU (Intersection over Union) is 0.0602, and the test pixel accuracy is 67.20%. It is worth noting that the training was stopped early at epoch 17, and the model achieved its best loss, accuracy, and IoU score at epoch 12. The best loss achieved during training was 2.6433, the best accuracy was 71.01%, and the best IoU score was 0.0688. All values have been rounded to 4 decimal places.

The test performance of the model was evaluated with three metrics. The test loss was found to be 2.6169, the test IoU (Intersection over Union) was 0.0926, and the test pixel accuracy was 71.34%. The values have been rounded off to 4 decimal places. The training process was stopped early at the 14th epoch. The best values of validation loss, accuracy and IoU score were 2.5667, 75.04%, and 0.1060, respectively, achieved at the 9th epoch.

The test performance of the model is reported as follows: the test loss is 2.6570, the test Intersection over Union (IoU) is 0.0649 and the test pixel accuracy is 72.15%. These values have been rounded off to 4 decimal places. The training process was stopped early at epoch 11. For validation the best loss achieved during training was 2.6111, the best pixel accuracy was 75.26%, and the best IoU score was 0.0710. These values were achieved at iteration 6.

Adjusting the learning rate with the number of epochs during training can have several benefits. Firstly, it helps the model converge faster and achieve better accuracy by reducing the learning rate as the training progresses. This can prevent overshooting of the optimal solution and stabilize the training process. Additionally, it can help the model generalize better and avoid overfitting to the training data by allowing the learning rate to decrease gradually. However, there are also potential drawbacks to this approach. For example, if the learning rate is decreased too quickly, the model may not have enough iterations to converge to the optimal solution. On the other hand, if the learning rate is decreased too slowly, the model may take longer to converge or even get stuck in a suboptimal solution.

Using the scheduler - cosine annealing helps the model gradually step through the learning rate to improve the updates applied to the parameters. This, in theory, should improve the direct performance metrics for classification like the loss and accuracy. But the same can not be said about the indirect metrics of performance - like IoU scores. This is re-affirmed my the results, where we observe that the test loss decreased from 2.4114 of baseline to 2.3946. The accuracy also improved from 71.31% to 72.86%. Interestingly, the IoU score improved only marginally from 0.0527 to 0.0529. This reflects that most of the improvement in performance is due to the overfitting on the background in dataset rather than a overall improvement in segmentation. Additionally, the model’s loss curve indicates that the model now takes a few more epochs to converge, which is understandable as the stepping when clubbed with early stopping’s patience mechanism results in abrupt-slight increases in performance. Also, from the segmentation map, we observe that similar to the baseline model, this model fails to predict the class of the object correctly, while it is still able to identify the boundaries correctly to an extent, thereby indicating some sense of boundary feature being detected by the convolution layers of the model. The model is also not robust to changes in dataset - like slight rotations, mirror image or a crop. This is further improved in the next stage.

In the figures 6(a), 5 and 6(b), the plots showcase the performance using cosine learning scheduler during training and validation. The graphs for training and validation loss demonstrate that the model’s performance improves quickly at the start and then stabilizes as it approaches convergence. While the training loss continues to decrease slightly during later epochs, the validation loss stays constant, indicating that the model may overfit if it continues training beyond the convergence point. Furthermore, the charts depicting the average IOU score and pixel accuracy show that the model’s performance increases as the number of epochs increases.

Augmenting the dataset by applying transformations to the input images has several benefits. Firstly, it can increase the size of the dataset, which can help to prevent overfitting and improve the generalization of the model. Secondly, it can help to make the model more robust to variations in the input data, by exposing it to a wider range of variations that it may encounter during testing. Additionally, it can help to improve the accuracy of the model, by providing it with more diverse examples to learn from. However, there are also some potential drawbacks to data augmentation. Firstly, if the transformations are too extreme, they may distort the original image and introduce unrealistic features, which could lead to poor performance on real-world data. Additionally, data augmentation can be computationally expensive, as it requires generating multiple variations of each image during training. Finally, it can be difficult to choose appropriate transformations that are relevant to the specific task and dataset, which may require domain knowledge or trial-and-error experimentation.

To address the robustness and generalization properties of our model, we augmented our dataset with images and labels with correspondingly-same the rotations, flips and crops as described in detail in the method section. Applying these transformations is expected to improve the segmentation maps and the IoU scores as well. This is confirmed by the fact that IoU score has improved from 0.0529 to 0.0585 on the test set, which is a significant jump! But at the same time, our accuracy has decreased to 69.88% and the test loss increased to 2.4242. This is due to the reduced effect of training on background which comprises the bulk in the accuracy metric. Additionally, we observe from the segmentation map of the bus that the model can now detect the class correctly and is able to identify complex boundaries in the bus. While the segmentation map is still not satisfactory, and can not segment the entire bus, it still demonstrates better ability to classify and identify boundaries when compared to the baseline model and the model with cosine annealing scheduler, of which both had low IoU and poor segment maps. It would also be better at segmenting images which are rotated, cropped or flipped (horizontally) without losing on the quality of segmentation map.

The figures 8, 9(a) and 9(b) demonstrate the performance of the model with dataset augmentation during training and validation. The curves for training and validation loss show that the model’s performance improves quickly initially, but then reaches a point where it is no longer improving significantly. Although the training loss decreases slightly further on, the validation loss remains more or less constant, indicating that additional training may cause the model to overfit and perform poorly on new data. Additionally, the graphs of average IOU score and pixel accuracy over time show that the model’s performance improves as the number of training epochs increases.

The benefits of using imbalanced class weights for cross-entropy loss are that it allows the network to pay more attention to the minority classes, thereby reducing the bias towards the majority class. This can lead to improved performance on the minority classes and better overall accuracy. However, there are also some potential drawbacks to using imbalanced class weights. Using imbalanced class weights can lead to overfitting on the minority classes, especially if the dataset is small. This can result in poor performance on new, unseen data. Finally, using imbalanced class weights can increase the training time and computational resources needed to train the network, as the loss function needs to be computed for each example and the gradients need to be calculated accordingly.

In the previous implementations it has been repeatedly pointed out that our model overfits on background and is unable to segment the rare classes. As described earlier, this has been tackled using imbalance weights in the cross entropy loss. These weights ensure that our loss function penalizes incorrect predictions on rare classes more, thereby forcing model to learn them and avoid predicting everything as background. Intuitively, this is expected to boost IoU score, but would also lead to higher loss. This is re-affirmed by our observations that the test IoU score has improved to 0.0596, whereas the test accuracy has dropped to 68.98%. As explained previously , due to less overfitting on background, this accuracy is lower than our baseline and the other previous models. Additionally, our segmentation map displays a great performance by our model relative to the previous models (baseline, 4a, 4b) which tend to overfit on the background or the major classes.

The figures 11, 12(a) and 12(b) depict the training and validation progress. The curves of training and validation loss indicate that the model’s performance converges relatively rapidly, and then reaches a plateau. While the training loss continues to slightly decrease in later epochs, the validation loss remains approximately constant, implying that further training may cause the model to overfit and perform poorly on new data. Furthermore, the graphs of average IOU score and pixel accuracy across epochs show that the model’s performance increases with the number of epochs, albeit pixel accuracy converges faster. This could be due to an imbalanced distribution of class labels in the training data, resulting in the model learning the background labels much quicker than other class labels.

We build a custom architecture which we call Advanced FCN as described in 4.3.1. The results show improvement over the baseline architecture with an improved IOU score of 0.0602. The same is reflected in the segmentation maps where the model is able to segment the horse(partly) and the person well. This shows that adding more layers to the baseline FCN along with adding some skip connections significantly impacts the performance.

Since this model uses more convolutional and batch normalization layers which increases the depth and complexity of the model, it helps the model learn more complex features and patterns in the input data. The use of skip connections between the encoder and decoder allows the model to retain more spatial information from the input data. This helps the model generate more accurate segmentations. The skip connections allow the decoder to access information from the encoder at multiple levels of abstraction, which can help preserve finer details and improve segmentation accuracy. This model also uses a smaller kernel size for the last convolutional layer which in turn helps in reducing the number of parameters that can help prevent overfitting of the model. But, as we have more layers and parameters in this architecture, the model would take more time and resources to train.

In addition to the above described architecture of our own, we also experimented with different number of layers and used different activation functions like softmax which did not give a better IOU score or accuracy than the ReLU activation function which seemed to perform best among all.

Figures 14, 15(b) and 15(a) illustrate the training and validation performance our custom Advanced-FCN model. The training and validation loss curves show that the model’s performance stabilizes relatively quickly, and then improves at a slower rate. The training loss continues to decrease slightly in later epochs, while the validation loss remains relatively constant. This suggests that further training may cause the model to overfit to the training data, and degrade its ability to generalize to new data. Additionally, the IOU (Intersection over Union) score and pixel accuracy graphs demonstrate how the model’s performance improves over time. As the number of epochs increases, both metrics improve, although pixel accuracy converges more quickly. This is likely due to the fact that the model is learning the background labels much faster than the other class labels, which leads to an imbalance in the training data.

In this experiment we have used transfer learning which results in faster training time, improved accuracy and IOU score, better generalization etc. On the other hand, transfer learning also brings in the biases of the pre-trained model and the data they were trained on. There is also the problem of limited transferability, not all pre-trained models can be transferred to all the tasks and we need to check the model compatibility. More over sometimes pre-trained models could lead to over-fitting.

For the pre-trained model resnet34 was used that resulted in reduced training time, improved pixel accuracy and IOU score, improved generalization. Some of the challenges using this architecture are large memory requirements and complex architecture. Based on these factors we got an accuracy of $71.33\%$ and IOU score of $0.092$ on the test dataset, which is the best model performance among all the experiments, surpassing the baseline, improved baseline, advanced-fcn and U-Net.

The increment in performance can be attributed to various factors.First, resnet-34 is a relatively deep neural network architecture, which allows it to capture complex features and patterns in the input data. This is especially important for semantic segmentation, which involves understanding and segmenting the different objects and regions in an image.Second, resnet-34 uses residual connections, which helps to reduce the vanishing gradient problem and improve the flow of gradients during training. This leads to better regularization and can help prevent overfitting, which is important in semantic segmentation where the model needs to generalize well to new images. Third, resnet-34 has been pre-trained on a large and diverse dataset (ImageNet), which allows it to capture generic features that can be useful for a variety of computer vision tasks. These are some of the important factors influencing the performance we got.

The figure 17 shows the training and validation loss across the epochs. We have used early stopping with a patience of $5$. We observe that the training and validation loss converges fairly quickly, and then becomes almost constant. In the later epochs, training loss continues to go down where as the validation loss starts to rise indicating that more training the model would lead to over-fitting. The figure 18(a) and figure 18(b) shows the average IOU score and pixel accuracy across the epochs respectively. We can observe that as number of epochs increase the IOU score as well as the pixel accuracy increases but the rate of convergence of the pixel accuracy is higher when compared to IOU score. This could be due to the fact that the data has imbalance classes and that the model learns the background labels much quicker than other class labels.

For this experiment the model architecture used was U-Net. U-NET is a powerful and popular neural network architecture for semantic segmentation, but like any other architecture, it has its own set of advantages and disadvantages. Some of the advantages are: First, it is designed to use memory and computational resources efficiently. Its efficient memory usage makes it a good choice for applications with limited resources. Second, it can handle variations in input size, which makes it useful for applications where images vary in size.Third, it preservation spatial information, due to the presence of skip connections and improve segmentation accuracy. On the other hand, some of the disadvantages are: that it has limited field of view, which is a result of downsampling operations and can result in missed object details. Second, its sensitivity to image variations like variations in lighting, rotation, and scale, which can make it less accurate in certain situations.Third, U-NET can struggle with class imbalance in the training data, where one class may be significantly underrepresented compared to others.

For the U-Net model, we got a pixel accuracy of $72.15\%$ and IOU score of $0.065$ on the test dataset, which is the second best model performance among the experiments. We also see a better segmentation map (fig 22(b)) in this case. The increase in performance can be attributed to the factors like skip connections, which preserved the spatial information, deeper network when compared to other models (except resnet34) and model compatibility to segmentation task, since the U-Net was specifically designed for medical data segmentation. The U-Net came in second to resnet34 architecture due to the factors like insufficient training data to capture more generic features and shallower network when compared to resnet34. It was better than the baseline and the improved baseline model performance in terms of IOU score.

The figure 20 shows the training and validation loss across the epochs. We have used early stopping with a patience of $5$. We observe that the training and validation loss converges fairly quickly, and then keeps decreasing slowly. In the later epochs, training loss continues to go down where as the validation loss starts to rise indicating that more training the model would lead to over-fitting. The figure 21(a) and figure 21(b) shows the average IOU score and pixel accuracy across the epochs respectively. The performance plots follow the same trend as the loss plot though in opposite direction. We can observe that as number of epochs increase the IOU score as well as the pixel accuracy increases and the rate of convergence of the pixel accuracy and IOU score are comparable in this case. This could be due to weighted loss which tried to take tackle the data’s class imbalance problem.