A Semi-automated Statistical Algorithm for Object Separation

This methodology is used for images, where the foreground is dominant over the background. In these type of images, the focus is on separating the background from the objects, as it will be distinct, having pixel values at the two ends of histogram. This methodology is also favorable for simple background at the ends of the histogram, having regions of pixel values, which are well separated.

Methodology-I is Arora et al.’s algorithm 30 with active usage of $\kappa$ parameter. Unlike Arora et al., where the value of $\kappa_{1}=\kappa_{2}=\kappa=1$, in methodology-$I$, $\kappa_{1}$ and $\kappa_{2}$ are independent of each other. In other words, methodology-$I$ incorporates the suggestion of 30 on using $\kappa_{1}$ and $\kappa_{2}$, in place of $\kappa$, and varying $\kappa_{1}$ and $\kappa_{2}$ to obtain optimal or desired segmentation. The main objective of this paper is object separation through image segmentation. Thus, methodology-$I$ applies Arora et al. algorithm by varying $\kappa_{1}$ and $\kappa_{2}$ actively for obtaining appropriate thresholds to carry out segmentation, and finally extracting the desired object.

The proposed procedure takes into account the fact that important information lie towards the weighted mean of the histogram. Furthermore, it also incorporates the fact that human eye is more sensitive to the gray pixel values as compared to black and white ones. Hence, the aim is to have wider segment size at the ends of histogram of a gray scale image 30 . This will separate the background from the foreground. In some images, a very small part of the background may have pixel values near to the foreground or object pixel values. Therefore, the segmented region in the histogram becomes finer as it reaches its weighted mean, to separate out these background pixel values from the object.

The thresholds for the histogram are defined by:

and

where $i\in[1,\frac{N-1}{2}]$, $j\in[(\frac{N+1}{2}),N]$.

$N$ is the number of thresholds, which is always odd. It is to be noted that initial values of $a$ and $b$, or $T_{0}$ and $T_{N+1}$ are 0 and 255 repectively, because they include the complete image. The segmented values are the weighted mean of the histogram between the thresholds, and are defined as,

Therefore, the segmented $F_{Y}$ defined as $F_{YS}$ can be written as:

The images for which methodology II is applied have dominant backgrounds. This methodology is found to be best suited for complex background. It is due to the fact that the object has to be separated out from the background, which itself is complex.

As the background is complex, the methodology incorporates the fact that important information lies at the end of the histogram plot. This approach segments the background into different regions to extract the desired objects. Hence, the methodology has finer segment size at the ends of histogram, while having wider segment size at the weighted mean. The wider segment size at the weighted mean of histogram is in conformity of not segmenting the object but the background.

The thresholds for the histogram are defined as:

and

The initial values of $T_{i+1}$ and $T_{j-1}$ are:

where $i\in[0,\frac{N-1}{2}]$, $j\in[(\frac{N-1}{2}),N+1]$.

$N$ is the number of thresholds. The segmented values are the weighted means of the histogram between the thresholds:

Therefore, segmented $F_{Y}\equiv F_{Y_{S}}$ is given by:

For the purpose of illustration, we explicitly demonstrate the histograms of images, in which the two methodologies are successfully applied at threshold level 5 for different sets of $\kappa_{1}$ and $\kappa_{2}$ values. The image histograms in $Fig.\,1$ and $Fig.\,2$ clearly demostrate the non$-$overlapping Gaussian distribution corresponding to different objects. The corresponding impulse functions extracted through the present procedure given in $Fig.\,1$ and $Fig.\,2$ show the mapping of Gaussian distributions into respective $\delta-$functions.

To assess the quality of the segmentation methodologies and compare them with Otsu’s multithreshold algorithm objectively, Mean Structural Similarity Index Measure (MSSIM) 31 is used in this paper. MSSIM assesses the quality of segmented image by calculating and comparing the structural information degradation w.r.t the original image. This evaluation parameter is considered because every object in an image has a structural form, easily perceived by human eye. Thus, the segmentation algorithm retaining the maximum structural features of the original image is most suitable for extracting object segments and hence, performing object separation. The formulae for finding MSSIM are defined as,

where $\mu$ is the mean; $\sigma$ is the standard deviation; $p$ and $q$ are the window sizes of the original and the reconstructed images which are typically $8\times 8$; $K_{1}$ and $K_{2}$ are the constants having values $0.01$ and $0.03$, respectively; $M$ is the total number of windows. It has be noted that MSSIM value ranges from 0 to 1 representing no similarity and complete similarity with the original image, respectively.

The results in $Table\,1$ show the MSSIM values of each segmentation algorithm i.e. methodology-I, methodology-II and Otsu’s multithreshold algorithm, at different number of segments. It can be seen from $Table\,1$ that MSSIM of Otsu’s method is substantially less in comparison to the two proposed methodologies in the paper, with minimum and maximum MSSIM ranging in 0.30’s and 0.80’s, respectively, for the given test images. The superiority of the two algorithms described in this paper can be seen from the fact that no MSSIM value at any number of segments and for any test images, lies below 0.84. This superiority can be attributed to the efficacy of the proposed algorithms to effectively separate the non-overlapping or minimally overlapping distributions. As mentioned earlier, images consist of objects with distinct set of intensity values, which results in non-overlapping or minimally overlapping distributions in the histogram of image. Hence, the structure of the original image is maximally preserved in the segmented image through proposed methodologies, while also retaining the objects’ structure at certain number of segments. On the other hand, as evident from MSSIM values from Otsu’s method of maximizing the class variance, separate the distributions with some part of other distributions resulting in the distortion of structural information.

Between the two methodologies, it can be noted that a particular methodology dominates the other, in terms of the MSSIM value for all segmentations with number of segments greater than two. When there are just two segments, both the methodologies would result in the same segmented regions because the formulae for calculating thresholds is same for both methodologies. The difference lies in the part of the histogram chosen for applying the algorithm recursively. The choice of methodology to be used for an image can somewhat be indicated by the MSSIM value in $Table\,1$. The gradient of the two MSSIM values is an important factor in objectively choosing the appropriate methodology for segmenting an image. However, the absolute MSSIM value or gradient is only suggestive. For example, in images ’Airplane’, ’Eagle’, and ’Coins’ (see $Fig.\,3-4,7$), methodology-I is used through subjective observations. The MSSIMs of ’Eagle’ and ’Coins’ have higher values than the other methodology, but not with a substantial gradient. However, MSSIMs of ’Airplane’ suggest the usage of methodology-II, again with small gradient. The gradient is too small to indicate a preference towards a methodology. Hence, subject evaluation by the user is the most reliable method. On the other hand, methodology-II is used in images ’House’ and ’Coin’ (see $Fig.\,5-6$). In this case, the MSSIMs and their gradients provide stronger inclination towards methodology-II. However, it is still advisable to trust a subjective evaluation by the user, especially if MSSIM difference is not sharp or substantial, because the definition of the object in an image is highly dependent on the user and the application.

At last, MSSIM does not provide any information about the appropriate threshold required to separate the object completely. This is because MSSIM evaluates the structure of the image, not explicitly the structure of the object to be separated. Thus, the MSSIM increases with the increase in the number of segments. Although, the proposed methodologies are designed to overcome the over-segmentation issue i.e. even after over segmentation, the desired object can be extracted, it would be futile in terms of time and processing to choose a higher number of thresholds or segments than the requirement. In addition, one cannot determine through the MSSIM, the minimum threshold value at which the problem of under segmentation is overcome for the desired object to be separated.

As mentioned in the previous section, MSSIM cannot be used for choosing appropriate thresholds. Also, the type of methodology to be used cannot be reliably determined by MSSIM where gradient between the methodologies are low. Therefore, following method is suggested for finding the right methodology, the required threshold, and the extracting object for a class of images.
$Step\,1$: Segment the image with the number of thresholds starting from 1 and increased by 2 at each iteration.
$Step\,2$: Compare the segmented image of methodology-I and methodology-II, subjectively. If the object to be extracted exclusively lies or seems to lie in one methodology, then choose that methodology.
$Step\,3$: Repeat steps 1-2 till the desired object seems to lie in a segment value or combination of segmented values which represents the desired object.

However, the above steps are only required to be followed for one image from a particular class. For rest of the images of the same class, the value of the threshold would remain the same. Moreover, the above steps are only suggestive, and with experience user would be able to correctly guess the required threshold, $\kappa$ values, and combination of segments to be extracted to obtain the object. Furthermore, the methodologies can also be chosen by observing the approximate intensity values of the desired object. If they lie towards the weighted mean, then methodology-I should be chosen, otherwise methodology-II should be utilized.

The results are summarized in $Fig.\,3-8$. In $Fig.\,3-7$, column 1 represents the original image, while column 6 contains the extracted object. Column 3 and 5 are the edges of the segmented Y component ($shown\,in\,Column\,2$) and Y component of the extracted object using Canny edge detection algorithm for comparison. It can be easily seen that the edges in column 3 are focusing on the overall segmented gray scale image, while on the other hand, column 5 is showing the edges within the object which has been extracted, providing no information about the background. Hence, the separated object can be easily perceived from column 5.

Canny edge detection 32 is optimal compared to other edge detection techniques like Sobel, Prewitt and Roberts – to name a few. Apart from being optimal, Canny edge detection method is also robust against noise which is suitable for noisy test images used to evaluate the algorithms. However, in this paper, the task of edge detection is to visualize the effectiveness of the segmentation process to extract the desired object. It does not have any involvement in thresholding, segmentation or/and object separation.

For each image in $Fig.\,3-7$, different threshold levels are taken depending upon the requirement. If the image background lies towards the weighted mean of the histogram, and overlaps with some part of the object to be extracted, then higher number of threshold levels are required to separate the two, more effectively with the help of methodology-I. On the other hand, if the background pixels lie at the ends of the histogram, mixed with some part of the object, then higher number of thresholds will be required using methodology-II.

$Table\,2$ states the number of thresholds, segmented values and extracted values for the results shown in $Fig.\,3-7$. Looking at the segmented values of the two proposed methodologies and Otsu’s method, all the algorithms’ inclination and the location towards different parts of histogram can be clearly observed. Comparison of segmented values shows that methodology-I converges towards the weighted mean of histogram, while methodology-II diverges to the ends of the histogram. Otsu’s method follows its own approach of maximizing class variance, neither converging nor diverging towards any part of the histogram. Segmented values of $Table\,2$ are displayed in $Fig.\,8$ for each algorithm, along with the original histogram of Y component for the test images. Moreover, the convergence and divergence observed in $Table\,2$ can easily be seen in $Fig.\,8$.

In addition, $Table\,2$ demonstrates the need for minimal number of threshold required to extract the desired object. It can be seen through the values extracted from the segmented values that the object would not have been separated from the rest of the image if the threshold chosen was less than the current threshold. As an example, for the image ’Airplane’, the extracted values are 62, 152, 200, 241 from the segmented values 62, 152, 170, 173, 200, 241 ($Table\,2$). The threshold level 5 ensures that the pixel values (152, 170) and (173, 200), which would be represented by single value at threshold 3 in methodology-I, are making a distinction between the object and the background pixel i.e., segmented pixel values 152 and 200 represent the object, while the values 170 and 173 represent the background. In the image ’Eagle’, only segmented pixel value 59 is required to separate the object from the background, and therefore threshold level 3 is sufficient to do the job. The choice of number of threshold levels can be better achieved by the psycho$-$visual discretion of user, hence proving the utility of a semi$-$automated approach. For the case, when methodology-II is adopted for images ’House’ and ’Coin’, threshold level 5 is performing the similar task as it did for images using methodology-I but in a converse manner (for the end of the histogram pixels). Having a higher number of threshold levels than required will not affect the object separation process, but a smaller number of threshold levels will. This is because higher number of thresholds will result in over segmentation of object or background, which can be negated by taking or rejecting more segment values respectively. In the case of less threshold levels, under segmentation would occur which may not be able to separate background and object pixels.

In some extracted images, a very small part of the background may appear with the separated object or some part of the desired object may not be extracted. This deficiency can be removed by varying the values of $\kappa_{1}$ and $\kappa_{2}$. The increase and decrease of $\kappa_{1}$ and $\kappa_{2}$ values will result in finer and broader segment size, respectively. It will help in separating out the background and the foreground pixels, which were overlapping in the initial case. For the results in $Fig.\,9$, $\kappa_{1}=\kappa_{2}=\kappa$. $Fig.\,9$ shows the original and extracted images at $\kappa$ values of 0.25, 0.5, 1 and 2. It can be seen that pixel values of the background and the object are segregating or merging with a change in $\kappa$ value. Amongst all the results displayed in $Fig.\,9$, objects in the images ’Aeroplane’, ’Coin’ and ’Coins’ are most efficiently extracted at $\kappa=1$, ’Eagle’ at $\kappa=0.5$, and ’House’ at $\kappa=2$. In the image House, it is seen that some part of the wall is of sky blue color, matching the color of the background. In this case, the background cannot be completely eliminated, but overlap can be minimized by varying the value of $\kappa$ between 1 to 2. The results can be further refined to get the desired object by taking different values of $\kappa_{1}$ and $\kappa_{2}$, depending on the skewness of the histogram. It is worth mentioning that skewness parameter $\kappa$ can be successfully tuned by the user, providing freedom to isolate the desired object.

1. 1.

   The number of thresholds is always odd which also implies that the number of segmented regions is always even i.e. one more than the number of thresholds. However, there might be a situation where, for desired segmentation, even number of threshold values are required. Although the proposed algorithms are incapable of achieving even number of thresholds, there are two ways in which proposed methodologies overcome the above problem. First, the variation of either $\kappa_{1}$ or $\kappa_{2}$, or both, can result in shifting the threshold values, which would bring the desired object values into single or distinct segments, in place of overlapping ones. Second, the number of thresholds can be increased. As the aim of this paper is to separate the specific object from the rest of the image, over-thresholding is not a concern. This is because an object represented by one segment, or more than one, would yield the same result. The only change would be in the number of segmented values extracted to isolate the object. The focus is on extracting the desired object. It may or may not be achieved by optimally segmenting the image.

2. 2.

   No method to obtain optimized parametric values is suggested in the paper because the statistical characteristics of an object to be extracted vary drastically among the distinct set of images. However, for a particular set of images, the fixed or optimized value of the parameters can be found manually by the user for one image, and then the same values can be utilized for the complete set in an automated manner. Manual intervention to define the best parametric values results in the algorithm being semi-automated. But, the algorithm can be completely automated for images belonging to certain areas, if the parameter values are manually initialized.

3. 3.

   $\kappa$ is an important parameter which can be used to refine segmentation and adjust thresholds. Its comprehensive role needs further exploration that lies beyond the scope of the paper. The paper provides an enabling framework for object separation requiring an active role of $\kappa$, but it would be difficult to claim that there is a generalized formula to find optimal value $\kappa$ for all classes of images. Hence, a separate study is required to investigate it. However, it is believed that for a specific class of images there can be a method to find an optimal $\kappa$ and therefore it is suggested to different field users to calculate their own optimal $\kappa$ formula. It would be difficult for the authors of this paper to survey the characteristics of every class of images, and find the formula for an optimal $\kappa$. Another argument against finding optimal $\kappa$ is the lack of knowledge whether a single $\kappa$ value is enough for an image, or there can also be different $\kappa$ values at each threshold levels.

4. 4.

   Although, the proposed methodologies are able to separate out the desired object from the image, there are still some small artifacts. These can be removed using three methods: (1) by segmenting the color spaces, (2) varying the $\kappa$ values, and (3) using some preprocessing or postprocessing operations. None of the previous mentioned methods are used in the results to remove the artifacts because choosing any one would intrinsically mean that others are inferior methods. Also, as stated earlier, choosing the appropriate $\kappa$ would automatically solve this problem.