Occlusion-Resistant Instance Segmentation of Piglets in Farrowing Pens Using Center Clustering Network

The data was from a previously published study (Ceballos et al, 2020) conducted at the Swine Teaching and Research Center, University of Pennsylvania School of Veterinary Medicine, United States, where 39 lactating sows and their offspring (Line 241; DNA Genetics, Columbus, NE) were initially used. All animal procedures in the previous study were approved by the University of Pennsylvania’s Institutional Animal Care and Use Committee (Protocol #804656). Six top-view videos collected from 6:00 to 18:00 in June 2018 were selected for the development of the method in the current study (Tables 1). The six videos, with a frame rate of 7 FPS and a resolution of 1024×768 pixels, were captured by overhead cameras located 2 m above the farrowing crates. Six datasets, containing 4,600 images in total, were extracted from the six videos and labeled for piglet centers, individual piglet masks, and sow masks (Huang et al, 2021b). Datasets 1, 2, and 3 contained data from pens with half-open farrowing crates, and Datasets 4, 5, and 6 contained data from pens with closed crates (Fig. 2). Each video included one sow, and the number of visible piglets varied from 7 to 17 (an average of 11.4).

Two types of partial occlusion on piglets were defined: body-separated occlusion (BSO) and part-missing occlusion (PMO) (Huang et al, 2021b). In BSO, the occlusions (e.g., occlusions from farrowing crates) separated a target into two or more parts, whereas in PMO, a terminal part of a target (e.g., the head of a piglet) was occluded. The statistical summary showed that on an average, 3.7 piglets were affected by BSO, 4.2 piglets were affected by PMO, and 1.7 piglets were affected by both BSO and PMO per image. BSO occurred in 95.4% of the images, and PMO occurred in 92.3% of the images. Piglets were completely occluded (e.g., occluded by the sow, sow feeder, and farrowing crates) in 19.1% of the images. Overall, 98.7% of the images had at least one type of partial occlusion (BSO or PMO).

Overview. The flow of CClusnet-Inseg is as follows: First, we feed an input image to an encoder–decoder network (Figs. 3b and 4); the network generates a semantic segmentation map (Fig. 3c) and a center offset vector map (Fig. 3d) as output. For each pixel belonging to a piglet, a center point is predicted using its corresponding offset vector, and thus we obtain scatter center points (Fig. 3e). In the second stage, we filter these predicted scatter center points to eliminate outliers (Eq. 6 in (Huang et al, 2021b)) and then we cluster them into different groups through the DBSCAN algorithm (Ester et al, 1996). After the clustering, a C2M step traces these clustered center points back to their original pixels, and original pixels whose center points are in the same group form a mask of an instance. As some centers are filtered or clustered as noise in the DBSCAN algorithm, we perform a RC2M step to further cluster these centers and trace them back to their original pixels in the same manner as in the C2M step.

The mean average precision (mAP), the most commonly used metric for instance segmentation, is used as the performance metric. To demonstrate and evaluate the performance of CClusnet-Inseg, we 1) perform an ablation study to determine the contributions and effects of each module—DBSCAN (compared with mean-shift as adopted in CClusnet-Count) and RC2M, and 2) compare it with six state-of-the-art instance segmentation methods—Mask R-CNN (He et al, 2017), SOLOv2 (Wang et al, 2020b), Blendmask (Chen et al, 2020), CondInst (Tian et al, 2020), YOLACT++ (Bolya et al, 2020), and HoughNet (Samet et al, 2020). The same training set and test set are used in all of the evaluations.

To demonstrate the potential applications of our method in animal monitoring, we apply the instance segmentation of piglets directly in the implementation of multi-object tracking in farrowing pens, which provides a solid foundation for subsequent in-depth animal monitoring tasks such as the characterization of the spatiotemporal distribution patterns of piglets. We select a half-minute video clip, from the video used in the generation of Dataset 1 and in which piglets and the sow are active, as a sample video. CClusnet-Inseg detects each frame in the video clip and provides the instance segmentation results as output. For pairing objects between two consecutive frames, the Intersection-over-Union (IoU) is calculated and used as the single metric to determine whether two instances are the same. We determine the object pair that had the maximum IoU among unpaired objects between two consecutive frames and assign the same ID to these objects. Then, we repeat the process for the remaining unpaired objects until no object pairs can be found. The unpaired new object in the current frame is assigned a new ID, and the unpaired old object in the previous frame is removed.

From the multi-object tracking based on instance segmentation, we further extract six types of information—individual animal movement (accumulated movement in the video), trajectory, average speed, body pixel size, spatial usage, and spatial distribution. When generating animal movement, trajectory, and average speed, the predicted object center—a conjunct output of CClusnet-Inseg—is used to represent the location of an object. This predicted object center is valid even when an object is partially occluded. When calculating the body pixel size, the mean of the top five maximum mask areas of an object is used to exclude the case in which an object is partially occluded. To represent the spatial distribution, we generate a heat map by accumulating all of the pixels that an object had occupied in the video over its entire duration.

All of the images from Datasets $1-6$ were used; they were randomly divided such that $60\%$ of the images were used as a training set, $20\%$ as a validation set, and $20\%$ as a test set. Thus, a training set of 2,760 images was constructed to train the model, which was validated and tested on a validation set and a test set that contained 920 images each. Image augmentations, such as rotation, flips, Gaussian blur, and pseudo-occlusion generation, were applied simultaneously to the training set during the training. The optimizer Adam (Kingma and Ba, 2014) with a learning rate of $1\times 10^{-4}$ was chosen for training. The weight $\lambda$ was set to $0.5$ (Eq. 4). The epoch was set to 300 iterations, and the batch size was set to 8. The three hyperparameters-threshold $t$ (Eq. 6 in Huang et al (2021b)), radius $\epsilon$, and the minimum number of points in DBSCAN-were set to 20, 2.5, and 50, respectively, based on the best results from the validation set. All experiments were conducted using Pytorch on a single NVIDIA GeForce RTX 3090 GPU.

To demonstrate the improvement and influence of each extension (i.e., DBSCAN and RC2M), an ablation study was conducted (Table 2). The results showed that the accuracy of the model using DBSCAN (82.8 mAP in Model 2) was improved than that of the model using mean-shift (81.1 mAP in Model 1), whereas the clustering speed of the model using DBSCAN (0.152 s/image) was approximately 20 times as fast as that of the model using mean-shift (2.846 s/image), leading to a faster total inference speed in the model using DBSCAN (0.229 s/image as against 2.939 s/image in the model using mean-shift). This difference in speed was mainly due to the algorithm itself, i.e., the time complexity of the DBSCAN algorithm is $O(n\log n)$, which is less than that of the mean-shift algorithm, which is $O\left(n^{2}\right)$. The RC2M improved the instance segmentation accuracy by 1.3 mAP (an improvement from an mAP of 82.8 in Model 2 to 84.1 in Model 3) because it reused the unclassified center points. However, this step resulted in additional computations, and therefore, the inference time increased slightly by $2.2\%$. Thus, the use of the RC2M involves a trade-off between speed and accuracy. Finally, when all these extensions were included, CClusnet-Inseg achieved an mAP of 84.1. Therefore, the inclusion of all extensions was used as the default setting for our method in the subsequent experiments.

The results of the performance comparison of our method with other instance segmentation approaches are shown in Table 3. Houghnet, Mask R-CNN, SOLOv2, Blendmask, CondInst, and YOLACT++ achieved an mAP of 41.1, 74.4, 76.6, 79.8, 80.6, 80.9, respectively. CClusnet-Inseg outperformed these methods by achieving an mAP of 84.1.

The instance segmentation results of these methods applied on the test set were visualized (Fig. 5). The HoughNet showed a great error for the sow and even wrongly classified the piglet pixels with the bounding box of the sow as a part of the sow (Figs. 5a1, 5a2, and 5a5). Similarly, the Mask R-CNN also did not accurately detect the large sow object, as the method misclassified farrowing crates that occluded the sow as a part of the sow (Figs. 5b1-b4). YOLACT++ undetected some piglets in two cases (Figs. 5f2 and 5f5). SOLOv2, Blendmask, and CondInst all showed double detection of a single piglet in Fig. 5c1, Figs. 5d2 and 5d4-5d6, and Figs. 5e2-5e6, respectively. This happened especially when a piglet was separated into different parts for Blendmask (Fig. 5d6) and CondInst (Fig. 5e6). CClusnet-Inseg demonstrated better instance segmentation results than other methods. Even in the space between two farrowing crates, our method correctly detected the piglets (Fig. 5g3) and the sow (Fig. 5g2).

To demonstrate the potential applications of our method in animal monitoring, the instance segmentation results were directly applied to multi-object tracking. The video file is available in the Appendix and at https://www.samcityu.com/insegcclusnet. Screenshots from the video are shown in Fig. 6. From the video, it can be observed that due to its pixel-level accuracy, instance segmentation with IoU showed a small number of errors when pairing objects in consecutive frames, as long as the instance segmentation results were correct. The results of multi-object tracking produced six additional animal characterizations, i.e., trajectory (Fig. 7a), spatial distribution (Figs. 7b–e), movement, speed, body size, and space usage (Table 4) for individual objects.

The trajectory and spatial distribution with spatial usage could be used to estimate animal comfort, e.g., to determine whether sows make full use of the extra space resulting from opening the crates, and to determine whether the heat pad is effective based on its usage by the piglets (Fig. 2). The movement or average speed of an individual piglet could serve as a piglet vitality index and could be used as a warning of piglet mortality if a piglet does not show movement for a long period. This monitoring is especially important for piglets in the pre-weaning phase. The body pixel size could be used as an interim index for body size estimation and, thus, for body weight and growth estimation; however, the relationship needs further investigation.

Our proposed method is not an end-to-end method and consists of two stages to achieve instance segmentation. Consequently, one of the most critical limitations of our method is the low speed of instance segmentation in comparison with other methods (e.g., Mask R-CNN and YOLACT++). A faster algorithm or even a real-time algorithm is necessary for practical applications; therefore, additional efforts are required for improving the speed, e.g., exploring ways to directly generate the object center as output instead of using a clustering algorithm.

For generalization, our proposed method uses all visible pixels to detect a target, thus a high utilization of the available information in an image. This high utilization enables our method to manage heavy occlusion cases even though a target is largely occluded. In addition, our method is a target-sensitive and non-occlusion-specific method, since we only detect the pixels belonging to a target and classify all non-target pixels as background. As a result, our method is only sensitive to target pixels, and what the occluder is, what shape it is, and how many parts a target is divided into have few effects on our method. This increases the generalization of our method to be applied to other scenes with different occlusions. Furthermore, our proposed framework can be further expanded using other modules. For example, the encoder-decoder networks can use light networks, including MobileNet (Howard et al, 2017), EfficientNet (Tan and Le, 2019), and GhostNet (Han et al, 2020); the clustering algorithm can also be replaced by other algorithms including K-Means and Gaussian Mixed Model. The multiple selections in both two stages enhance the flexibility of our approaches between speed and accuracy, and thus enhance the capacity to fit different animal applications.

The object center detected and predicted through our method serves an accurate representation of the object’s location, especially when an object is occluded. When occlusions occur, general instance segmentation can identify only the visible pixels of an occluded piglet, and therefore, the location of the piglet (i.e., the center of the physical geometry of the piglet) cannot be accurately represented (Fig. 8). Our method was able to generate an occlusion-resistant object location (using predicted object centers) and, thus, occlusion-resistant movement and trajectory. Thus, our method is advantageous in applications related to animal monitoring, especially for scenes with heavy occlusion. Nevertheless, the detection and prediction of object centers increase the workload related to data labeling in our method.

We presented only a simple video as an example. Future studies could explore the numerous possibilities in multi-object tracking, e.g., use of the Hungarian algorithm (Kuhn, 1955) for object pairing and Kalman filter (Kalman, 1960) for trajectory prediction. In future work, we will explore robust multi-object tracking, and systematic animal monitoring and analysis.

The authors have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Funding was provided in part by the National Pork Board and the Pennsylvania Pork Producers Council in the U.S., and the new research initiatives at City University of Hong Kong (Project number: 9610450), Hong Kong SAR, China.

The datasets were generated from animal videos collected in one of our previous studies which was approved by the University of Pennsylvania’s Institutional Animal Care and Use Committee (Protocol #804656).