Autonomous Apple Fruitlet Sizing and Growth Rate Tracking using Computer Vision

The Fruitlet Growth Model [16] specifies that growers select the fruitlet clusters⁴⁴4A cluster is a group of fruitlets that grow out of the same bud they want to size and measure the growth rates of all fruitlets in each cluster. Our system is designed to replicate this process by both sizing the fruitlets and autonomously identifying which fruitlets in the image belong to the cluster intended to be sized. We refer to this cluster as the target cluster.

Images are captured in the field using an illumination-invariant flash stereo camera [28]. Once images of all clusters are acquired, the data is passed through our sizing and growth-rate tracking pipelines (Fig. 1). In the first stage, the fruitlets belonging to the target clusters are extracted and sized. All fruitlets in each image are segmented using a Mask-RCNN [29] instance segmentation network, and disparities are extracted using RAFT-Stereo [30] to build point clouds of each fruitlet. The point clouds are used to cluster the fruitlets using a multi-stage Louvain Community Detection [31] approach, and the fruitlets corresponding to the target cluster are identified. Ellipses are fit to the fruitlets that belong to the target cluster, and the diameter of each fruitlet is calculated using the minor axis of the fit ellipse and the disparity values.

The second stage consists of fruit association and growth rate tracking. Fruitlets in images taken in the same and across different days are associated using an approach that makes use of Iterative Closest Point and the Hungarian Algorithm [32]. The sizes of the resulting temporally associated fruitlets are used to calculate growth rates, which in turn are used to predict the percentage of fruitlets that will persist and abscise.

The Fruitlet Growth Model specifies that growers choose the number of trees and the number of clusters per tree they want to size. While the original model suggests using 15 clusters per tree across 7 trees for a total of 105 clusters, we instead use 7 clusters per tree across 10 trees for a total of 70 clusters. This is because using fewer clusters has been previously adopted by apple growers to save time acquiring data. For example, Michigan State University⁵⁵5https://mediaspace.msu.edu/media/How+to+Use+the+Fruit+Growth+Model+to+Guide+Apple+Thinning+Decisions/1_o359pflp and the University of Massachusetts⁶⁶6https://ag.umass.edu/fruit/fact-sheets/hrt-recipe-predicting-fruit-set-using-fruitlet-growth-rate-model suggest using 5 trees with either 15 or 14 clusters per tree respectively. Additionally, spreading clusters over more trees allows us to collect data on a wider range of fruitlet sizes and appearances. This is advantageous as it allows us to evaluate our system’s performance on a more diverse and representative set of fruitlets. Because our objective is to compare predicted thinner response between computer vision and hand-caliper growth rates, and not compare against actual abscise rates, this is a fair sampling strategy for the purpose of this work.

We hang AprilTags [33] to identify the selected clusters and fruitlets. AprilTags were selected because they allow for fast identification in computer vision systems. The AprilTag is only used for identifying the cluster id in addition to determining which fruitlet cluster is the target cluster, as described in Section III-E. Each fruitlet in each cluster is assigned a unique id written on the back which is used for evaluating against ground truth. An example of tagged fruitlet trees and clusters can be seen in Fig. 2.

Images are captured using an in-hand version of the illumination-invariant flash stereo camera presented by Silwal et al. [28]. Stereo camera systems are commonly used in agriculture to extract 3D information as a result of their reliability and low cost. They resolve finer details compared to systems that use LiDAR [34, 35, 36], and flash-based systems are more resilient to varying illumination conditions (Fig. 3 c-d) where RGB-D sensors inconsistently perform [24, 22].

To facilitate data collection, we designed a custom setup that consists of the stereo camera connected via USB to a laptop to save the captured images. A phone is mounted on the back and connected via USB-C to the same laptop to allow the user to visualize and assess the quality of the images in real-time. When imaging, one to two images per cluster are captured per day. The reason for this is because we want to limit both the amount of time apple growers need to spend in the field acquiring images and the computational time required to post-process the data. Taking many images or long video sequences significantly adds time to both processes, which results in reduced benefit. Ideally, one image per cluster is taken, but two images are used when not every fruitlet is visible in a single image as a result of occlusions.

Two differences between our sizing approach and the one presented by Qadri [26] is we replace the MADNet [27] disparity and pix2pix [37] segmentation networks with RAFT-Stereo [30] and Mask-RCNN [29] respectively. RAFT-Stereo is a state-of-the-art deep learning-based stereo matching network that outperforms traditional disparity generation methods, such as SGBM [38], on benchmarks such as the Middleburry Stereo Evaluation⁷⁷7https://vision.middlebury.edu/stereo/eval3/. Additionally, RAFT-Stereo does not require fine-tuning, which is advantageous as the network does not need to be retrained between datasets or when switching out the stereo camera being used. In addition, Mask-RCNN is a more standard and widely adopted instance segmentation network compared to pix2pix, which is traditionally used for pair-to-pair image translation.

To account for the difficulties in obtaining ground truth segmented data, the Mask-RCNN network is trained following a two stage process. In the first stage, bounding box annotations are used to train the bounding box regression and classification heads. In the second stage, the weights of these heads are frozen, and a subset of the fruitlets in each image are segmented and used to train the mask head. An overview of the process can be seen in Fig. 4. We use a customized detectron2 [39] Mask-RCNN implementation.

In order to measure and compare the growth rates of the fruitlets, our system needs to be able to identify which fruitlets belong to the target cluster. We achieve this using a multi-stage Louvain Community Detection [31] approach. Louvain Community Detection is an algorithm that extracts communities within a graph using a heuristic method based on modularity optimization. A community is a subset of nodes whose connections among themselves are denser than connections with the rest of the graph. The algorithm iterates through nodes and moves each node into the community that maximizes modularity gain. The process repeats for each node until no modularity gain can be found. We use the NetworkX [40] Louvain Community Detection implementation.

Point clouds of each fruitlet are first extracted using the segmentations and disparities from Section III-D. The point clouds are filtered using a bilateral filter and a depth discontinuity filter [41]. Point clouds with too few points or inconsistent depth values are discarded.

We then perform our clustering approach as described in Algorithm 1. We first construct a graph $G_{1}$ consisting of the centroids $c_{i}\in\mathbb{R}^{3}$ of each fruitlet $i\in\{1,...,N\}$ as a node. An edge $e_{ij}$ is created between nodes if their euclidean distance is less than a pre-determined threshold $\tau_{1}$. The weight $w_{ij}$ of edge $e_{ij}$ is determined based on the euclidean distance, with closer nodes receiving a greater edge weight. Louvain Community Detection (LCD in Algorithm 1) is then run on $G_{1}$ to produce a set of communities $H_{1}$.

As a result of the inconsistent spacings between fruitlets and fruitlets falling off during the thinning period, naively using only Louvain Community Detection often results in fruitlet clusters being incorrectly divided into multiple communities (Fig. 5 (b)). We observed that when this occurs one of the communities almost always contains two fruitlets. To account for this, we iterate through all communities of size two and combine them with their nearest communities if the distance between the centroids of all the fruitlets in the combined community and the centroid of the combined community is less than a threshold $d_{c}$ (MergeCommunities in Algorithm 1).

There are occasionally clusters all of whose fruitlets are unassigned to a community. To overcome this, we repeat the above process for all unassigned fruitlets using a relaxed threshold. A graph $G_{2}$ is built consisting of the unassigned fruitlet centroids, and edges and weights are created as before with a new threshold $\tau_{2}>\tau_{1}$. Louvain Community Detection is run on this graph to produce a new set of communities $H_{2}$, which is combined with $H_{1}$ to form $H$. Once again, communities of size two are merged. As a final step, all remaining unassigned fruitlets are merged with their nearest communities (MergeUnassigned in Algorithm 1) if the distance between the centroids of all the fruitlets in the combined community and the centroid of the combined community is less than $d_{u}$. A visualization of the final result can be seen in Fig. 5 (c). The fruitlets belonging to the community closest to the AprilTag are determined to belong to the target cluster and are used for sizing and growth rate tracking (Fig. 5 (d)).

An ellipse is fit to the segmented fruitlets in the target cluster following the process demonstrated in Fig. 6. First, for each fruitlet in the target cluster, the contour surrounding the segmented points is extracted. Then, an ellipse is fit using the OpenCV [42] fitEllipse function. Finally, the size is calculated as $\frac{b\times a_{\text{minor}}}{d_{\text{med}}}$, where $b$ is the baseline of the stereo camera, $d_{\text{med}}$ is the median disparity value of the segmented pixels, and $a_{\text{minor}}$ is the length of the minor axis of the fit ellipse. The derivation for this equation can be found in [26].

While the sizing pipeline presented in Fig. 1 (a) automates the fruitlet sizing process by removing the need for hand-caliper measurements, the fruitlets still need to be associated between images taken on the same and across different days in order to measure growth rates. We accomplish this by using a coarse cluster alignment to fine fruitlet matching approach. First, for each image, a point cloud of the target cluster is created by combing the point clouds of the individual fruitlets. Iterative Closest Point is run on the cluster point clouds to align them. The fruitlets are then associated using the Hungarian algorithm [32] on the pairwise euclidean distances of their transformed centroids. Matches whose euclidean distance exceeds a specified threshold are discarded. This simple matching method provides reasonable performance to predict persist and abscise percentages.

As demonstrated in Fig. 1 (b), the fruitlets are first associated in images taken on the same day, and the larger size of each fruitlet is used for calculating growth rates. If only one image is taken of a cluster on a particular day, this step is skipped. Next, the fruitlets are temporally associated across days and growth rates are calculated by finding the difference between the end and start sizes.

Lastly, thinner response in the form of persist and abscise rates are predicted following the process described in the Fruitlet Growth Model [16]. This is achieved by calculating the average growth rate of the top 20 fastest growing fruitlets. All fruitlets with growths rates greater than 50% of this value are predicted to persist, and the rest abscise. We validate our calculations by comparing our results to those produced by the Predicting Fruitset spreadsheet available at the Michigan State University Extension Horticulture website ⁸⁸8https://www.canr.msu.edu/apples/horticulture/.

Our datasets will be made publicly available at ⁹⁹9https://labs.ri.cmu.edu/aiira/resources/ by the time of publication.

We evaluate our fruitlet sizing approach. After all fruitlets are sized, outliers are removed from our computer vision pipeline using a Z-score with threshold 3. The number of outliers removed is less than 0.25% of all fruitlets sized. The distribution of measured sizes using our computer vision sizing pipeline (CVSP) and the ground truth caliper measurements (GT) are reported Fig. 8. While the computer vision pipeline produces slightly larger results on Day 1 for most datasets, the size and growth trends are similar across the 5 day period with regards to mean, median, and size distribution.

Table II shows the Mean Absolute Error (MAE), the Mean Absolute Percentage Error (MAPE), the percentage of fruitlets sized, and the $R^{2}$ linear fit score of the computer vision sizes compared against the caliper method for each day, as well as across all days. All linear fit plots can be seen in in Fig. 12. The MAE is consistently under 1mm, and the MAPE remains below 10% except for on the first day of the Fuji 2021, Fuji 2022, Gala 2022, and Honeycrisp 2022 varietals. Additionally, we are able to size approximately 85% - 95% of fruitlets, and achieve $R^{2}$ scores between 0.724 and 0.899 across all days.

As expected, fruitless sized on the first day typically have a higher MAPE and lower $R^{2}$ score compared to the later days. One possible reason for this is because is it more difficult to accurately segment smaller fruitlets as their flowery top covers a greater percentage of the fruit. On the other hand, the errors may stem from the inconsistencies in measuring ground truth. The size variations from using calipers would have a more significant effect when the fruitlets are small.

We also plot the cumulative distribution functions (CDF) of fruitlet sizes in Fig. 12. The CDFs follow similar shapes across all varietals and days. The most noticeable difference is on Day 1 for most varietals, where there is a horizontal shift between the CVSP and ground truth distributions. This is again because of the difficulty in sizing smaller fruit.

We also investigate the effect of replacing RAFT-Stereo with the classical disparity estimation method SGBM [38] (SGBM-CVSP). As shown in Table II, fewer fruitlets are able to be sized using SGBM, especially for the fruitlets imaged in 2022. When a similar number of fruitlets are able to be sized (the 2021 varietals), using SGBM results in higher MAE and MAPE values and lower $R^{2}$ scores. These results are likely because SGBM is unable to assign a disparity value to every pixel. As well, the performance of its disparity estimation is sensitive to the algorithm parameters used.

We assess the speed of our sizing pipeline. Table III shows the distribution of sizing times from image input to sizing output. We ran our evaluation on an NVIDIA GeForce RTX 3070 GPU. The average processing time is approximately 4.74s, with RAFT-Stereo consuming a majority of the time with an average of 4.32s. The speed of RAFT-Stereo and the pipeline will be affected by the GPU used.

We asked apple growers how long it usually takes to size fruitlets with calipers. The response we received is that it takes approximately 30s per cluster, which we confirmed when collecting ground truth measurements. Based on this preliminary study, our computer vision pipeline was able to size fruitlets 6 times faster compared to hand-caliper measurements. We plan to evaluate this performance improvement on a larger scale in the future.

We evaluate the performance of our fruitlet association method and the ability for our full Fruitlet Growth Measurement Pipeline (FGMP) to predict persist and abscise rates. We also evaluate our pipeline against one that uses the classical disparity estimation method SGBM instead of RAFT-Stereo (SGBM-FGMP), and one that naively associates fruitlets based solely on order of size (Size-FGMP). This association method is often done in practice when growers use the modified Ferri Version of the Fruitlet Growth Model ¹⁰¹⁰10https://ag.umass.edu/sites/ag.umass.edu/files/fruit/predictingfruitsetmodelferri.pdf and apps such as Malusim ¹¹¹¹11https://malusim.org/ in order to save time from having to manually associate fruitlets. The full results can be seen in Table IV. We compare each method’s ability to predict fruitlet persist and abscise rates with assumed known associations (KA Persist / Abscise %) and compare the absolute abscise percentage difference (KA AAPD) against ground truth. Additionally, we evaluate the precision and recall performance of the association method used in addition to the performance of the full end-to-end pipeline with regards to predicted thinning response (E2E Persist / Abscise %) compared against ground truth (E2E AAPD). Because the only difference between FGMP and Size-FGMP is the association method, both will have the same KA Persist / Absicse % and KA AAPD values. These are reported only in the FGMP section of Table IV. Known and E2E association results are provided to demonstrate the performance should the grower choose to manually or autonomously associate the fruitlets respectively.

Among the six clusters, our method achieves the lowest average absolute percentage difference compared to ground truth using both known associations and the full end-to-end pipeline, with 3.75% and 3.33% average KA AAPD and E2E AAPD respectively. This is what best demonstrates the effectiveness of our approach. Apple growers would be able to draw similar conclusions about when to spray using our automated pipeline that does not require any manual sizing or fruitlet identification. Additionally, associating by size also achieves reasonable performance, with an average E2E AAPD of 4.52%. Replacing RAFT with SGBM performed the worst. The SGBM-FGMP pipeline acheived an average KA AAPD and E2E AAPD of 10.5% and 10.9% respectively, with AAPD values greater than 15% for both the 2021 datasets of the Fuji and Gala varietals.

Our method is also able to associate fruitlets with an average precision and recall of 92.2% and 76.1% across all datasets. This is reasonable given the simplicity of the approach. The results suggest that association performance does not have the most significant effect on the pipeline’s ability to estimate abscise rates, as the Size-FGMP method is able to produce reasonable predictions with average association precision and recall values of 46.1% and 43.6%. While more sophisticated deep-learning and slam-based methods may improve association, the small improvement in abscise predication might not be worth the additional computation time and loss of generalizability that our method provides.

We test our sizing pipeline on images collected by a robotic system in the field. The flash stereo camera is is attached to a 7 DoF robotic arm consisting of a UR5 and linear slider [43]. Images of 30 gala clusters were taken at the University of Massachusetts Amherst Cold Spring Orchard on 3 different days spanning a five day window: 05/22/2022, 05/24/2022, and 05/26/2022, which we denote as Day 1, Day 3, and Day 5 respectively. The robot was controlled to follow a predefined semi-spherical motion path around the detected AprilTag, collecting images at viewpoints equally distributed throughout the motion path (Fig. 9). For each cluster on each day, the two images with the greatest fruitlet segmented area of the target cluster were used to size and calculate growth rates.

The same process described in Section III was used to size the fruitlets. When associating, Iterative Closest Point was still used to correct noise in the robot extrinsics, and fruitlets belonging to the detected target cluster were matched using the Hungarian algorithm on point cloud centroids.

We record the size distributions (Fig. 10) for measurements taken by both our computer vision sizing pipeline with robot captured images (RCVSP) and the caliper method (GT). Across all days, the sizes produced by the computer vision pipeline are larger on average than those produced by the caliper method, and the distributions are much narrower. This is because the robot is naively following pre-set path and not reasoning about where to capture an image. As a result, smaller fruitlets are likely to be completely occluded and unsized.

The MAE, MAPE, and percentage of fruitlets sized are presented in Table V. While the MAE remains under 1mm, the MAPE is higher on average compared to the hand-held camera results presented in Section IV-B. This is because of the partial occlusion of the fruitlets in the images as a result of following the pre-set path. Additionally, the percentage of fruitlets sized decreases as time goes on. This is a result of the larger fruitlets fully occluding the smaller ones, which has a more significant effect as the fruitlets grow.