Autonomous Navigation in Rows of Trees and High Crops with Deep Semantic Segmentation

The first variant improves the approach proposed in [21]. After processing the segmentation mask, a sum over the column is performed to obtain a histogram $\textbf{h}\in\mathbb{R}^{w}$, quantifying how much vegetation is present on each column. Hereafter, a moving average on a window of $n$ elements is performed over the array to smooth the values and make the control more robust to punctual noise derived from the previous passages. Ideally, the minimum of this histogram $x_{h}$ corresponds to the regions where less vegetation is present and, therefore, identifies the desired central path inside the crop row. If more global minimum points are present (i.e., there is a region where no vegetation is detected), the mean of the considered points is considered to be the global minimum and, in consequence, the continuation of the row.

The second proposed approach consists of a variant of the previous algorithm, devised for wide rows with tall and thick canopies, which in the previous case would generate an ambiguous global minimum due to the constant presence of vegetation above the robot. This variant multiplies the previously processed segmentation mask for the normalized inverted depth datum.

where $\bigcap$ represents the element-wise multiplication between the binary mask $\hat{\textbf{X}}_{segDepth}^{t}$ and the inverted depth frame $\hat{\textbf{X}}_{segDepth}^{t}$ normalized over the depth threshold $d_{th}$. As in the previous case, the sum over the column is performed to obtain the 1D array h and, later on, the smoothing through a moving average. The introduced modification allows the closer elements to exert a greater influence on identifying the row direction.

We adopt the same network used in previous works on real-time crop segmentation [21, 6]. The model consists of a MobilenetV3 backbone for feature extraction and an efficient LR-ASPP segmentation head [24]. In particular, the LR-ASPP leverages effective modules such as depth-wise convolutions, channel-wise attention, and residual skip connections to provide an effective trade-off between accuracy and inference speed. The model is trained with a similar procedure to [6] on the AgriSeg dataset¹¹1https://pic4ser.polito.it/AgriSeg. Further details on the training strategy and hyperparameters are provided in Section 3.

The objective of the controller pipeline consists in keeping the mobile platform at the center of the row, which, in this work, is considered equivalent to keeping the row center in the middle of the camera frame. Therefore, as defined in the previous step, the minimum of the histogram should be centered in the frame width. The distance $d$ from the center of the frame and the minimum is defined as:

The linear and angular velocities are then generated through custom functions as in [25].

where $v_{x,max}$ is the maximum achievable linear speed and $\omega_{z,gain}$ is the andular gain. In order to avoid abrupt changes in the robot’s motion, the final velocities $\bar{v}_{x}$ and $\bar{\omega}_{z}$ commands are smoothed with an Exponential Moving Average (EMA) as:

where $t$ is the time step and $\lambda$ is a chosen weight.

The proposed control algorithm was tested through the use of Gazebo²²2https://gazebosim.org simulation software. The software was selected because of its compatibility with ROS 2 and can incorporate plugins that simulate sensors, such as cameras. A Clearpath Jackal model was utilized to assess the algorithm’s effectiveness. The URDF file, available through Clearpath Robotics, contains all the necessary information regarding the mechanical structure and joints of the robot. During the simulation, an Intel Realsense D435i plugin was utilized, positioned 20 cm in front of the robot’s center, and tilted $15^{\circ}$ upwards. This positioning gave the camera a better view of the upper branches of trees.

The navigation algorithm was tested in four different custom simulation environments: a common vineyard, a pergola vineyard characterized by vine poles and shoots above the row, a pear field constituted by small size trees, and a high trees field where canopies of the trees are merged above the row. Each simulated field adopts a different terrain, miming the irregularity of uneven terrain. The detailed measurements of the simulation world are described in Table 1.

During the experimental part of this work, we consider frame dimensions equal to $(h,w)=(224,224)$, which is the same size as the input and the output of the neural network model, with the number of channels $c=3$. The maximum linear velocity has been fixed to $v_{x,max}=0.5m/s$, and the maximum angular velocity has been fixed to $\omega_{z,max}=1rad/s$. The angular velocity gain $\omega_{z,gain}$ has been fixed to $0.01$ and the EMA buffer size has been fixed to 3. The depth threshold has been changed according to the various crops. In particular, it has been fixed to 5 m in the case of vineyards, while it was increased to 8 m for pear trees and pergola vineyards and 10 m for tall trees.

We train the crop segmentation model using a subset of the AgriSeg segmentation dataset [6]. In particular, for the High Tree and Pear crops, we train on Generic Tree splits 1 and 2, and on Pear; for Vineyards, we train on Vineyard and Pergola Vineyard (note that the testing environments are different from the ones from which the training samples are generated). In both cases, the model is trained for 50 epochs with Adam optimizer and learning rate $3\times 10^{-4}$. We apply data augmentation by randomly applying cropping, flipping, greyscaling, and random jitter to the images. Our experimentation code is developed in Python 3 using TensorFlow as the deep learning framework. We train models starting from ImageNet pretrained weights, so the input size is fixed to (224 × 224). The All the training runs are performed on a single Nvidia RTX 3090 graphic card.

The overall navigation pipeline of SegMin and its variant SegMinD are tested in realistic crops fields in simulation using relevant metrics for visual-based control without precise localization of the robot, as done in previous works [21, 11]. The camera frames are published at a frequency of 30 Hz, while the inference is carried out at 20 Hz, and the controllers publish the velocity commands at 5 Hz.The evaluation has been performed using the testing package of the open-source PIC4rl-gym³³3https://github.com/PIC4SeR/PIC4rl_gym in Gazebo [26]. The selected metrics aims at evaluating the effectiveness of the navigation (clearance time) as well as the precision, quantitatively comparing the obtained trajectories with a ground truth one though Mean Absolute Error (MAE) and Mean Squared Error (MSE). The ground truth trajectories have been computed averaging the curve obtained interpolating the plants poses in the rows. For the asymmetric pergola vineyard case, the row is intended as the portion of the pergola without vegetation on top, as shown in Figure 4 (c). The response of the algorithms to terrain irregularity and rows geometry is also studied including in the test significant kinematic information of the robot. The cumulative heading average $\gamma[rad]$ along the path is considered, together with the mean linear velocity $v_{avg}[m/s]$ and the standard deviation of the angular velocity $\omega_{stddev}[rad/s]$ commands predicted to keep the robot correctly oriented. The mean value of $\omega$ is always close to zero due to the consecutive correction of the robot orientation.

The complete results collection is reported in Table 2. For each metric, an average value and the standard deviation are indicated, since all the experiments have been repeated over 3 runs on a 20 m long track in each crop row. The proposed method demonstrates to solve the problem of guiding the robot through trees rows with thick canopies (high trees and pears) without a localization system, as well as in peculiar scenarios such as the pergola vineyards. The identification of plants branches and wooden supports hinder the usage of previously existing segmentation-based solutions, that were based on the assumption of finding a free passage solely considering the zeros of the binary segmentation mask [21]. We refer to this previous method as SegZeros in the results comparison, that we tested using the same segmentation neural network.

The SegMin approach based on histogram minimum search demonstrate to be a robust solution to guide the robot through trees rows. The introduction of the depth inverse values as weighting function allows SegMinD to be further increase the precision of the algorithm in following the central trajectory of the row in complex cases such as wide rows (high trees) and curved rows (curved vineyard). The different sum histograms obtained with SegMin and SegMinD are directly compared in Figure 5, showing the sharper trend and the global minimum isolation obtained including the depth values. Moreover, the novel methods show competitive performance also with standard crops rows where a free passage to the end of the row can be seen in the mask without the disturbance of canopies. The histogram minimum approach significantly reduce the navigation time and the trajectory precision in vineyard rows (straight and curved) compared to previous segmentation-based baseline method. The search of plant-free zero clusters in the map results to be less robust and efficient, leading the robot to undesired stops during the navigation, and to an overall slower and more oscillating behaviour.

Nonetheless, the trajectories obtained with the SegMin, SegMinD and SegZeros algorithms are also visually shown in Figure 6 inside representative scenarios: a cluttered, narrow row with small pear trees, a wide row with high trees, and curved vineyards with state-of-the-art method SegZeros.