3D Map Reconstruction of an Orchard using an Angle-Aware Covering Control Strategy

In modern agriculture, the relevance of the role of Unmanned Aerial Vehicles (UAVs), also known as drones, is rapidly growing (Mammarella et al. (2021)). Thanks to their enhanced capability to perform in-field operations in a precise and autonomous way, this typology of vehicles is leading to improvements in the context of the Agriculture 4.0 framework (Radoglou-Grammatikis et al. (2020) Mammarella et al. (2020)). As detailed in Comba et al. (2019a), UAVs could allow to extend, both in terms of spatial and temporal dimensions, the capability to monitor the crop status during the whole growing season, thanks to light and transportable sensors. Within this context, UAVs are exploited, for examples, to reveal crop water stresses (Guidoni et al. (2019)), soil erosion (Lima et al. (2021)), fungal and pest infestations (Calou et al. (2020)), etc. Recently, the potential of 3D crop model informative content for agricultural applications has been investigated in Comba et al. (2019b), as an alternative to widely-exploited 2D maps (Primicerio et al. (2015)). 3D map reconstruction through techniques like Structure-from-Motion (SfM) represents a powerful tool, even if it still represents a challenging task, due to the fact that the crop fields usually have poor and/or repetitive textures. Preliminary promising results have been achieved in the vineyard context where, thanks to a semantic interpretation approach, first proposed by Comba et al. (2020b), several relevant crop parameters, such as the leaf area index (see Comba et al. (2020a)), have been remotely measured. To speed up and improve the data acquisition process, the use of fleets of drones is nowadays considered a promising solution.

Classical covering control techniques typically requires drones to patrol over the environment by raising or lowering a density function (see e.g. Sugimoto et al. (2015)), according to the history of the past coverage states. On the other hand, to define and apply an effective covering control strategy to the selected field, it is crucial to identify which areas are of higher relevance, e.g. crop canopy, and needs to be detected from various viewing angles to improve the quality of the resulting map. This information can be resumed into a priority map, describing the distribution of the importance index function over the field. Approaches similar to the one presented in Comba et al. (2021) can be exploited to automatically retrieve the 3D-points density function distribution according to the field characteristics, starting from multi-spectral maps.

The main drawback of available covering control strategies is the lack of situation-adaptive features, that does not properly adapt the control action to the relevance and observation rate of each region. This is due because the importance index is defined as a monotonically decreasing function. Hence, if a drone is approaching a well-observed region with low importance index, the objective function gets low and the standard gradient-based coverage schemes tend to decelerate the drones instead of making them quickly escape from the same region.

To overcome all the aforementioned limitations of the current covering control strategies and to properly customize and optimize the density function distribution to the orchard of interest, in this paper we combine the semantic interpretation approach presented in Comba et al. (2021) for deriving a priority map to the angle-aware covering control scheme described in Shimizu et al. (2021) with the objective of providing an effective strategy to reconstruct a 3D map from rich viewing angles. The proposed scheme has been validated in simulation considering an apple orchard, first using a convolutional neural networks and an averaging filtering to identify the crop canopy and obtain the importance index field from a multi-spectral map, and then applying the distributed control strategy to a fleet of three UAVs. The preliminary results highlight the effectiveness of the covering controller and the adherence of the fleet behavior according to the evolution of the density function, demonstrating the capability of achieving situation-adaptive and angle-aware monitoring.

The remainder of the paper is structured as follows. In Section 2, we present the selected covering control strategy to obtain a 3D map reconstruction using a fleet of drones. Then, Section 3 we describe the algorithm exploited to extrapolate from a given multi-spectral map the corresponding priority map. The preliminary results obtained exploiting the angle-aware covering control to an apple orchard are presented in Section 4 while main conclusions are drawn in Section 5.

The case study selected envisions an apple orchard of about 0.4 ha located in Chiamina, Piedmont, North-west Italy (44°35’26” N, 7°29’30” E) (see Fig. 2). Gala apples are cultivated in the selected orchard, and the distance between trees is about 0.9 m and the inter-row space, which is covered by dense grass, is 3.5 m wide.

To properly apply the proposed covering control strategy to the selected agricultural scenario, it is crucial to retrieve a specific importance index field $\phi^{(0)}$, adherent with the features of the orchard and the requirements related to the main objective of 3D map reconstruction. The distribution of $\phi^{(0)}$ over the field is named priority map and it is used to select regions of an orchard which are the target of the remote sensing task for 3D reconstruction. In particular, high priority regions will be those representing the crop canopy, while the rest of the map is considered less relevant.

In details, the priority map of the orchard was derived by processing a multi-spectral orthomosaics of the site, obtained by an aerial imagery campaign performed on September, 4th 2020. In particular, a MAIA S2 multi-spectral camera (SAL Engineering and EOPTIS) was mounted on a s900 DJI hexacopter with a GNSS receiver (u-blox 6, Pixhawk avionics). The UgCS Pro mission planning software (SPH Engineering) was used to define a set of waypoints, in order to maintain the height of the UAV flight close to 35 m with respect to the terrain and to guarantee a forward and side overlap between adjacent images greater than 80%. The resulting ground sample distance (GSD) and the field of view (FOV), at 35 m of altitude, were equal to 1.75 cm pixel and 22.4x16.8 m, respectively. Image correction and true reflectance ratios calculation have been performed thanks to the incident light sensor (ILS), mounted on the top of the drone, which measure the ambient light level for each shot in each band. In addition, geometric correction, co-registration (or stitching) and radiometric correction was done by MAIA images software (MultiCam Stitcher Pro). Finally, the multi-spectral map of the whole orchard has been obtained by processing the image block with Agisoft Metashape (2020) software. Using the position of six ground markers (in-field determined with a differential GNSS), the map was also georeferenced in the WGS84 EPSG:4326 reference system.

The procedure to retrieve the priority map from the multi-spectral ortho-mosaics is based on 3 main steps: 1) the semantic interpretation by a convolutional neural network approach; 2) a refinement by morphological operations; and, finally, 3) a filtering and rescaling task. Pixels representing the crop canopy within the multi-spectral imagery are thus detected by a properly trained U-Net convolutional neural network (Mathworks Matlab®, 2020) and reported in a categorical map (see Fig. 3(b)). To train, validate and test the U-Net, the ortho-mosaic was processed to select three different subsets of pixels, as described in Comba et al. (2021). The raw categorical map provided by the U-Net (Fig. 3(b)) was then processed with a sequence of morphological operators, in order to remove noise and small objects from the map, and to refine crop canopies boundaries (Fig. 3(c)). In particular, the sub-sequentially performed operations are a closing and an opening operation, with a circular flat morphological structuring element with radius equal to 5 and 10 pixels, respectively. Then, to properly eliminate sharp gradient between clusters, an averaging filter with circular kernel was adopted (Fig. 3(d)). The final priority map is represented in Fig. 4 and used as the initial distribution of the importance index $\phi^{(0)}$ for the covering control strategy as described in the following section.

In this section, we describe the preliminary results obtained applying the proposed approach to an apple orchard. In the selected frameworks, we assume to cover the selected area with $N=3$ drones, whose initial positions were selected as $p_{1}=[5.0,0.5]^{\top}$, $p_{2}=[7.5,0.5]^{\top}$, and $p_{3}=[10.0,0.5]^{\top}$, to cover the selected orchard. The local controller allowed to maintain a relative altitude of $10$ m with respect to the terrain while the UAV velocity is constrained into the input space $\mathbb{U}=\{u\in\mathbb{U}\,|\,\|u\|\leq 5\}$, limiting the drones acceleration to less than 5 m/s². The other controller parameters were set as summarized in Table 1. In particular, $\sigma=1$ allowed to drive $h\rightarrow 0$ at $3.5$ m, which is the orchard inter-row space, while $\gamma=5000$ set the drone velocity around $2$ m/s when monitoring high-importance region.

From the priority map, retrieved as described in Section 3 and depicted in Fig. 4, we can observe how the importance index $\phi^{(0)}$ is distributed over the orchard. Indeed, the semantic interpretation algorithm allowed to automatically identify the areas of higher interest, i.e. the tree rows, and to assign them a higher value of $\phi^{(0)}$ (yellow area) whereas we have much lower values of the importance index (blue areas) on the orchard boundaries and in the inter-rows zones. This information are later fed to the Python simulator, that requires a further down sample of the priority map into a $325\times 200$ pixels representation (i.e. each pixel measures $0.3\times 0.3$ m) and then a compression of the map itself into a $80\times 130$ points field. In details, we have that each cell of the virtual field $\mathcal{Q}$ is a $0.3$m$\times 0.3$m$\times\frac{\pi}{30}$rad$\times\frac{\pi}{30}$rad polyhedron whose volume $A=\pi^{2}\times 10^{-4}$ m²rad², obtaining $m=1.95\times 10^{7}$. About the drone field $\mathcal{P}$, it was discretized by $l=10,400$ $0.75\times 0.77$ m² polygons $\mathcal{A}$.

The next step consisted in validating the proposed covering control algorithm in a ROS environment, where CVXOPT was used to solve the QP of the proposed controller with an update frequency of $20$ Hz. In Fig. 5, we reported some frames from the simulation interface where it is possible to observe the evolution of the performance index $\psi$ according to the field coverage performed by the drones. In each frame, we can observe also the importance function $\phi$ with respect to the viewing angles for some specific check points. The drones took around $4$ minutes to cover the field and to collect the images from the selected viewing angles. In the second frame of the upper row of Fig. 5 we can observe the collision avoidance strategy that makes the drones move away each other. On the other hand, in the third frame we can observe how the two drones on the right follows the priority map to collect images along the rows. Then, in the lower row of frames, it is possible to observe how progressively the drones covered the entire area from rich viewing angles, achieving the primary objective.

Fig. 6(a) shows the timeseries of the objective function $J$. It is possible to observe that $J$ is monotonically decreasing and converges to $0$. This implies that the UAVs would be able to collect ideal images when $J\rightarrow 0$. Next, we needed to verify the secondary objective related to the adherence of the drone behavior according to the image sampling. In particular, we had to verify that the UAVs escape regions characterized by small $\phi$ (low importance or well-observed in the past) and to slow down when overflying high-relevance areas. From Fig. 6(b) we can observe that the UAVs average velocity when covering a high-relevance area was close to $2$ m/s whereas the airspeed was increased by the algorithm to 5 m/s when the parcel had low $\phi$. This can further easily verified introducing an additional performance parameter defined as the sum of importance index near the drone $p_{1}$, namely $J_{near}\doteq\sum_{\iota\in\mathcal{M}\,s.t.\,\|p_{1}-\zeta_{\iota}\|<2\sigma}\psi_{\iota}A$. The results depicted in Fig. 6(b) shows that the UAVs speed $\|\dot{p}_{1}\|$ tends to be small when the index $J_{near}$ is large (large $\phi$), and vice versa, thus highlighting the fulfillment of the secondary objective.

In this paper, we applied a customized covering control strategy, that takes into account rich viewing angle to reconstruct 3D maps of an apple orchard. The proposed scheme is based on the preliminary extrapolation from a multi-spectral map of a priority field $\phi$ using a semantic interpretation approach. This allowed to determine the area of higher interest for the map reconstruction, coinciding whit the crop rows. The effectiveness of the proposed scheme was finally demonstrated through numerical simulations.