WeedNet: A Foundation Model-Based Global-to-Local AI Approach for Real-Time Weed Species Identification and Classification

Weeds have adapted to natural and human-modified environments, which leads to competition with cultivated crops for essential resources such as water, nutrients, light, and space, significantly affecting crop yields [1]. In the United States and Canada, the competition from weeds caused an average loss of $26.7 billion in maize (Zea mays L.) from 2007 to 2013 [2], while wheat (Triticum aestivum L.) produces losses averaging $2.19 billion for winter wheat and $1.14 billion for spring wheat during the same period [3]. The dry bean (Phaseolus vulgaris L.) faced an average loss of $622.2 million between 2007 and 2016 due to weed infestations[4], demonstrating the extensive economic impact of weeds on crop productivity.

Accurate identification of weeds is a crucial initial step in effective integrated weed management [5]. An important consideration is to correctly classify and categorize weed species. Weeds can be categorized according to the plant life cycle into annual, biennial, or perennial weeds; based on morphology into grasses, broad leaves, sedges and others; based on habitat into aquatic, terrestrial, and parasitic weeds; based on control method into herbicide-susceptible or herbicide-resistant weeds; on economic or agricultural impact into noxious weeds or crop-specific weeds and based on origin into native, introduced, or invasive weeds.

Invasive weed species pose a significant threat not only to agriculture but also to natural ecosystems. These species, which are not native to the environments they invade, can lead to the extinction of native plants, reduce biodiversity, and permanently alter habitats [6]. For example, leafy spurge (Euphorbia esula) is a highly invasive weed that reproduces through seed and vegetative structures. Its presence significantly reduces the species richness and, in some cases, native species can be eliminated [7]. Palmer amaranth (Amaranthus palmeri) is one of the most economically damaging weeds in corn, cotton and soybean production. Even when established in June or July, this weed exhibits intense competition with crops. It can continue to flower under the canopy of the crop for an extended period, resulting in a significant accumulation of biomass and a depletion of resources within the cropping systems [8]. However, the ecological impact of these invasive species is often realized only after they have become widespread, making early detection and classification critical for effective management [9].

The traditional method of manual weed identification is time-consuming and labor-intensive, involving visual inspection in the field, making it less effective for quick real-time weed management decisions, especially on large-scale farms. In production agriculture, where a herbicide-tolerant crop is grown, a blanket spray with an appropriate herbicide can somewhat circumvent the need for detailed weed scouting; however, if used throughout the season, it is an undesirable practice, as it causes propagation of herbicide-tolerant weed species and long-term production challenges [10]. Resistance to herbicides is a prevalent problem, with an average of five new cases emerging annually between 1990 and 2015 [11]. For example, Palmer amaranth and common waterhemp (Amaranthus tuberculatus), two similar species of the Amaranth family, show different herbicide resistance profiles, highlighting the need for an accurate identification[12, 13].

Correct identification in an early growth stage is crucial for implementing targeted control measures, optimizing weed management costs, and preventing the spread of weed populations [14, 15]. Misidentification of weed species can lead to ineffective control methods, exacerbating infestations [16, 17]. During scouting, environmental factors such as lighting and soil conditions can affect visibility, while the spatial variability of the weed distributions makes the comprehensive coverage complex [18, 19]. Timing is crucial, as scouting must be done early and frequently for optimal weed management and control [20]. Detecting new invasive species in diverse landscapes adds to the challenges of weed identification and its management [21]. In addition, achieving high accuracy in manual weed mapping and density estimation for large fields is problematic in itself. This highlighting the need to integrate manual techniques with emerging technologies such as AI-based detection systems [22], particularly edge devices such as smartphones [23, 24], unmanned aerial vehicles (UAVs) [25], and ground robots [26] to improve accuracy and efficiency in early identification and management of weeds.

Artificial intelligence (AI) is rapidly emerging as a transformative technology that aids farmers and scouts in the timely identification and mitigation of plant diseases [27, 28, 29], insects [30, 31], and weeds [32, 22]. Image-based phenotyping has significantly advanced weed detection through non-destructive high-throughput methods for accurate plant classification and spatial distribution [33]. Drones and ground robots equipped with RGB, multispectral, and hyperspectral cameras have been used to automatically capture high-resolution field images, supporting the development of algorithms for automatic weed detection and mapping [34, 35], diseases [27, 28, 29], and insects [30, 31]. Satellite imagery combined with AI-driven analytics has also facilitated large-scale monitoring of weed infestation dynamics [36, 37]. Remote sensing technologies integrated with AI offer scalable and precise solutions for weed management in precision agriculture by enabling real-time species-level identification of weeds. A key challenge is developing AI models capable of distinguishing subtle differences between crops, weeds, and lookalike species under field conditions, which the PlantCLEF challenge addresses [38]. This global competition uses a dataset of 80,000 plant species to tackle issues associated with many classes, class imbalance, variable image data quality, and noisy field backgrounds. For example, PlantCLEF’s 2023 dataset included more than 1.2 million annotated images to train convolutional neural networks (CNN) for fine-grained identification. These advances improve the accuracy of AI-driven weed detection tools while minimizing false positives, bridging the gap between scalable data collection and actionable weed management strategies [39, 40].

Several challenges are associated with the automated identification of weeds in crop production scenarios. These challenges include (a) data, (b) model, (c) performance, and (d) deployment.

Data Challenge: The primary challenge in developing robust AI-based deep learning models for identifying weeds in agriculture is the limited quantity and quality of image-based datasets. Quantity concerns include the total number of images and the adequate representation of each class. High-quality data must include diverse high-resolution images taken under varying conditions such as lighting, growth stages, soil types, and moisture levels. Previous studies have reviewed publicly available weed datasets [34, 41, 42] (see Supplementary Material S1). However, many contain images from controlled environments, such as greenhouses, lacking phenological variability of plants [43, 44]. In addition, these datasets often collect weed images from their local region, which reflects regional biases and makes it difficult to compile diverse datasets for comprehensive AI models. Expanding data set diversity is crucial to improve model generalization and performance. Citizen science platforms such as iNaturalist provide large-scale data collections throughout the world, offering research-grade photos with species-level identification [45]. An example of a research-grade observation shows the characteristics of the plant in the leaf blade, sheath, head, and seeds of green foxtail (Setaria viridis) [46]. Previous studies have compared the quality of research-grade iNaturalist observations with digitized herbarium specimens, finding a low misidentification rate between the two sources, which supports the utility of iNaturalist data [47]. However, there are concerns about class imbalance, as iNaturalist data often underrepresents rare or difficult-to-identify species and overrepresents common ones due to observer preferences and accessibility.

Model Challenge: Selecting the appropriate deep learning model for weed identification requires balancing model complexity, computational resources, data availability, and the need for high accuracy in real-world agricultural applications. Early attempts used hand-made features and traditional machine learning (ML) techniques such as support vector machines (SVM) and random forests (RF) to differentiate crops and weeds based on color, shape and texture descriptors [48, 49, 50]. However, these models struggled with variations in the morphology of the weeds and complex environmental conditions, limiting their applicability. Recent studies have used deep neural network architectures such as CNNs to improve accuracy in diverse field conditions. Pre-trained CNN architectures like ResNet-50, EfficientNet, and DenseNet have shown success in multiclass weed identification in cotton fields [51]. Graph-based deep learning models have also been developed for complex conditions [52]. Deep CNN-based models have been used to classify multiple weed species under various weather and sunlight conditions [53, 54]. Semi-supervised methods, such as semi-supervised generative adversarial networks (SGAN), have emerged to address the challenge of obtaining labeled data, achieving high accuracy with minimal labeled data [55]. SemiWeedNet integrates labeled and unlabeled images for weed segmentation and identification, improving the robustness of the model [56]. Despite these advancements, traditional deep learning models like CNNs face difficulties in generalizing and scaling due to insufficient data diversity and variability in training datasets [57, 34]. Recent models like transformers, built around attention mechanisms such as vision transformers (ViT), offer better scalability, precision, and efficiency for large-scale image classification tasks compared to CNNs [58]. ViTs capture global contextual relationships in images, making them effective for applications such as autonomous systems [59] and precision agriculture [60]. A significant challenge arises when weed identification models provide inaccurate predictions with high confidence, affecting user trust and system reliability [61, 62]. Misclassifying weeds and crops can lead to unintended crop removal or weed survival, adversely affecting yields and profitability. The inability to warn users of the uncertainty of the model in predicting weed species similar to the lookalike can erode confidence in AI applications. Environmental variability, such as different lighting conditions and soil types, can cause inconsistent model performance and erode user confidence. Data bias from nonrepresentative training datasets can lead to unreliable predictions. The lack of explainability in complex ”black box” models makes it difficult for users to trust their output, especially when errors occur [63].

Performance Challenge: The performance of weed identification models in real-world scenarios depends on their ability to accurately detect various weed species under variable field conditions, including common noxious, invasive, and look-alike species. Intra-species dissimilarity refers to individual differences within the same species (Figure 1) [38]. For example, common lambsquarters (Chenopodium album) show considerable intraspecific variation in leaf shape and size, influenced by environmental factors such as soil nutrients and water availability [64]. Horseweed (Erigeron canadensis) also exhibits distinct visual differences between its rosette and flowering stages, which complicates identification [65, 66, 67]. In contrast, inter-species similarity involves similarities between different species, including shared physical characteristics and ecological niches [68]. Morphological similarity occurs when two or more weed species resemble each other in leaf shape, growth habit, flower structure, or seeds [69] (Figure 2). For example, the amaranth species, such as redroot pigweed (Amaranthus retroflexus), Palmer amaranth, and common waterhemp, often co-exist in disturbed habitats, making differentiation difficult. Poisonous and non-poisonous species within the Apiaceae family (also known as the carrot family) share traits like pinnately divided leaves and umbel-shaped flower clusters, complicating identification [70]. Accurate identification is essential, as the wild parsnip (Pastinaca sativa) has a phototoxic sap that poses risks to public safety. In contrast, caraway (Carum carvi) and Golden Alexanders (Zizia aurea) have different ecological importance; Golden Alexanders is a beneficial native species and caraway is a useful culinary herb. Monocot species, such as yellow foxtail (Setaria pumila), green foxtail (Setaria viridis), and giant foxtail (Setaria faberi), require a close examination of morphological traits. Key identification features include the presence and distribution of hairs in leaf blades, the color and shape of the bristly seed heads, and the structure of the ligules and auricles for accurate identification [71] (Figure 2: Foxtail weeds).

The National Invasive Species Information Center (NISIC) lists approximately 60 terrestrial invasive plant species, including trees, shrubs, vines, grasses, and herbaceous plants [6]. These invasive plants include trees, shrubs, vines, grasses, and herbaceous plants. Examples include giant hogweed (Heracleum mantegazzianum), purple loosestrife (Lythrum salicaria), purple starthistle (Centaurea calcitrapa), kudzu (Pueraria montana), garlic mustard (Alliaria petiolata), Japanese knotweed (Reynoutria japonica), Johnsongrass (sorghum halepense), mile-a-minute (Persicaria perfoliata), tree of heaven (Ailanthus altissima), common ivy (Hedera helix), common buckthorn (Rhamnus cathartica) and witchweed (Striga asiatica) (Supplementary Figure S1). Early detection is essential to prevent their establishment and spread, as they can outcompete native vegetation, reduce biodiversity, and disrupt ecosystems. These species cause significant ecological and economic damage by altering habitats and reducing crop productivity. For example, witchweed parasitizes cereal crops, leading to total yield loss in subsistence farms [72]. Fast-growing invaders such as kudzu and Japanese knotweed smother native vegetation, incurring billions in control costs [73, 74]. Herbicide-resistant weeds, such as Johnsongrass, common waterhemp, giant ragweed, and Palmer amaranth, complicate eradication and challenge existing management strategies [75, 10]. Several invasive weeds also host damaging insects, amplifying their agricultural impact. The tree of heaven supports the invasive pest spotted lanternfly (Lycorma delicatula), a pest of fruit trees and vines [76]; common buckthorn shelters soybean aphids [77]; Johnsongrass harbors sugarcane aphids and crop viruses [78]; common barberry (Berberis vulgaris) serves as an alternate host for black stem rust (Puccinia graminis), a devastating disease of wheat, barley, and other small grains [79]. and the kudzu weed supports the invasive pest kudzu bug, which reduces soybean yields [80]. Rapid identification and management efforts informed by early detection protocols can mitigate their impacts and reduce long-term control costs [9].

Operational Challenge: The deployment of AI models to detect weeds poses significant computational challenges, as complex deep learning models require substantial computational resources [81]. An ongoing research challenge is to balance algorithm complexity with processing capacity for real-time field applications and optimize AI models for efficient processing on edge devices [82]. Integrating AI-based weed detection models into edge devices such as smartphones, rovers, and drones requires seamless integration for automated identification and real-time decision-making.

Integrating the weed identification model into a smartphone app provides a convenient tool for users to connect to the AI model for fast and accurate identification and take timely action for weed control. The traditional app works more like the conventional weed guide, which requires the user to select plant characteristics, such as plant type, color, and leaf shape [83]. In comparison, AI models use computer vision and are more convenient for giving automated results. During a five-year period, a weed specialist tested several smartphone apps for plant identification purposes [84]. The data set used included broadleaf, grassy, ornamental, and shrub species. The study emphasized that a simple process is key to user-friendliness and overall experience when using such tools. Among the tested applications, PictureThis delivered the highest performance, with an average accuracy of 73% [84]. However, these apps were designed primarily to identify general plant species rather than pest weed species for agricultural or ecological concerns.

These gaps motivate this research to develop a global model for weed identification for common noxious and invasive plant species that are considered weed species in production and non-production agriculture settings. A global model is trained on a large and diverse collection of plant images, such as those on platforms such as iNaturalist, using self-supervised learning (SSL). This model learns general visual patterns of a wide range of plant species from various regions, without relying on labeled data. Although the global model captures rich and transferable features for plant identification, it may not be specifically optimized for detecting or classifying agriculturally relevant weeds in a specific region or cropping system. A local model is trained exclusively on region-specific and expert-validated data sets focused on weeds that affect particular cropping systems or areas, for example, weeds that impact corn and soybean production in the Midwestern US. This model is highly accurate for the narrow task of identifying locally important weed species, but it may struggle with generalization to broader or unfamiliar contexts. Therefore, we used a global-to-local approach to fine-tune a pre-trained global model for weed species in Iowa, US, demonstrating its effectiveness even with limited local imagery [31]. This method starts with a broadly trained model and fine-tunes it using smaller, high-quality, expert-labeled datasets tailored to specific regions or crop systems. By combining the generalization strength of the global model with the precision of local data, this hybrid approach reliably identifies key weeds in targeted agricultural settings while reducing the need for extensive local annotation.

To address the above challenges in data quantity and quality, model generalization and performance, and deployment, this study presents WeedNet. This global-scale weed identification model combines self-supervised learning with a global-to-local fine-tuning strategy [85]. Built using the ViT architecture [86], the foundation model (global model) was trained on a large-scale dataset derived from iNaturalist, capturing broad variability across weed species and environmental conditions. To improve relevance and accuracy in regional applications, the global model is fine-tuned using smaller, expert-validated local datasets [87, 31]. This transfer learning approach takes advantage of the generalizability of large models while requiring less data and computation, reducing overfitting on limited datasets [88]. By integrating citizen science data with expert labels, WeedNet ensures reliable real-world performance. Its adaptability across platforms such as smartphones, drones, and ground rovers supports efficient, locally optimized weed detection for diverse agricultural environments and stakeholders worldwide.

We organized the results to follow the four main challenges in developing and deploying an AI model for real-time weed identification and classification.

Data Challenges. This study represents a pioneering effort to use biodiversity science resources to advance AI models for weed classification. The manual annotation process is one of the most labor intensive and time-consuming processes in AI model development [125]. Using publicly available datasets online can significantly reduce the effort and time required to build a training dataset from scratch for AI model development. The extensive collection of images and metadata collected, particularly from the iNaturalist dataset, marks a significant milestone for WeedNet. With more than 14 million images covering 1,593 weed species, this dataset provides unparalleled training data. Most species benefit from more than 500 verified training images, contributed by observers who captured various plant phenotypes, colors, viewpoints, backgrounds, light conditions, and growth stages. This diversity enriches WeedNet’s training capacity and enhances its overall performance. This comprehensive approach contrasts with previous studies, which have often focused on only economically significant or regionally dominant weed species [123, 126, 127, 51]. These models, while effective within their limited scope, lack greater applicability. In contrast, WeedNet is designed for deployment across diverse regions and cropping systems due to its global scale and species diversity.

Furthermore, some studies have developed weed classification models that distinguish mainly between crops and weeds [128, 129, 130]. These models typically overlook species-level weed identification, which limits their applicability in precision weed management, where species-specific herbicide recommendations are critical, especially for differentiating between herbicide-resistant and susceptible weed populations. WeedNet includes a diverse range of globally relevant weed species, encompassing invasive, noxious, poisonous, and conserved categories. Furthermore, WeedNet uses scientific and common names for each species, supporting consistent taxonomy and improving the reliability of species-level identification across diverse geographic and agronomic contexts.

WeedNet and the fine-tuning approach lead the refined Iowa model to achieve great accuracy across all expert-verified datasets with a small number of training images (see Table 3). This improvement may result from WeedNet serving as a foundational model that already captures most plant characteristics using the existing 1,593-species database, compared to using a generic image database, such as ImageNet, Objects365, or MS COCO, as a pre-trained model [132, 125, 133, 134]. The comparison between models demonstrated that the global-to-local strategy could improve the accuracy and adaptability of the model (see Table 1). Fine-tuning the WeedNet model with a local weed species collection can reduce the computational power and image collection resources typically required to develop an accurate local weed classifier.

One notable limitation of WeedNet is the class imbalance within the dataset. Rare weed species often have significantly fewer images than common or invasive species. Our results show that minority species are frequently misclassified as closely related species from the same family, likely due to shared phenological features and a substantially larger number of training images for the latter. This imbalance can lead to overfitting the model in underrepresented classes and introduces bias favoring dominant classes during training [135, 51, 136]. In other domains, class-imbalance learning approaches such as advanced multiclass algorithms [137], transfer learning with resampling [138], or reweighting techniques [139] have been proposed to address this challenge. However, these methods in weed identification require collaboration with weed scientists to define a minority class since species with few images but distinct features can still achieve high classification accuracy. Encouraging farmers, researchers, and agricultural extension workers to contribute labeled images can improve the model’s performance and create a more comprehensive database. WeedNet can overcome inconsistencies in crowd-sourced datasets by integrating expert data with crowd-sourced contributions.

As ensuring the reliability and trustworthiness of WeedNet is critical for its practical applications, we incorporate uncertainty quantification and out-of-distribution detection mechanisms. These features help identify cases where the model lacks confidence, prompting users to seek expert verification [110, 31]. By incorporating confidence scores and providing visual indicators of uncertainty, users can make more informed decisions and mitigate the risks associated with misidentification. These features can also help identify new invasive species in a region when the model consistently reports that the captured weeds are out of distribution, and can help farmers investigate further to identify the invasive species correctly.

Furthermore, our results demonstrated that the inter-species similarity significantly decreased the accuracy of the model (Sections 3.2.1 and 3.3.2). Few studies have highlighted the challenge of look-alike species in weed classification, a persistent problem in real-world scenarios. This study demonstrates that look-alike species are one of the main challenges influencing the global-scale weed classification model.

Traditionally, weed identification has relied on farmer or weed specialist observations of a limited number of well-known species, using phenological cues to infer which species are prone to misidentification. This approach can delay the detection of newly emerging or look-alike invasive species. WeedNet, which includes the most widely known invasive weeds and their classification accuracy, offers a novel way to identify species that are often confused with others. For example, the northern dock is mostly misidentified as a curled dock (see Supplementary Figure S5). It would not be misidentified as a green foxtail, which means species in the Dock family have similar leaf morphology, and these two docks are nearly identical. The identification output of each species tells which species are most commonly misidentified and which species they are confused with, which can further expand to a comprehensive list of look-alike species. This insight can guide the collection of additional images for look-alike species in future studies and help weed scientists pinpoint key distinguishing features for a more accurate identification [142]. The image tests focusing on the grass collars provide information on the importance of identifying and capturing the distinct morphological characteristics of species that are difficult to classify for the end users of the model (Figure  10).

Future research can also explore solutions from other fields facing similar challenges of interclass similarity and intraclass dissimilarity. For example, aerial scene classification has addressed such issues using Euclidean distances between class feature statistics [136] while embedding learning approaches in face recognition have leveraged softmax dissection techniques [143] to enhance discriminative learning.

When comparing manually captured images with those from the robotic platform, the primary difference lies in image resolution and the amount of plant detail captured. Previous studies have shown that image resolution drives the accuracy of individual weed plant discrimination [144, 145]. In particular, the UAV images in this study achieved resolutions of 0.1 to 0.12 cm/pixel, significantly contributing to the model’s effectiveness by preserving fine morphological traits. Meanwhile, a subset of citizen images focused only on specific plant characteristics (e.g., leaves, flowers, or stems), whereas the robotic platform typically captured the entire plant. These differences in citizen datasets and robotic applications highlight the importance of comprehensive plant characteristic coverage and high-resolution image data when training and deploying models for weed discrimination across imaging platforms.

For future applications, WeedNet has the potential to be extended for multi-object classification and object detection in complex field settings. High weed infestations often result in overlapping plants, making it difficult to distinguish between species in a single image. Furthermore, variations in lighting and shading in different parts of the image can pose additional challenges [54]. To address this, the images used in this study could serve as pre-training data to transition WeedNet from an image classification model to an object detection model [125, 146]. Incorporating WeedNet into autonomous platforms such as sprayer drones and precision sprayers would enable real-time species-level weed detection, improving control accuracy, protecting crop yield, reducing input costs and potential herbicide resistance.

WeedNet can also expand beyond identification to include segmentation and quantification of weed density estimation. Using the trained classification encoder for a segmentation model and explainable AI, images can be segmented to identify the spatial distribution of weeds without significant annotation cost [147, 22]. Farmers can gain insight into weed infestation patterns and plan targeted interventions. Furthermore, density estimation can help quantify the coverage of weeds in fields, providing data-driven recommendations for site-specific weed control. These advancements will enhance the utility of the model, making it an indispensable tool for precision agriculture and large-scale weed management strategies.

Furthermore, WeedNet has practical implications beyond field identification, particularly at critical checkpoints such as US ports of entry, where invasive weed species are frequently intercepted as disseminules, including seeds, fruits, and other plant parts. These propagules often contaminate commercial seed stocks, spices, shipping containers, and baggage. Currently, quarantine officials rely on tools such as the Federal Noxious Weed Disseminules of the United States (FNWD) and visual inspections to detect these threats [148]. The prominent species of concern include giant hogweed, Johnsongrass, Japanese knotweed, and witchweed. The Animal and Plant Health Inspection Service (APHIS), in collaboration with Customs and Border Protection (CBP), inspects incoming shipments and determines appropriate mitigation actions. With its high accuracy and adaptability, the Global WeedID model can be fine-tuned for disseminule-level identification and integrated into port-of-entry workflows. It has the potential to serve as a first line of defense by enabling rapid AI-assisted screening, ultimately supporting expert verification and timely mitigation decisions to prevent the spread of invasive species.

The study successfully developed a global scale weed identification model, WeedNet, using a citizen-sourced image repository (iNaturalist) that includes over 14 million images representing 1,593 weed species, encompassing diverse categories such as invasive, poisonous, noxious, and conserved plants. A two-stage training strategy was employed, combining self-supervised learning and fine-tuning with advanced architectures, including Masked Autoencoders (MAE), Vision Transformers, Out-of-Distribution (OOD) detection, and Conformal Prediction (CP) to improve model accuracy, robustness, and trustworthiness in the face of imbalanced datasets. WeedNet achieved outstanding performance, with 91.2% accuracy across all 1,593 species, 97.4% accuracy in 84 Midwest US weed species and 98% accuracy using expert-verified data added for species with conformal prediction. The model was further evaluated for performance under real-world challenges such as intra-species dissimilarity, inter-species similarity, and agriculturally and ecologically relevant invasive weed species, demonstrating that distinguishing morphological traits strongly correlates with accurate classification. Furthermore, the successful prediction of captured images from UAV and robotic platforms confirms the model’s adaptability to varying resolutions, lighting conditions, and overlapping plant instances, highlighting its potential for future real-time weed identification. Integrating WeedNet into PestIDBot establishes its role as a powerful and intelligent decision support tool for farmers and researchers in modern agriculture.

The dataset of 1593 species, all from the Plantae Kingdom and includes eight divisions, seven classes, eighty-one orders, and over a hundred families. The details of the taxonomy for each species are attached in the list of weeds.

For pretraining, we organize our datasets into varying sizes: 66k, 660k, 2M, and 6M, while for finetuning we used the all of the 14M images, from which we utilized 20 images for training and test sets. Our data processing pipeline involved resizing them to a format of 224x224x3 (height, width, channels) and normalizing their values. To improve our model’s generalizability and robustness, we employed various augmentation techniques to artificially expand the dataset during training. This included geometric and color-space transformations such as flipping, cropping to allow the model perform well under multiple camera positions, and adjustments to brightness and contrast to improve the model’s performance during various times of the day. Additionally, during pretraining, we incorporated advanced augmentation strategies like CutMix, and Mixup-Alpha, which have been shown to significantly improve classification performance. However, we observe a negative effect during finetuning, particularly across look alike species.

We utilized a Vision Transformer (ViT) architecture with patch size 14 by 14 and masking as the self-supervised learning (SSL) strategy for pertaining the encoder backbone. The pretrained encoder backbone was subsequently finetuned with a fully connected layer for classification. Both pretraining and fine-tuning were conducted in an end-to-end manner to optimize feature learning and representation. The model was trained for 10 epochs using a batch size of 16 and a base learning rate of 5e-4. We employed a layer-wise learning rate decay of 0.75 and a weight decay of 0.02 to regulate optimization. Additionally, dropout techniques such as drop path (0.1), stochastic depth regularization, and augmentation strategies, including mixup and cutmix were integrated to improve generalization. Pretraining and finetuning were completed on an A100 GPU cluster, utilizing 8 GPUs and 16 CPUs per task. Each epoch required an average training time of 8 hours.

In the zero-shot evaluation, accuracy was reported on the overlap between our classification model and various benchmark datasets as some classes were present, while others were absent. For CottonWeedDet12, 11 out of 12 classes were common, with the exception of Cutleaf Groundcherry (Physalis angulata). In DeepWeeds, 5 classes were common, while Chinee apple (Ziziphus mauritiana), Parkinsonia (Parkinsonia aculeata), and Siam weed (Chromolaena odorata) were not included. For WeedsAI, 15 classes overlapped with our classification list, including Amaranthus tuberculatus, Amaranthus palmeri, Amaranthus retroflexus, Ambrosia artemisiifolia, Bassia scoparia, Chenopodium album, Cryptostegia grandiflora, Erigeron canadensis, Lantana camara, Taraxacum officinale, Parthenium hysterophorus, Raphanus raphanistrum, Vachellia nilotica, Xanthium strumarium, and Zea mays.