FoodSAM: Any Food Segmentation

Foundation models, trained on broad data for adapting to diverse downstream tasks, have driven recent advances in machine learning. This paradigm often incorporates techniques like self-supervised learning, transfer learning, and prompt tuning. Natural language processing has particularly benefited, with the Generative Pre-trained Transformers [16, 17] series pre-trained on massive text corpora enabling models [18, 19] to tackle translation, question answering, and other applications. Contrastive Language-Image Pre-training [20, 21] trained on image-text pairs can effectively retrieve images given text prompts, enabling image classification and generation applications. While most extract knowledge from available data, the Segment Anything Model [7] uniquely co-develops alongside model-in-the-loop annotation to construct a custom data engine with over 1 billion masks, conferring strong generalization. These foundation models [22, 23, 24] have achieved state-of-the-art performance across domains. The paradigm shows immense promise to profoundly advance machine learning across disciplines.

Image segmentation [25, 26] encompasses various techniques to delineate distinct regions based on specified criteria. Each approach possesses unique characteristics and applications. Interactive segmentation leverages user guidance to enhance accuracy [27, 28] with users providing foreground/background markers to steer the algorithm. Superpixel methods group pixels into larger units called superpixels based on shared attributes like color and texture [29]. This simplifies the image representation while retaining key structural aspects. Additional prevalent methods include thresholding, edge-based, region-based, and graph-based segmentation, exploiting intensity, discontinuity, similarity, and connectivity cues respectively [30]. The optimal technique depends on factors like task goals, data properties, and required outputs. Segmentation [31] remains an active area of research with existing methods limited in fully handling real-world complexity.

Semantic segmentation represents a comprehensive approach in which each pixel is classified into a particular category, effectively partitioning the image according to semantic entities [32, 33, 34]. Recent methods like DANet [35], OCNet [36], and SETR [37] focus on extracting richer feature representations. Building upon semantic segmentation, instance segmentation [38] also delineates individual objects of the same class as distinct instances [39, 40]. Panoptic segmentation unifies semantic and instance segmentation, assigning each pixel both a class label and a unique instance identifier if part of a segmented object [41, 42, 43]. This provides a holistic scene understanding by categorizing and differentiating all present elements. Each formulation provides unique information - semantic conveys categorical regions, instance delineates object quantities, and panoptic characterizes both detailed semantics and individual entity counts.

Promptable segmentation has emerged as a versatile paradigm leveraging recent advancements in natural language models [44, 45]. This approach employs careful ”prompt engineering” to guide the model toward desired outputs [46, 47], departing from traditional multi-task frameworks with predefined tasks. At inference, promptable models adapt to new tasks using natural language prompts as context [44]. The Segment Anything Model [7] exemplifies this, training on 11 million images with 1.1 billion masks to segment objects specified in user prompts without additional fine-tuning. While SAM shows promising zero-shot generalization, a key limitation is the lack of semantic meaning in its mask predictions.

Food image segmentation constitutes an essential and indispensable technology for enabling health applications such as estimating recipes, nutrition[48], and caloric content[49]. Compared to semantic segmentation of general objects, food image segmentation poses greater challenges due to the immense diversity in food appearances [50] and often imbalanced distribution [51] of ingredient categories. For images containing multiple foods, segmentation represents a necessary precursor for dietary assessment systems. Segmenting overlapping, amorphous, or low-contrast foods lacking distinct color or texture features proves highly challenging. Furthermore, lighting conditions introducing shadows and reflections can adversely impact segmentation performance. Overall, the complex visual properties and arrangements of foods render this domain exceptionally demanding. Advanced segmentation techniques capable of handling food variability in unconstrained environments remain imperative for practical deployment.

Wu et al. [11] proposed ReLeM to reduce the high intra-class variance of ingredients stemming from diverse cooking methods by integrating it into semantic segmentation models [37, 52]. Wang et al. [53] combined a Swin Transformer and PPM module in their STPPN model to achieve state-of-the-art food segmentation performance. Honbu et al. [54] explored zero-shot and few-shot segmentation techniques in USFoodSeg for unseen food categories. Sinha et al. [55] benchmarked Transformer and convolutional backbones for transferring visual knowledge to food segmentation. While these approaches have driven progress, semantic segmentation provides only categorical predictions without differentiating individual food items.

However, to the best of our knowledge, no prior work has explored promptable or instance segmentation for food images and no existing datasets have supported instance-level or panoptic segmentation for food images. Semantic segmentation only provides categorical predictions without differentiating individual food items. In contrast, instance segmentation can delineate and count distinct food objects, enabling more accurate nutrition and calorie estimation. Furthermore, panoptic segmentation can characterize the surrounding environment, discerning attributes like the food’s container and utensils. Such contextual cues provide meaningful signals about food properties and consumption habits. Therefore, advancing instance and panoptic food segmentation represent an important direction, as these task formulations are more informative than semantic alone for downstream food computing applications.

Revisit of SAM: The Segment Anything Model (SAM) [7] represents the first application of foundation models to the image segmentation task domain. As shown in Fig.2 The proposed model is comprised of three key components - an image encoder, a prompt encoder, and a lightweight mask decoder module. The image encoder implements a computationally intensive vision transformer architecture with millions of parameters to effectively extract salient visual features from the input image. SAM provides three scale-specific pre-trained image encoder configurations: ViT-B (91M parameters), ViT-L (308M parameters), and ViT-H (636M parameters) [56, 57]. The prompt encoder enables four types of textual or spatial inputs: points, boxes, freeform text, and existing masks. Points and boxes are represented using positional encodings[58], and the text is encoded using a pre-trained text encoder from the CLIP model [20], while mask inputs are embedded using convolutions. These prompt embeddings are summed element-wise with the image features. The mask decoder module employs a Transformer-based architecture, applying self-attention to the prompt and cross-attention between the prompt and image encoder outputs. This is followed by a dynamic mask prediction head that outputs pixel-level mask probabilities and predicted Intersection over Union (IoU) metrics. Transposed convolutions upsample the mask decoder features. Critically, the mask decoder can generate multiple mask outputs to account for inherent ambiguity in the prompt. The default configuration predicts 3 masks per prompt input. Notably, the image encoder extracts the feature only once per input image, allowing the cached image embedding to be reused across different prompts for the same image. This separation of the expensive image inference from the lightweight prompt interaction enables novel interactive use cases like real-time mobile Augmented Reality prompting. SAM was trained on a large-scale dataset of over 11 million images with 1 billion masks, yielding strong zero-shot transfer capabilities. As the name suggests, SAM can segment virtually any concept, even completely novel objects unseen during training.

Variants of SAM: Very recently, there are serval concurrent works proposed to address the limitations of SAM. Here we introduce them by following.

RAM [59] is an innovative image tagging foundational model based on SAM. Trained on a large set of image-text pairs, this model efficiently recognizes common categories, enabling the acquisition of a vast amount of image tags without the need for manual annotations.

SEEM [60] is an interactive segmentation model that can perform image segmentation in an all-pixel, all-semantics manner simultaneously while supporting interactive segmentation with various prompt types(including click, box, polygon, scribble, text, and referring region from another image). Their experiments demonstrate that SEEM achieves remarkable performance in open-vocabulary segmentation and interactive segmentation tasks, and exhibits robust generalization to diverse user intents.

SSA [61] is a novel open framework that pioneers the application of SAM in the domain of semantic segmentation tasks. Its primary aim is to facilitate the seamless integration of users’ pre-existing semantic segmenters with SAM, obviating the necessity for retraining or fine-tuning SAM’s parameters. This integration empowers users to achieve enhanced generalization capabilities and finer delineation of mask boundaries in semantic segmentation tasks.

We explore applying SAM, a powerful mask generator, to food image segmentation. Although SAM segments food images with high quality, the generated masks lack categorical semantics. Standard semantic segmentation provides category labels but with low quality in food images.

To address this, we proposed FoodSAM, a framework that merges the benefits of both approaches. We assign SAM’s high-quality masks with semantic labels based on the mask-category match. Furthermore, due to the ingredients of food being randomly cut and placed when cooking, they are supposed as independent individuals, which motivate us to achieve instance segmentation for food images. Besides, to segment fine-grained objects apart from the background, FoodSAM contains an object detector for detecting non-food objects, such as a table, plates, spoons, etc. Therefore, FoodSAM also can achieve high-quality panoptic segmentation for food images.

Inspired by prompt learning, we also introduce promptable segmentation to food images, which is a novel task proposed by SAM. In the object detector, we convert the SAM’s prompt learning way to the prompt-prior selection, which also supports the point, box, and mask prompts. In this way, FoodSAM supports interactive prompt-based segmentation across multiple levels of granularity on food images.

FoodSAM consists of three main models: the segment anything model $M_{a}$, a semantic segmentation module $M_{s}$, and an object detector $M_{d}$, which also designs novel and seamless integration methods including the mask-category match, merge strategy, and prompt-prior selection. As shown in Fig.3, we here describe them by the order of the pipeline: enhance segmentation, semantic-to-instance segmentation, and instance-to-panoptic segmentation. And promptable segmentation is inserted in this pipeline with its corresponding prompt.

Enhance Semantic Segmentation: Suppose the input food image $I\in R^{H\times W}$ forward by the semantic segmentation module model $M_{s}$ and output the semantic mask $m_{s}=M_{s}(I)$, which has the same shape as $I$ and the pixel value is corresponding to the category of the pixel. SAM $M_{a}$ forward the image $I$ and output the binary candidate masks $m_{a}=M_{a}(I),m_{a}\in R^{K\times H\times W}$, where $K$ is the number of masks and the pixel value is True or False to indicate the pixel is foreground or background.

In the mask-category match, we use the i-th binary mask $m_{a}^{i}$ as the indices to get the i-th local semantic value set $D_{i}=m_{s}[m_{a}^{i}]$. Then, we use the voting scheme to choose the highest frequency semantic value $s_{i}$ in $D_{i}$ as the semantic label of the i-th mask $m_{a}^{i}$. Meanwhile, we also calculate the confused degree of the category mask by $d_{i}=\text{Count}(S_{i},D_{i})/\text{Len}(D_{i})$, where $d_{i}$ denotes the i-th confused degree of category mask whose semantic label is $s_{i}$. And Count function summarizes the number of that label $s_{i}$ in the set $D_{i}$, Len function summarizes the number of the positive values $D_{i}$ in the mask. We also will filter the mask whose $d_{i}$ is lower than a threshold $\tau$, which means the mask label is confused and unstable.

Therefore, with the merge strategy, we can get the enhanced semantic mask $m_{s}^{e}$ by merging the $K$ binary masks $m_{a}^{i}$ with the semantic label $s_{i}$. In this strategy, we take consideration into the conflict area when those masks exist overlap. We first sort them by the area size $\text{Len}(D_{i})$ and put them on the original semantic mask from large to small, which can preserve fine-object results This process fully takes advantage of the high-quality mask of SAM and the semantic label of the semantic segmenter.

Semantic-to-Instance Segmentation: In the enhanced process, we get the semantic labels for each binary mask. Since the food is randomly cut and placed when cooking, we suppose the ingredients of food as independent individuals. Here we merge small masks into the nearby mask which has the same category label, and filter very small masks if their location is separate. After filtering the binary masks which correspond to the background category, we obtain the foreground masks. For those foreground semantic labels, the t-th binary mask $m_{a}^{t}$ with the voted semantic label $s_{t}$ is supposed as the t-th instance’s mask $m_{i}^{t}$, where $t$ is the index of the foreground mask (instance id). Finally, we get the instance mask $m_{i}$ by merging the $T$ masks $m_{i}^{t}$ as well as the instance id.

Instance-to-Panoptic Segmentation: In the food image, the non-food objects, such as a table, plate, spoon, etc, are also important for the food image, which indicates the attributes of the food. Here we also suppose the non-food objects as independent individuals by introducing the object detector $M_{d}$. The object detector $M_{d}$ forward the image $I$ and outputs the non-food bounding boxes $B_{d}$ with the corresponding class labels $C_{d}$. In whole mask segmentation, SAM uses the dense gride points as the prompt, and remain the $K$ points which corresponding to $m_{a}$. Here, we take each point of SAM-generated mask to select the non-food object candidates by judging if located in the bounding box $B_{d}$. We next collect the binary masks whose semantic label is background, and then calculate the IoUs of the non-food candidate bounding boxes and that binary mask. When the highest IoU is large than another threshold, we suppose the corresponding class is the category of that mask. Otherwise, it will be preserved its background label. And non-food masks also will be merged followed by semantic-to-instance practice. Finally, we get the panoptic mask $m_{p}$ by merging the instance mask $m_{i}$ and the non-food object’s semantic masks.

Promptable Segmentation: SAM proposes a novel task promptable segmentation, which can achieve interactive prompt-based segmentation. In the FoodSAM, we also introduce promptable segmentation to food images and further estimate the food category. By leveraging the original semantic segmentation, we first segment interest food objects by SAM and can achieve interactive prompt-based segmentation at semantic and instance levels. Moreover, we propose a novel design for the object detector $M_{d}$ to support promptable detection across multiple levels of granularity. In the object detector, we convert the SAM’s prompt learning way to prompt-prior selection for the interest object. The concurrent work FastSAM[62] also introduces a selection strategy, but it is used after finished segmentation.

Different from FastSAM, we use the prompt-prior selection to segment the non-food objects in the background. It mainly involves the utilization of point prompt, box prompt, and mask prompt. Those non-food objects in the background are all assigned the category label from object detector. For the point prompt, we first remain object’s box information as prior boxes and use the point prompt to check the points whether in the prior boxes or not and most center to select the prior box. For the box prompt, we take IoU matching between the box prompt and the detected box from the object detector. The goal is to find the prior box with the highest IoU with the detected box. For the mask prompt, we sample points in the mask and check the points whether in the detected box, then followed by the same way as the point prompt. Specifically, for the regular prompt (the mask shape is the same as the input image), it uses near all non-food information which contain $K$ points for the next segmentation. Consequently, our FoodSAM achieves promptable segmentation on both food and non-food objects across multiple levels of granularity.

To compare with state-of-the-art methods, we have conducted experiments across multiple levels of granularity containing semantic, instance, and panoptic. Also, we compare our FoodSAM with SAMs in promptable segmentation tasks. We next discuss the results in detail.

In this subsection, we also explore the improvement by leveraging the SAM segment performance. As shown in Tab.I and Tab.II, FoodSAM achieves obvious improvement compared with baselines, which indicates that SAM performs impressing segment performance and compensates for the original semantical segmenter limitation. SAM produces many masks for each may exist object, we here experiment with the improvement of the number of merging masks.

Specifically, we choose the top-k area masks from FoodSAM to merge with the original semantic mask from the baseline. As shown in Tab.IV and Tab.V, the performance increase with the increasing k. When using 80 masks, FoodSAM achieves 66.14 mIoU, 78.00 mAcc, 88.47 aAcc on UECFoodPix Complete and 46.42 mIoU, 58.27 mAcc, 84.10 aAcc on FoodSeg103. These experiments demonstrate that it is feasible for the fusion between SAM and food segmenter, which shows significant improvement with FoodSAM.

To verify the function of each part, we also implement the ablation study in these parts on FoodSeg103 benchmark: filtering the mask with a confused category label (FCC) or not, and with SAM-generated mask (WSM) or not. In only WSM, we use zero-mask as the init background mask and the label from the original semantic mask, while in the case of combining the baseline, we also use the original semantic mask as the init background. As shown in Tab.VI, only WSM also could achieve comparable performance and the combination could get significant improvement compared to the baseline. With FCC, we merge with useful category label, and the mask with confused category use the corresponding area with semantic meanings.

And in the merging process, it needs to deal with the conflict area when masks exist the overlap. We also conduct an ablation study to verify which way is most effective, which considers two factors: SAM-predicted IoU and the number of positive values in a mask. As shown in Tab.VII, sorted by area, from large to small, achieves the best performance. It will cover those small fine areas when using large areas at the latter. We explore the cues that the SAM predicted IoU has a similar value close to 1, without strong distinguish.

Before the merging process, if the number of the most category label is relatively low in a mask, we suppose the category of that mask is confused and filter them to the later merging process. Therefore, we also experiment with which threshold is effective for FoodSAM. When the threshold $\tau$ equals zero, it means not using this practice. The experiments on FoodSeg103 in shown in Tab.VIII and UECFoodPix Complete shownn in IX. Using the area of the original semantic mask to represent the confused mask can make a minor improvement in FoodSeg103, and a significant increase in UECFoodPix Complete. The reason behind this is that FoodSeg103 is the dataset for fine-grained ingredients, while UECFoodPix Complete only contains the food label, its label is coarse-grained. Therefore, when the number of confused labels is larger, our method achieves higher improvement.