By semantic segmentation we mean [1] segmentation using Deep Neural Networks (DNN). Sometimes it is also referred to as segmentation using AI (Artificial Intelligence). It tries to divide the image into classes with semantic meaning. I.e. unlike superpixels, it does not have to be an area of points with similar properties (e.g. an element from the class human can be in different clothes every time, or be dressed in different colors).

At this point, it is important to clarify the relationship between object detection and segmentation. Object detection algorithms are often only able to tell whether an object is present or not in the image. Sometimes they can also tell how many objects are there, sometimes they also know the coordinates and the bounding rectangle. I.e. the priority is identification and only then localization. Segmentation, on the other hand, refers to belonging to an object at the level of image points. That is, it primarily refers to localization and subsequently to interpretation.

A schematic representation of the DNN structure for semantic segmentation is shown in Fig. 1. Several examples of how to train and use DNN for semantic image segmentation in the MATLAB environment are presented in [1] [2] [3]. In Fig. 2 we see an example of semantic segmentation of a traffic-themed image. We see individual classes marked.

In semantic segmentation, there is also important the explainability , i.e. reasoning why DNN thinks that the given part of the image belongs to the object of the given class. An example of such an analysis using Grad-CAM method is shown in Fig. 3. Heatmaps are used to show the importance (how they contributed to the decision) of individual image points during segmentation.


Obr. 1 Schematic representation of the DNN structure for semantic segmentation [1]


Obr. 2 Example of semantic segmentation of a traffic-themed image [4]


Obr. 3 Example of explainability analysis using the Grad-CAM method [4]

References

[1] Mathworks, Getting Started with Semantic Segmentation Using Deep Learning, online: https://www.mathworks.com/help/vision/ug/getting-started-with-semantic-segmentation-using-deep-learning.html

[2] Mathworks, Example: Semantic Segmentation Using Deep Learning,  online: https://www.mathworks.com/help/vision/ug/semantic-segmentation-using-deep-learning.html

[3] Mathworks, Example: Semantic Segmentation Using Dilated Convolutions, https://www.mathworks.com/help/vision/ug/semantic-segmentation-using-dilated-convolutions.html

[4] Mathworks, Explore Semantic Segmentation Network Using Grad-CAM, online: https://www.mathworks.com/help/vision/ug/explore-semantic-segmentation-network-using-grad-cam.html


The PDF version of this text is available here: segmentation_semantic_eng.pdf