Sentinel 1 and 2: Searching for Physical Image Content
Schwarz, Gottfried; Datcu, Mihai

ESA’s upcoming Sentinel-1 and Sentinel-2 missions open new perspectives for the application-oriented use of SAR and optical multispectral images. We expect short and regular revisit times as well as easily available and well documented products with attractive features such as cross-polarized SAR images and optical images delivered, for instance, as spectral reflectance data.

Thus, users do not have to live any longer with simple digital units or detector counts; instead, the data provided as Sentinel products can be understood as samples of calibrated and validated physical quantities. As a consequence, users can concentrate immediately on the physics and quantitative details of the observable phenomena.

This also affects content-based image retrieval, where a user searches for images containing phenomena being similar to given examples. While retrieval systems based on visible image data can only exploit characteristic shapes or patterns, the use of Sentinel data will address the determination of real physical relationships.

This physics-based approach will allow us to employ content-based image retrieval as an attractive tool for the analysis of SAR and optical images:

  • Feature extraction from radiometrically stable cross-polarized SAR images: Human-made objects will appear with many relationships that are not visible in conventional co-polarized SAR images. Quite a number of recent publications dealing with the analysis of SAR images concentrate on the understanding of polarization effects for object detection and identification. We can train an image retrieval system to learn and detect such polarization effects. A user can then exploit characteristic polarization effects for detailed scene understanding.
  • Reliable classification of spectral phenomena in optical images of vegetated areas: Details of vegetation phenomena will appear as physical data that can be exploited quantitatively. A large variety of derived parameters will become available as products that need to be interpreted and understood from a user perspective. Again, an image retrieval system can be trained for specific cases and will support the search for similar (or even dissimilar) events.

    Within the framework of an existing content-based image retrieval system, we address the additional capabilities of image mining due to new Sentinel products. Typical image retrieval results from existing data sets will be used to demonstrate the capabilities and limitations of current missions. These results will be extrapolated to show what gain we expect when the Sentinel products become available. The gain the we expect will be described within the limitations of the quality levels of the Sentinel products and given by the conventions being available for quality assessments via precision/recall measurements .

    Another step is the use of an image mining system for semantic auto-annotation of images. While image annotation in a pre-Sentinel ground system can be understood as attaching labels typically referring to image land cover and/or land use content of image sub-areas, image annotation in a Sentinel era ground segment can be extended to new labels related to the physics-related scattering behavior of objects in SAR images or reflectance phenomena in multispectral images.

    In order to define a robust set of new labels, we have to define characteristic scenarios and train our image retrieval system accordingly. This definition and training stage has to be done efficiently; the selection of a global strategy is another point that will be discussed.

    The discussion will be based on the experiences gained during the design phase of an image mining system under ESA contract ( This project serves to set up the next-generation of image information mining systems, implementing novel techniques for image content exploration. It will exploit information about Earth observation product contents which is usually hidden in raster data, image time series and metadata, thus enabling content-based search in very large archives of high resolution image data. The developed software will be interfaced to and operated in a multi-mission payload ground segment.