Task oriented multi-sensor fusion based on a differentiable approach: Application to coastal areas

Remote sensing has demonstrated great potential for characterizing the chemical and physical properties of the Earth’s surface, including coastal area. Several systems are equipped with multiple imaging sensors with complementary specifications in terms of resolution (e.g. Pléiades, PRISMA and Sentinel-2). Image fusion is a necessary step to obtain an image with both optimal spatial and spectral resolution. Most of the image fusion algorithms focus on generating a high (spatially and spectrally) resolution image that will then be used in other downstream tasks (e.g., object detection, land cover classification, change detection…). Most of the fusion algorithms proposed in the literature rely on task-agnostic assumptions to find the optimal fusion product, using explicit energy functional, or implicitly learning them from the data. The validation protocols rely typically on strategies to cope with the absence of a reference image at the target resolution using standard image quality metrics at low resolution. There is an extensive literature on these metrics, most of which are based on the assumption of scale invariance of the fusion process, an assumption that is questionable. Additionally, the correlation between these image metrics and the downstream task has not been extensively studied.

The objective of this PhD thesis is to propose an alternative approach to the image fusion process by considering the various stages of the processing chain— including image acquisition, the fusion itself, downstream tasks, and validation metrics—as tunable processes including learnable parameters, whose precise forms can be optimized according to a specific objective. This approach aims to not only provide a principled comparison between fusion algorithms but also improve the quality of downstream task performance, by optimizing previous elements of the processing chain to this end. These parameters can be optimized using conventional image quality metrics and/or task-based indicators. Such a unified view of the whole processing chain as a digital twin of operational acquisition and processing is made possible by the recent democratization of automatic differentiation frameworks (Pytorch, Tensorflow, JuliaDiff, JAX), and have been used in many areas of science recently (see e.g. the CLIMA initiative for global climate modeling). An end goal of this thesis is  to unify the characterization of sensor specification, design of fusion algorithms, and their application to downstream tasks, in order to characterize the impact of the sensor characteristics on the quality of the fused images.

As a first step in the thesis, selected representative algorithms belonging to the main category of fusion algorithms will be chosen and rewritten using an automatic differentiation framework. These main categories include machine learning based fusion, variational algorithms as well as classical fusion algorithms (based on component substitution or multi-resolution analysis). By combining this with simulated acquisition processes, this framework will allow us to identify optimal sensor specifications when considering a multi-sensor configuration (e.g. PAN + MS/HS). An example of a question that this framework could address is whether it is more advantageous to use a panchromatic sensor or a multispectral sensor alongside a hyperspectral sensor to optimize the fusion result. This first step would serve as a low risk task that will provide a proof of concept that components of the processing chain can be optimized to improve performance of a task that is further in the pipeline.

In the second part of the thesis, an application case in coastal areas will be addressed (considering a specific multi-sensor configuration) to showcase the interest of optimizing the fusion algorithm using downstream task related metrics, rather than image quality metric. This proof of concept will be developed using simulated data provided by the CNES, and real data (from EnMAP, PRISMA or Pleaides), coupled with high-resolution reference data (bathymetry or biological measurements). In order to assess the robustness of the approach to site specific sources of errors (water and benthic composition, impact of atmosphere...), two study sites in overseas and mainland France will be considered.

Based on the results of the first two tasks, the final part of the thesis could take on the more ambitious goal of assessing the combined impact of sensor parameters and the fusion algorithm on task performance, aiming to design dedicated data pipelines for each specific task.

Open-sourcing this work will enable researchers to adapt it for other use cases. Follow-ups include integrating sensors from different modalities to improve ecosystem state mapping. Another potential direction is optimizing sensor configurations, particularly for future specification mission like HYSP.

=================

For more Information about the topics and the co-financial partner (found by the lab !); contact Directeur de thèse - mauro.dalla-mura@gipsa-lab.grenoble-inp.fr

Then, prepare a resume, a recent transcript and a reference letter from your M2 supervisor/ engineering school director and you will be ready to apply online  before March 14th, 2025 Midnight Paris time !