Self-supervised training strategies for deep hyperspectral unmixing and uncertainty quantification

19 Novembre 2024

---

Catégorie : Stagiaire

---

The full subject, including images and references, can be found at https://partage.imt.fr/index.php/s/4kZRXNwMfaZrscn

Context

The remote sensing field aims at exploiting satellite or aerial images for earth observation. This, in turns, has an increasingly important social impact, as the applications of these methods include tackling key challenges such as environment monitoring (e.g. urban monitoring, deforestation, crop and agriculture monitoring...). This internship specifically focuses on remote sensing HyperSpectral Images (HSIs), in which the considered scene is observed in many (typically hundreds) of spectral/colorimetric bands. Such a high spectral information enables in principle to identify the materials (water, sand, vegetation...) present in the scene by looking at the spectra acquired in each pixel. Nevertheless, despite their high spectral resolution, HSIs suffer from a low spatial resolution, making such an identification difficult, since several materials are usually present in each pixel.

Hyperspectral unmixing and mathematical formulation

Since there are usually several materials contained in each pixel of a HSI, the spectrum acquired in each pixel is a mixture of the spectra of the individual materials. This calls for hyperspectral unmixing (HSU) methods, enabling to recover both the spectra of the individual materials present in the scene (the so-called endmembers) and their relative concentration in each pixel (the abundances).

Mathematically speaking, most HSU methods assume a linear mixture model: the considered dataset X (in which each column corresponds to a pixel of the original HSI) is assumed to have been generated as X = A*S* + N where A* is the sought-after endmember matrix and S* is the sought-after abundance matrix. The N matrix accounts for unknown additive noise. Since solving (1) is a severely ill-posed problem, further constraints have to be imposed to recover the A* and S* factors from the sole knowledge of X. Therefore, several families of HSU algorithms have emerged. While most of them are based on matrix factorization (ICA, NMF, sparse BSS), new promising methods rather rely on deep neural networks. Nevertheless, current deep learning methods are impeded by two problems: Pb-1) the lack of supervised datasets for training; Pb-2) since they only provide a point estimate, it is difficult to assess the estimate quality.

Internship objectives

Concerning Pb-1), due to the lack of supervised datasets, the authors often rely on simple reconstruction loss functions ||X-AS||, which is not fully satisfying as it can lead to spurious solutions. Consequently, the first research path of this internship is to consider better - self-supervised - methods to train HSU deep neural networks. Of particular interest to us, we will investigate HSU-specific contrastive-based training methods. In various domains, such as medical imaging, these approaches have obtained results in the unsupervised settings that are on par with the supervised ones. Generally speaking, the main insight of contrastive learning is to compare unlabeled data points against each other to teach a model which points are similar and which are different. In practice, a way of using contrastive learning which we will consider during this internship, is to train the considered neural networks to be robust to specific data transformations (for instance, rotations, crop, colorimetric perturbations...). It is expected that training so the architectures will lead to improved results compared to mere reconstruction errors. As a related alternative, we might also consider the equivariant imaging paradigm to perform self-supervised.

Concerning Pb-2), a challenge of deep-learning based HSU is to assess the reliability of the estimates provided by the considered neural network. Consequently, the second research path of this internship is to explore uncertainty quantification for HSU. We will also consider self-supervised training strategies, either the ones developed in the first part or the one we already developed.

Candidates

The candidate must be a Master 2 student (or equivalent) with a good knowledge of signal/image processing and machine learning. Ideally, the candidate will be familiar with the Python language (and in particular with pytorch). Knowledge in contrastive learning or hyperspectral imaging is a plus.

The candidate will acquire an expertise in remote sensing, deep learning and self-supervised training strategies. Such competences are largely sought-after, both in academia and in private companies, and are transferable to many other applications such as biomedical imaging, astrophysics... In addition, if relevant, the work conducted during the internship might lead to a scientific publication.

An extension of this subject to a PhD might be considered.

Contact

The internship (6 months) will take place in the IMAGES team (Télécom Paris) under the supervision of Christophe Kervazo.

- Contact: christophe.kervazo@telecom-paris.fr

- More informations on: https://sites.google.com/view/christophekervazo/