Annonce

Les commentaires sont clos.

Multi-temporal and multi-modal Earth Observation latent space decoding using physically aware Deep Learning

19 December 2023

Catégorie : Doctorant

This offer is part of the RELEO (REpresentation Learning for Earth Observation) of ANITI-2, the follow-on of the Interdisciplinary Artificial Intelligence Institute in the frame of the French ANR "AI Clusters". The PhD will be funded by CNES (the French Space Agency) and Thales Alenia Space.

RELEO aims at building an AI foundation model for the exploitation of Earth Observation Satellite Image Time Series (EO SITS). This model will fuse multi-modal data (optical, SAR, thermal) into AI-Ready chunks of latent features (also known as embeddings), where traditional spatial, temporal and spectral dimensions of Earth Observation data have been collapsed. These latent chunks will benefit from the complementarity and the correlations of these different data sources and will provide essentialized (fully encoding the useful information) and strongly compressed information. The fusion will be done with deep neural networks whose training will be guided by physical models of the observed processes (bio/geo-physical models) and image formation models (radiative transfer and sensor models).

 

The proposed PhD subject will contribute to RELEO's WP3 whose goal is decoding the dimensionless compressed embeddings to produce time series of input variables for the physical models used to constrain the training problem through self-supervision. Some of these variables will be Essential Climate and Biodiversity Variables, as for instance soil moisture, land surface temperature or leaf area index. The work consists in building a neural decoder for the data cube unfolding: going from the compressed multi-modal data fusion back to the temporal and spatial resolutions needed by downstream applications. The decoder training will be done through the optimization of the above-mentioned physical models' output. Reference bio/geo-physical products from operational mono-sensor processing chains will be used together with field surveys to validate the decoder outputs.

To give a concrete example, one can start with 2 sources of data, high-resolution optical Sentinel-2 (10 m. resolution every 5 days) and radar Sentinel-1 (10 m. resolution every 6 days) time series covering a full country like France. These are encoded as spatio-temporal chunks of 1km × 1km × 10 days as vectors of 256 features. The goal is to decode these vectors in order to produce weekly Leaf Area Index maps at 10 m. resolution.

The originality of the proposed approach is two-fold.

  • First of all, the use of physical models to constrain the learning (PINN, Physically Informed Neural Networks) is very recent in EO and its extension to the multi-modal case is a challenging problem, but it will allow the development of downstream application processors which are independent of the availability of a particular sensor.
  • Second, the generation of data at tailored temporal and spatial resolutions (including spatial resolutions not acquired at the chosen time stamps) allows to foresee an accuracy improvement of the downstream products by avoiding repeated resampling steps which are known to degrade the input data. This will entail the development of neural architectures which are able to produce image time series from embeddings which don’t have explicit spatial nor temporal dimensions. Generative models either based on diffusion approaches, normalizing flows or similar approaches will be investigated.

Work plan:

  1. State of the art on foundation models and PINN starting from what is already available at CESBIO.
  2. Method development: latent space decoding for on-demand resolution using physical model guided training.
  3. Validation and assessment using reference products and field surveys.

The PhD will take place at Cesbio[1] in Toulouse. The PhD candidate will be integrated into the /Observation Systems/ team and more precisely, within the AI unit.

The team works on CNES' (the French Space Agency) high performance computing (HPC) infrastructure (250 nodes with 8000 CPU, 53 GPU) which also hosts a full mirror of all Sentinel-1 and Sentinel-2 data.

Candidate profile: Masters in at least one of the following areas: applied mathematics, physics of measure, optimization, machine learning. Skills in and eagerness for computer programming in the areas of scientific computing or machine learning.

Send Curriculum Vitae, motivation letter and recommendation letters to jordi.inglada@cesbio.eu before February 28 2024.

[1] <https://www.cesbio.cnrs.fr/homepage/>