Gaussian Process Prior Variational Autoencoders for Earth Data Time Series Anlaysis

18 December 2023

---

Catégorie : Stagiaire

---

Context

Over the last ten years, Earth Observation (EO) has made enormous advances in terms of spatial and temporal resolutions, data availability and open policies for end-users. The increasing availability of complementary imaging sensors allows land ecosystems state variables and processes to be observed at different spatio-temporal scales. Big EO data can thus enable the design of new land monitoring systems providing critical information in order to guide climate change monitoring, mitigation and adaptation.

Conventional machine learning methods are not well adapted to the complexity of multi-modal, multi-resolution satellite image time series (SITS) with irregular sampling. Therefore they are not suitable for extracting and processing all the relevant information. On the other hand, methods based on deep neural networks have shown to be very effective to learn low-dimensional representations of complex data for several tasks and come with high potential for EO data. However, they often emerge from the computer vision (CV) and natural language processing (NLP) communities and need to be extended and properly instantiated to handle the very specificities of Earth Observation data.

Previous works at the CESBIO-lab have shown that generative encoder-decoder architectures such as the Variational Auto-Encoder (VAE) or the U-NET models perform very well for a variety of EO tasks: estimation of biophysical parameters or Sentinel-1 to Sentinel-2 translations, to cite a few.

However, such approaches appear to be inadequate to handle data coming from more than 2 sources and acquired at different time and spatial resolutions, as prioritized in the RELEO chair within ANITI. In particular, the generative capability of these models may generalize poorly to unseen regions or temporal periods. Processing such streams of data requires to jointly encode all sources into a structured latent space where each complementary information carried by each source can be embedded while ensuring long-term encoding of newly acquired data (from possibly new sensors).

 

Keywords

Machine learning, remote sensing, Earth observation, signal & image processing.

 

Work-plan

The objective of this internship is to investigate Gaussian process (GP) prior for Variational Auto-Encoders (VAEs). Usually, VAEs assume independence between samples. This assumption is generally made for sake of simplicity and computational efficiency of the training and inference steps. However, assuming independence of samples amount to ignoring the correlation between adjacent pixels in the temporal domains. Furthermore, because of the very deterministic nature of such neural networks architectures, they do not properly encode uncertainty related to missing/noisy data.

Adopting this GP prior is expected to model correlations between times. However, due to the irregular and unaligned nature of SITS and their massive volume, approximation are required to maintain fast training and inference.

The work-plan of this Master internship is as follows:
- Define GP prior VAEs for pixel satellite time series with different approximations,
- Implement the models in PyTorch,
- Perform experiments on massive SITS and compare with others VAEs on downstream tasks.

 

Scientific environment

The M.Sc. student will benefit from a favorable context and will be able to rely on the most recent results and advances in machine learning and signal & image processing. He/she will be mainly co-advised by the following researchers within the CESBIO-lab:
- Mathieu Fauvel, INRAe Researcher
- Nicolas Dobigeon, Professor at Toulouse INP
- Julien Michel, CNES Engineer

 

Period and continuation as a Ph.D. thesis

This internship shall take place in 2024, from February for a 6-month duration. The precise starting and ending dates can be adjusted according to the availability of the selected candidate. The grant is about of 620 euros per month.

A Ph.D. position co-funded by CNES (French space agency) will be considered as a possible continuation of this M.Sc. training period, as part of the AI Research Chair RELEO within ANITI.

 

Profile & requirements

Master or Engineering school students with major in applied mathematics, computer science or electrical engineering.

The knowledge needed for this work includes a strong background in machine learning or data science, signal & image processing or remote sensing data processing. Good scientific programming skills (e.g., Python) and good communication skills in English, both written and oral are also expected. Interests in Earth observation will be appreciated.

 

Contact & application procedure

Applicants are invited to send (as pdf files)
- a detailed curriculum,
- official transcripts from each institution you have attended (in French or English).
to the co-advisors
- Mathieu Fauvel, mathieu.fauvel@inrae.fr
- Nicolas Dobigeon, nicolas.dobigeon@toulouse-inp.fr
- Julien Michel, julien.michel4@univ-tlse3.fr

You will be contacted if your profile meets the expectations. Review of applications will be closed when the position is filled.