Multispectral video analysis for deformable texture recognition using a snapshot camera

25 Janvier 2024

---

Catégorie : Doctorant

---

Application only on

https://adum.fr/as/ed/voirproposition.pl?site=adumR&matricule_prop=52649#version

Classical methods for motion identification by video analysis try to estimate the motion of each surface element of the observed objects. During the movement of an object or the deformation of a texture, each moving surface element in the scene is projected onto different pixels in the successive images acquired by the camera. The main objective is then to retrieve the spatial coordinates of the pixel associated to the same given surface element in each of the images of the sequence. A common assumption to achieve this goal is that the energy reflected by each surface element is constant throughout the object displacement, hence the pixel associated to this element is also constant over all the image sequence. Classical motion analysis methods therefore use correlation measures or optical flow approaches to estimate motion [3].

Plant surfaces have deformable textures that can be observed in several spectral domains. Though the visible domain is easily accessible, near infrared allows one for a more efficient distinction of different plant species, such as weeds and crops. However, classical cameras (either gray-level or color ones) cannot acquire image sequences in the infrared domain.

Several snapshot multispectral cameras have been proposed in the literature in past years. Such scanless cameras acquire multispectral images at video rate. The so-called (M)SFA ((multi-)spectral filter array) technology has recently attracted the interest of the image processing community [4,5]. It is a generalization of the ground concept of single-sensor color cameras, namely a compromise between spatial and spectral resolutions. The sensor is covered by a mosaic of spectral filters, each of which being associated to a pixel and sensitive to a narrow spectral band. A single spectral component is therefore available at each pixel of the resulting image, such that this so-called raw (or MSFA) image has the same storage cost as a gray-level image. A multispectral video delivered by such a snapshot camera is thus made of a sequence of raw images acquired at video rate and stored as a gray-level image set.

The spectral component values that are missing in a raw image can then be estimated by demosaicing to reconstruct a fully-defined multispectral image [6,7]. The quality of the estimated multispectral image depends on the SFA, on the number of spectral bands, on their nature, and on the demosaicing procedure. Because demosaicing causes errors on the estimated values of missing components, this process degrades the quality of the texture representation with regards to raw images. Furthermore, a sequence of fully-defined multispectral images is highly memory greedy. We therefore propose to directly analyze raw image sequences to identify deformable textures. To our knowledge, very few works deal with multispectral video analysis [4,8,9]. This doctoral subject then tackles original open problems.

Motion identification by raw image analysis cannot rely on the hypothesis of gray-level constancy of the pixel associated to a given surface element in the successively acquired images. Indeed, a same moving surface element is likely to be presented by neighboring pixels in two successively acquired images, hence by raw values that are associated to different spectral bands. As a result, nor multi-scale approaches nor phase-based ones in the spatial frequency domain can be directly used to extract motion from raw image sequences.

Such raw image sequences will have to be analyzed by taking the SFA pattern of the camera into account. A first way is to proceed sparsely and to exploit raw images for keypoint detection [10], then to match the detected keypoints on the successive raw images. A second approach is to follow a dense strategy and to take inspiration from texture descriptors adapted to the SFA pattern to analyze textures from raw images, as designed by the Imagerie Couleur team [11]. Similarly, the challenge will be to design descriptors that are adapted to deformable textures that change over time.

Thesis planning

M1-M6 : Bibliography about motion estimation by classical video analysis methods and about formation and analysis of multispectal raw image sequences.

M7-M12 : Motion estimation by local correlation measures within raw multispectral image sequences.

M13-18 : Motion estimation by optical flow-based and phase-based analyses of multispectral videos.

M19-M24 : Design of texture descriptors by mono- and multi-scale analyses of raw image sequences. Tests on raw images either simulated from fully-defined multispectral images or acquired by IMEC snapshot mosaic camera available at the EquipEx IrDIVE-Continuum.

M25-M30 : Application to the recognition of moving plants.

M31-M36 : Manuscript writing.

 

References

[1] Anis Amziane. Texture features extracted from multispectral images acquired under uncontrolled illumination conditions—Application to precision farming. PhD thesis, Oct. 2022, https://www.theses.fr/s323059.

[2] Anis Amziane, Olivier Losson, Benjamin Mathon, Aurélien Duménil, and Ludovic Macaire. Reflectance Estimation from Multispectral Linescan Acquisitions under Varying Illumination—Application to Outdoor Weed Identification, Sensors, 2021, 21 (11), 3601, https://dx.doi.org/10.3390/s21113601

[3] Cédric Marinel, Benjamin Mathon, Olivier Losson, Ludovic Macaire. Comparison of Phase-based Sub-Pixel Motion Estimation Methods , Procs. IEEE International Conference on Image Processing (ICIP), Oct. 2022, Bordeaux, France. pp.561-565, https://doi.org/10.1109/ICIP46576.2022.9897338

[4] Kathleen Vunckx and Wouter Charle. Accurate Video-Rate Multi-Spectral Imaging Using IMEC Snapshot Sensors, Procs. 11th Workshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing (WHISPERS), 2021, pp. 1-7, https://doi.org/10.1109/WHISPERS52202.2021.9483975

[5] Ying Shen, Jie Li, Wenfu Lin, Liqiong Chen, Feng Huang, and Shu Wang. Camouflaged Target Detection Based on Snapshot Multispectral Imaging. Remote Sensing, 2021, 13 (19), 3949,https://doi.org/10.3390/rs13193949

[6] Sofiane Mihoubi, Olivier Losson, Benjamin Mathon, and Ludovic Macaire. Multispectral demosaicing using pseudo-panchromatic image, IEEE Transactions on Computational Imaging, 2017, 3 (4), pp. 982-995, https://doi.org/10.1109/TCI.2017.2691553

[7] Grigorios Tsagkatakis, Maarten Bloemen, Bert Geelen, Murali Jayapala, and Panagiotis Tsakalides, Graph and Rank Regularized Matrix Recovery for Snapshot Spectral Image Demosaicing, IEEE Transactions on Computational Imaging, 2019, 5 (2), pp. 2333-9403, https://doi-org/10.1109/TCI.2018.2888989

[8] Raffaele Vitale, Cyril Ruckebusch, Ingunn Burud, and Harald Martens. Hyperspectral Video Analysis by Motion and Intensity Preprocessing and Subspace Autoencoding, Frontiers in Chemistry, 2022,https://doi.org/10.3389/fchem.2022.818974

[9] Lulu Chen, Yongqiang Zhao, and Seong G. Kong. SFA-guided mosaic transformer for tracking small objects in snapshot spectral imaging, ISPRS Journal of Photogrammetry and Remote Sensing, 2023, pp. 223–236, https://doi.org/10.1016/j.isprsjprs.2023.09.015

[10] Xiangyu Zhang, Ling Zhang, and Xin Lou. A Raw Image-Based End-to-End Object Detection Accelerator Using HOG Features, IEEE Transactions on Circuits and Systems I: Regular Papers, 2022, 69 (1), pp. 322-333, https://doi.org/10.1109/TCSI.2021.3098053

[11] Sofiane Mihoubi, Olivier Losson, Benjamin Mathon, and Ludovic Macaire. Spatio-spectral Binary Patterns Based on Multispectral Filter Arrays for Texture Classification. Journal of the Optical Society of America A, 2018, 35 (9), pp.1532-1542, https://doi.org/10.1364/JOSAA.35.001532