This PhD concerns the end-to-end design of a camera and deep learning algorithm for urban scene analysis while preserving privacy. The laboratory is ONERA/DTIS/MIC at Palaiseau in collaboration with the company Upciti. To apply to this offer, please send a resume and a cover letter to : pauline.trouve@onera.fr and marcela.carvalho@upciti.com.
Cameras are ubiquitous in the public space. The images captured are usually adopted as input to different deep models for object detection, object tracking, or semantic segmentation. These tasks can also include extracting privacy-sensitive information which raises legal issues related to data protection. Indeed, some software-based techniques, such as blurring, can be used to hide sensitive information, but, they are prone to error and do not include a level of security on the camera’s raw output. Another solution is to add security directly to the camera lens. In this regard, recent works [1,2] propose to optimize lens parameters to add privacy while simultaneously ensuring the objective task still achieves high performance. To design such a camera, a co-design approach is required to jointly optimize the optical and neural network parameters. The idea is to combine deep co-design and adversarial training to design an optical element that produces an image that favors a scene analysis task (utility task) but compromises a task that could affect privacy (adversarial task).
However in the state of the art, only Fourier Optics based optical model have been used for the camera privacy preserving design, assuming small field of view (FOV) and only considering the optimization of a single optical element such as a phase mask. Specifically ONERA has been working on the end-to-end design of lenses and neural networks with a differentiable optical model using ray tracing (Formidable) [3,4,5]. This model takes as input an optical system defined by a set of lenses and simulates its impulse response as well as its Jacobian with respect to the lens parameters. Using this tool, we have performed the joint design of the lens parameters for a single task such as image restoration [4,5] and both depth estimation and image restoration [5]. However these works only considered optimization of a limited set of refractive lenses parameters, within a restricted FOV.
In contrast, the aim of this thesis is the end-to-end design of privacy preserving camera with a large FOV and a differentiable optical model based on ray tracing. This work is part of an ANR project in collaboration between ONERA and Upciti, a private company that develops smart city solutions. The PhD will be organized as follows: The first aim will be to replace the Fourier Optical model used in the preliminary work conducted by Upciti and ONERA [2] with the differentiable ray tracing software Formidable.
Within this work new investigations on the adversarial learning framework will be conducted, such as how to make it general to non-sensitive purposes. Then, the second challenge will be to increase the camera FOV as well as optimization of a large set of optical parameters. To handle this challenging task, curriculum learning could be investigated as proposed in the literature [6]. The last objective is to integrate the optimization of unconventional lenses such as phase masks or freeform lenses into the end-to-end joint optical and neural network optimization framework, using a ray tracing optical model such as Formidable. Indeed, these unconventional lenses have been introduced in a joint optical/neural network design, as they provide additional degrees of freedom to control the response of the lens for a given application [7,8].
The last part of the thesis will be dedicated to the realization of a demonstrator for a selected application.
Bibliography
[1] C. Hinojosa et al., “Learning privacy-preserving optics for human pose estimation”, ICCV, 2021.
[2] M. Dufraisse et al., (2024). « Physics Based Camera Privacy: Lens and Network Co-Design to the Rescue, » CVPRW 2024.
[3] A. Halé et al., “End-to-end sensor and neural network design using differential ray tracing. Optics express, 29(21), 2021.
[4] M. Dufraisse et al., (2023). Deblur or denoise: the role of an aperture in lens and neural network co-design. Optics Letters, 48(2), 231-234.
[5] M Dufraisse et al. “Multitask deep co-design for extended depth of field and depth from defocus”, Proc SPIE, 2024.
[6] X. Yang et al., “Curriculum learning for ab initio deep learned refractive optics “, Nature communications, 2024.
[7] S. Elmalem et al., “Learned phase coded aperture for the benefit of depth of field extension,” Opt. Express 26, 2018.
[8] J. Chang and G. Wetzstein, “Deep optics for monocular depth estimation and 3d object detection,” ECCV 2019
