Semantic image interpretation using a neurosymbolic approach
26 Juillet 2024
Catégorie : Post-doctorant
Semantic image interpretation consists in extracting structured semantic descriptions from data. The aim of this work is to propose semantic image interpretation methods that recognize patterns (objects, environments, logos, etc.), determine their relationship in the image, and semantically interpret the scene to build high-level knowledge. The use of neuro-symbolic approaches [2], such as Logic Tensor Networks (LTN) [3], is envisaged for this task.
Semantic image interpretation consists in extracting structured semantic descriptions from data. The aim of this work is to propose semantic image interpretation methods that recognize patterns (objects, environments, logos, etc.), determine their relationship in the image, and semantically interpret the scene to build high-level knowledge. The use of neuro-symbolic approaches [2], such as Logic Tensor Networks (LTN) [3], is envisaged for this task.
The context of the study is an ANR project [1] which proposes to analyze, through the prism of three major crises (asylum, Covid-19, war in Ukraine), the interactions between international migration and the associated inequalities. The notion of a migratory journey implies the idea of continuity, without being synonymous with linearity or a predefined direction. Migrants share itineraries marked out by places and evolve within one or more collectives, notably through images. These images, made up of objects (tents, vehicles, everyday objects, etc.) and references (logos, landscapes, etc.) referred to under the generic term of "signs", enable to reconstruct a reality, that of the migrant's daily life, to be compared and mirrored with official discourse and images. The proposed approach is designed to :
- identify these signs in images: this will involve the use of object detection and recognition techniques, with a focus on deep learning approaches. A state-of-the-art review of classic methods (R-CNN family, Panoptic Feature Pyramid Networks, PointRend, OneFormer, SAM, etc.) will enable to identify, depending on the context, an efficient algorithm for detecting objects of interest,
- integrate expert knowledge, using first-order logic, into a tensor formalism derived from LTNs,
- train this neurosymbolic approach to determine links between signs,
- deduce a semantic interpretation to "tell" the migrant's story.
This story will then be compared and mirrored to official speeches and images.
Interactions: Regular meeting are expected with partners of the ANR, and especially social science researchers.
Applicant’s profile: PhD in Computer Science with background in machine/deep learning. Good programming skills in Python. French skills are not required.
Application procedure: Applications must be sent by email to Vincent BARRA (vincent.barra@uca.fr), and have to include:
- a detailed resume (in pdf)
- a motivation letter (in pdf)
- the names and complete addresses of two reference persons.
The email subject has to be **Postdoc HYCI Application**
Net salary: 2302€ / month
Contact: Vincent BARRA(mail:vincent.barra@uca.fr)
LIMOS, UMR 6158 CNRS
1 rue de la chébarde
63178 AUBIERE CEDEX
Références
[1] https://anr.fr/Projet-ANR-22-CE55-0010
[2] I Donadello, L Serafini, A d'Avila Garcez, Logic Tensor Networks for Semantic Image Interpretation, Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence (IJCAI-17)
[3] S Badreddine, A d'Avila Garcez, L Serafini, M Spranger, Logic Tensor Networks, Artificial Intelligence (303), 2022