M2 internship at CESI LINEACT "Evaluation and Application of Foundation Models for 6D Pose Estimation in Industrial Environments Using Digital Twins"

19 Novembre 2024

---

Catégorie : Stagiaire

---

Title: M2 Internship –Evaluation and Application of Foundation Models for 6D Pose Estimation in Industrial Environments Using Digital Twins

Scientific Fields: Computer Vision, Industry of the Future

Keywords: Foundation Models, 6D pose estimation, Digital Twin, Synthetic and Hybrid Dataset, Manufacturing environments

Supervision

Name	Position, Title	@
Nicolas Ragot	Associate professor	nragot@cesi.fr
Vincent Havard	Associate professor, HDR	vhavard@cesi.fr

Works

Details of the tasks

This M2 internship is part of the FUSION project and its Work Package 3 (WP3), whose goal is to update the digital twin of an industrial environment based on the robot vision-based perception.

Foundation models are increasingly used in the literature across a wide range of applications. Also, this is the case in 6D pose estimation, with Wen et al.’s proposal, titled FoundationPose, which achieves excellent results compared to state-of-the-art methods. The approach has been tested on various datasets.

We aim to evaluate its performance in the context of manufacturing industrial environments through training performed using the digital twin of the production workshop.

The tasks assigned to the intern are as follows:

• Implement FoundationPose.
• Generate a synthetic dataset of the production workshop using the digital twin developed in Unity.
• Train FoundationPose on this data and evaluate the algorithm's performance.
• Evaluate the algorithm's performance on real-world data.
• If applicable, adjust the initial dataset by incorporating labeled real-world data into the synthetic dataset and assess the performance of the hybrid dataset.

Project Context

This recruitment is part of the FUSION project (Framework for Universal Software Integration in Open Robotics), which was selected under the "I-Démo – France 2030 Regionalized Normandie" call for projects. The project's partners are Conscience Robotics (lead), OREKA Ingénierie, and CESI LINEACT.

The main objective of the FUSION project is to democratize the use of robotics by introducing a paradigm shift that places the user at the center of the system through:

• The introduction of XR for designing robotic missions and teleoperating robots via a digital twin;
• The reuse and sharing of software modules accessible to everyone;
• An innovative robotic perception approach using a semantic map to update the digital twin, making robots increasingly autonomous.

The targeted use case focuses on dismantling operations within a nuclear site cell, specifically the cutting of contaminated pipelines. Currently, these operations are carried out by operators remotely controlling the robotic arm using only cameras installed on the intervention site and mounted on the robotic arm. This significantly complicates teleoperation due to the lack of depth perception.

Our proposal aims, first, to reduce the complexity of robot teleoperation by replacing environment perception through cameras with immersion in a real-time-generated digital twin of the work area. Secondly, the project seeks to "teach" robots naturally to perform repetitive tasks that require only occasional supervision.

Work Program

	Work Period
Literature review on recent approach on 6D pose estimation
Implement FoundationPose
Generate a DT synthetic dataset, train the model and evaluate the performances
Evaluate the performances on the real data from the Physical Twin
Adjust the initial dataset by incorporating labeled real-world data and evaluate the peformances

Expected Scientific/Technical Production

• L.D1 : 1 software module related to the implementation of FoundationPose
• L.D2: 1 synthetic dataset + 1 real dataset (optional)
• L.A1: 1 report summarizing the performances of FoundationPose in an industrial manufacturing environment (possibility of a publication depending on the student internship’s progress)

Laboratory Presentation

CESI LINEACT (UR 7527), the Digital Innovation Laboratory for Businesses and Learning in support of Territorial Competitiveness, anticipates and supports technological transformations in sectors and services related to industry and construction. CESI's historical ties with businesses are a determining factor in its research activities, leading to a focus on applied research in partnership with industry. A human-centered approach coupled with the use of technologies, as well as regional networking and links with education, have enabled cross-disciplinary research that centers on human needs and uses, addressing technological challenges through these contributions.

Its research is organized into two interdisciplinary scientific teams and two application domains:

· Team 1, "Learning and Innovating," is primarily focused on Cognitive Sciences, Social Sciences, Management Sciences, Education Science, and Innovation Sciences. The main scientific objectives are understanding the effects of the environment, particularly instrumented situations with technical objects (platforms, prototyping workshops, immersive systems), on learning, creativity, and innovation processes.

· Team 2, "Engineering and Digital Tools," is mainly focused on Digital Sciences and Engineering. Its main scientific objectives include modeling, simulation, optimization, and data analysis of cyber-physical systems. Research also covers decision-support tools and studies of human-system interactions, especially through digital twins coupled with virtual or augmented environments.

These two teams cross and develop their research in the two application domains of Industry of the Future and City of the Future, supported by research platforms, primarily the Rouen platform dedicated to the Factory of the Future and the Nanterre platform dedicated to the Factory and Building of the Future.

Organization

Funding: CESI Nord-Ouest (FUSION project, i-démo régionalisé co-funded by Région Normandie and the European Union)

Workplace : Rouen Campus (in Saint Etienne du Rouvray)

• Start date: February 2025
• Duration: 6-month internship

Your Recruitment

Profile Sought: Master's in Computer Science with a focus on artificial intelligence, computer vision.

Skills:

Scientific and technical skills:

Skills	Technical stack		Operating Systems
Artificial Intelligence and Computer Vision	Python & C++ C# (optional)	PyTORCH, DOCKER UNITY (optional)	LINUX & WINDOWS

Interpersonal Skills:

· Autonomy, initiative, curiosity

· Teamwork ability and good interpersonal skills

· Rigorousness

Application Process: by dossier and interview.

Send your application to Nicolas Ragot (nragot@cesi.fr), Vincent Vauchey (vhavard@cesi.fr), with the subject line: "[Application] Title on page 1".

Your application should include:

• A detailed CV. If there is a gap in your academic career, please provide an explanation.
• A motivation letter explaining your interest in pursuing a doctoral thesis.
• A link to your previous works on GitHub would be an asset.
• Any other documents you consider useful.

Please send all documents in a single zip file named LASTNAME_firstname.zip