Contact : Théo Mariotte (LIUM, Le Mans Université) theo.mariotte@univ-lemans.fr
Context
What if machines could learn to understand speech and sound without anyone telling them what to listen for?
This is the promise of Self-Supervised Learning (SSL): a model learns a rich representation of audio directly from unlabeled data, by solving a pretext task built from the data itself. The resulting representation can then be transferred to many downstream tasks.
In audio, models such as WavLM [1] for speech and BEATs [2] for sound scenes have become true backbones, powering speech recognition, speaker recognition and sound event detection. But this success comes at a price. These models contain hundreds of millions of parameters and rely on stacks of Transformer blocks that are expensive to run, which puts them out of reach of CPU-only and edge devices. The challenge is clear: how can we make these models small and fast without losing their versatility?
This internship tackles this question for both speech and sound scenes.
Objectives of the internship
The intern will explore the compression of SSL audio models through knowledge distillation, where a compact student learns from the predictions or representations of a large teacher. Existing work on speech [3-6] mostly targets HuBERT [7] with Transformer students, so the compressed models remain costly at inference. For general audio, a recent study [8] shows that small students can match their teacher, but only on tasks that are not temporally aware (one label per audio clip). Much remains to be explored.
The main intership objective is to push this further with the Polynomial Mixer (PoM) [9], a linear-complexity alternative to multi-head attention. The student will design student architectures built on PoM and distill state-of-the-art SSL models: HuBERT [7], MelHuBERT [10] and WavLM [1] for speech, as well as BEATs [2] and MATPAC++ [11] for general audio. This internship builds on results obtained last year, so you will not start from scratch. It is likely that the results lead to a paper publication in a conference of the domain (ICASSP, Interspeech).
Environment
Location and working environment The internship will take place in Le Mans. The intern will have an office at the Computer Science Laboratory (LIUM) and supervised by Théo Mariotte and Nicolas Dugué, both assistant professors in the lab.
The intership is co-supervised by Alexandre Brouste from the Mathematics Laboratory (LMM). This is a great opportunity to work alongside researchers from a different field and broaden your scientific perspective.
Why Le Mans?
Le Mans may not be a household name, but it is well worth discovering. It is a human-scale city where everything is easily accessible by bike/tram, with a lively cultural scene that includes the Théâtre des Quinconces, Superforma, the Jazz Tangentes festival and Le Mans Fait Son Cirque, among others.
It is also one of the most affordable cities in France, so rent remains low compared with other cities. Finally, Le Mans is very well connected by train: Paris is just 1 hour away, Lyon 3 hours, Marseille 4 hours, and many other destinations are within easy reach.
Funding The internship is funded by Louis Bachelier institute in the context of the Numerical Efficiency and Sobriety chair.
References
[1] Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li,
Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. Wavlm: Large-scale self-supervised pre-
training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing,
16(6):1505–1518, 2022.
[2] Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu, Daniel Tompkins, Zhuo Chen, and Furu
Wei. Beats: Audio pre-training with acoustic tokenizers. arXiv preprint arXiv:2212.09058,
2022.
[3] Heng-Jui Chang, Shu-wen Yang, and Hung-yi Lee. Distilhubert: Speech representation learn-
ing by layer-wise distillation of hidden-unit bert. In ICASSP 2022-2022 IEEE international
conference on acoustics, speech and signal processing (ICASSP), pages 7087–7091. IEEE, 2022.
[4] Yeonghyeon Lee, Kangwook Jang, Jahyun Goo, Youngmoon Jung, and Hoirin Kim. Fithubert:
Going thinner and deeper for knowledge distillation of speech self-supervised learning. arXiv
preprint arXiv:2207.00555, 2022.
[5] Tzu-Quan Lin, Tsung-Huan Yang, Chun-Yao Chang, Kuang-Ming Chen, Tzu-hsun Feng, Hung-
yi Lee, and Hao Tang. Is smaller always faster? tradeoffs in compressing self-supervised
speech transformers. In 2025 IEEE Automatic Speech Recognition and Understanding Workshop
(ASRU), pages 1–7. IEEE, 2025.
[6] Hyung Gun Chi, Zakaria Aldeneh, Tatiana Likhomanenko, Oggi Rudovic, Takuya Higuchi, Li-
Wei Chen, Shinji Watanabe, and Ahmed Hussen Abdelaziz. Dicehubert: Distilling hubert with
a self-supervised learning objective. arXiv preprint arXiv:2507.02911, 2025.
[7] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdi-
nov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by
masked prediction of hidden units. IEEE/ACM transactions on audio, speech, and language
processing, 29:3451–3460, 2021.
[8] Mohammed Ali El Adlouni, Aurian Quelennec, Pierre Chouteau, Geoffroy Peeters, and Slim
Essid. S-sondo: Self-supervised knowledge distillation for general audio foundation models. In
ICASSP 2026-2026 IEEE International Conference on Acoustics, Speech and Signal Processing
(ICASSP), pages 15942–15946. IEEE, 2026.
[9] David Picard, Nicolas Dufour, Lucas Degeorge, Arijit Ghosh, Davide Allegro, Tom Ravaud,
Yohann Perron, Corentin Sautier, Zeynep Sonat Baltaci, Fei Meng, et al. Pom: A linear-time
replacement for attention with the polynomial mixer. arXiv preprint arXiv:2604.06129, 2026.
[10] Tzu-Quan Lin, Hung-yi Lee, and Hao Tang. Melhubert: A simplified hubert on mel spectro-
grams. In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU),
pages 1–8. IEEE, 2023.
[11] Aurian Quelennec, Pierre Chouteau, Geoffroy Peeters, and Slim Essid. Matpac++: Enhanced
masked latent prediction for self-supervised audio representation learning. arXiv preprint
arXiv:2508.12709, 2025.
