TY - GEN
T1 - GMP-TL
T2 - 2024 IEEE Spoken Language Technology Workshop, SLT 2024
AU - Pan, Yu
AU - Yang, Yuguang
AU - Huang, Yuheng
AU - Jin, Tiancheng
AU - Yin, Jingjing
AU - Hu, Yanni
AU - Lu, Heng
AU - Ma, Lei
AU - Zhao, Jianjun
N1 - Publisher Copyright:
© 2024 IEEE.
PY - 2024
Y1 - 2024
N2 - The continuous evolution of pre-trained speech models has greatly advanced Speech Emotion Recognition (SER). However, current research typically relies on utterance-level emotion labels, inadequately capturing the complexity of emotions within a single utterance. In this paper, we introduce GMP-TL, a novel SER framework that employs gender-augmented multi-scale pseudo-label (GMP) based transfer learning to mitigate this gap. Specifically, GMPTL initially uses the pre-trained HuBERT, implementing multi-task learning and multi-scale k -means clustering to acquire frame-level GMPs. Subsequently, to fully leverage frame-level GMPs and utterance-level emotion labels, a two-stage model fine-tuning approach is presented to further optimize GMP-TL. Experiments on IEMOCAP show that our GMP-TL attains a WAR of 80.0% and an UAR of 82.0%, achieving superior performance compared to state-of-the-art unimodal SER methods while also yielding comparable results to multimodal SER approaches.
AB - The continuous evolution of pre-trained speech models has greatly advanced Speech Emotion Recognition (SER). However, current research typically relies on utterance-level emotion labels, inadequately capturing the complexity of emotions within a single utterance. In this paper, we introduce GMP-TL, a novel SER framework that employs gender-augmented multi-scale pseudo-label (GMP) based transfer learning to mitigate this gap. Specifically, GMPTL initially uses the pre-trained HuBERT, implementing multi-task learning and multi-scale k -means clustering to acquire frame-level GMPs. Subsequently, to fully leverage frame-level GMPs and utterance-level emotion labels, a two-stage model fine-tuning approach is presented to further optimize GMP-TL. Experiments on IEMOCAP show that our GMP-TL attains a WAR of 80.0% and an UAR of 82.0%, achieving superior performance compared to state-of-the-art unimodal SER methods while also yielding comparable results to multimodal SER approaches.
KW - Speech emotion recognition
KW - gender-augmented
KW - model fine-tuning
KW - multi-scale pseudo-label
KW - transfer learning
UR - https://www.scopus.com/pages/publications/85217435528
UR - https://www.scopus.com/pages/publications/85217435528#tab=citedBy
U2 - 10.1109/SLT61566.2024.10832339
DO - 10.1109/SLT61566.2024.10832339
M3 - Conference contribution
AN - SCOPUS:85217435528
T3 - Proceedings of 2024 IEEE Spoken Language Technology Workshop, SLT 2024
SP - 496
EP - 501
BT - Proceedings of 2024 IEEE Spoken Language Technology Workshop, SLT 2024
PB - Institute of Electrical and Electronics Engineers Inc.
Y2 - 2 December 2024 through 5 December 2024
ER -