Automatic music emotion tag generation for short-video platforms: a self-supervised audio-lyric fusion network with valence-arousal regularization

Background music serves as a primary carrier of emotional narrative on short-video platforms, where hundreds of millions of user-generated clips are uploaded daily and organized, searched, and recommended largely through metadata. Emotion-related tags, however, are still produced mainly by manual annotation or by generic audio taggers that were never designed for fifteen-to-sixty-second clips recorded in uncontrolled environments. This paper addresses the problem of automatic music emotion tag generation for short-video platforms, a setting characterized by truncated clips, ubiquitous audio degradation, and unreliable lyric availability.

To tackle these challenges, we propose SALF-Net, a self-supervised audio-lyric fusion network specifically designed to be robust to modality absence and input noise. The audio branch adapts a music-domain self-supervised transformer encoder to clip-level inputs through masked segment sampling and statistics pooling, while the lyric branch encodes automatically transcribed or metadata-provided lyrics with a pretrained language model. A gated cross-modal fusion module with modality dropout keeps the system functional when lyrics are missing, which is common for instrumental tracks and noisy recordings.

A multi-task objective jointly optimizes multi-label tag prediction and valence-arousal regression, so that the psychological structure of emotion space acts as a regularizer for tag geometry. Experiments on the mood and theme subset of MTG-Jamendo, on the DEAM dimensional emotion benchmark, and on a newly constructed short-video test collection of 30,000 clips show that the proposed model reaches a precision-recall AUC of 0.389 and an ROC-AUC of 0.921, outperforming strong self-supervised baselines by a clear margin, while a distilled student variant preserves 95% of this quality at 2% of the teacher’s latency. The results indicate that accurate, robust, and deployable emotion tagging is feasible at short-video scale without relying on platform-private annotation budgets.

The authors received no specific funding for this work. College of Music, Jimei University, Xiamen, 361021, China Obstetrics and Gynecology Department, Xiang’an Hospital of Xiamen University, Xiamen, 361000, China The authors declare no competing interests. The annotation and evaluation procedures were reviewed by the Jimei University Ethics Committee and were exempted from formal ethical approval.

All methods were performed in accordance with relevant guidelines and regulations. No personally identifiable or sensitive information was collected, and informed consent was obtained from all participants. This article does not contain any identifiable personal information, images, or individual-level data requiring consent for publication.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it.