The key facts
- The SALF-Net model combines an acoustic block and a language model, working even without song lyrics
- Experts deemed 61.8% of its labels accurate; it makes half as many mistakes as Qwen-Audio and Audio Flamingo
- The lightweight version processes a clip in 7–11 ms; the label vocabulary is still fixed and English-language
A model for labeling the mood of short videos
The SALF-Net neural network assigns emotional labels to music in short videos — 'cheerful', 'romantic', 'energetic' — even if the recording is distorted and the song has no lyrics. The model was created by Liu Zhiqi from the Music College of Jimei University in China and his colleagues; the article was published in the journal Scientific Reports.
Why platforms need this
TikTok and similar platforms receive hundreds of millions of clips per day, and every track needs mood labels for search, recommendations and advertising. Human labelers cannot keep up with such a flow, while conventional audio analyzers are designed for full-length songs, not for 8–30 second snippets with overlaid voice and noise.
How the system is built
SALF-Net combines an acoustic block pre-trained on a large music collection and a language model that reads the song lyrics. During training, the text is periodically hidden, so the system does not lose functionality on instrumental tracks. An additional support is the Russell psychological scale: each emotion corresponds to coordinates 'pleasant — unpleasant' and 'calm — energetic'. According to the authors, 'the fusion of audio and text channels, in which the text can be switched off, turns a laboratory advantage into a system that withstands missing song lyrics, instrumental tracks and recordings overloaded with speech'.
Test results
The model was tested on a new collection of 30,000 clips labeled by humans. SALF-Net outperformed specialized systems and audio versions of the AI models Qwen-Audio from Alibaba and Audio Flamingo from NVIDIA. Experts deemed 61.8% of the issued labels accurately fitting, and the newcomer makes half as many mistakes as its competitors. Sometimes the system supplements the authors' tags with fitting neighbors: to 'energetic' music of a dance challenge it adds 'motivating' and 'party'. Song lyrics help most with labels like 'romance' and 'heartbreak', whereas tempo and energy are conveyed by the music even without words.
Speed and limitations
The lightweight version retains 95% of the full model's quality and spends 11 milliseconds per clip, and seven after additional compression. A single mid-range graphics card is enough to label more than 12 million clips per day. Limitations: the label vocabulary is fixed and English-language, and the system does not recognize fresh online moods.
