HeartMuLa Logo

HeartMuLa: A Family of Open Sourced Music
Foundation Models

Abstract

We present a family of open-source Music Foundation Models designed to advance large-scale music understanding and generation across diverse tasks and modalities. Our framework consists of four major components:

  • HeartCLAP: an audio–text alignment model that establishes a unified embedding space for music descriptions and cross-modal retrieval.
  • HeartCodec: a low-frame-rate (12.5 Hz) yet high-fidelity music codec tokenizer that captures long-range musical structure while preserving fine-grained acoustic details and enabling efficient autoregressive modeling.
  • HeartTranscriptor: a robust lyric recognition model optimized for real-world music scenarios.
  • HeartMuLa: an LLM-based song generation model capable of synthesizing high-fidelity music under rich, user-controllable conditions. HeartMuLa supports multi-conditional music creation with inputs including textual style descriptions, lyrics, and reference audio

In addition, it provides two specialized modes:

  • fine-grained musical attribute control, which allows users to specify the style of different song sections (e.g., intro, verse, chorus) using natural language prompts.
  • short, engaging music generation, which is suitable as background music for short videos.

Together, this suite of models forms an extensible open-source ecosystem that unifies music understanding, alignment, and controllable generation. We expect these foundation models to serve as strong baselines for future research and to facilitate practical applications in multimodal content production.

Framework

HeartMuLa Framework

Figure 1: The architecture of HeartMuLa.

Video

Song Generation Comparison

Tag (Input) Lyrics (Input) Generated Audio Samples
English
warm, reflection, pop, Cafe
Click to Collapse Lyrics
[Verse]
Sunlight spills across the scratched oak floor
I’m not checking on the time anymore
Just watching steam spiral up from the tea
A quiet moment meant only for me

[Chorus]
It’s a slow motion kind of afternoon
Humming along to a coffee shop tune
Let the traffic spin outside the glass
While I let the heavy hours pass

[Verse]
A stranger smiles as they turn a fresh page
We’re all taking a break from the stage
The scent of roasted beans fills up the air
Settling the dust of a worry or care

[Bridge]
I used to run until my lungs would burn
Now I’m sitting still to live and learn
Just breathing it in

[Chorus]
It’s a slow motion kind of afternoon
Humming along to a coffee shop tune
Let the traffic spin outside the glass
While I let the heavy hours pass

[Outro]
Just warmth in a cup
No need to look up
Suno v4.5 Mureka V7.6 YUE
DiffRhythm2 ACE-Step HeartMuLa
R&B, Keyboard, Regret, drum machine, electric guitar, synthesizer, soft
Click to Collapse Lyrics
[Chorus]
Standing on the podium with lights in my eyes
Finally broke the ceiling and touched the skies
They said I wouldn't make it but here I stand
The whole world is waiting in the palm of my hand

[Verse]
Sweat on the concrete and blood on the floor
Knocking 'til my knuckles bled on the door
Every time I fell down I got up twice
Paid the heavy toll yeah I paid the price

[Prechorus]
Now the doubt is fading out
Hear the crowd begin to shout
This is what it's all about

[Chorus]
I'm a winner baby look at me now
Wearing the gold and taking the bow
Nothing can stop this runaway train
Dancing through the fire and the rain

[Bridge]
It wasn't luck it was destiny
Unlocking the chains setting me free
Top of the mountain looking down
This is my kingdom this is my town

[Chorus]
Yeah I'm a winner see the flag unfold
Turning the dust into solid gold
History written in a single day
I did it my own specific way

[Outro]
Victory tastes sweet
Can't be beat
Suno v4.5 Mureka V7.6 YUE
DiffRhythm2 ACE-Step HeartMuLa
energetic, electronic, synthesizer, drum machine, self-discovery
Click to Collapse Lyrics
[Verse]
Wired to the grid,I was playing a part
Just a steady signal with a hollow heart
But the neon lights are flashing in my eyes
Cutting through the static and the old disguise

[Prechorus]
Turn the voltage up,watch the meters redline
I am stepping out of the rigid design
The bass is kicking in,shaking off the rust
Turning all the shadows into diamond dust

[Chorus]
I’m finding my frequency,loud and clear
Nothing left to hide,nothing left to fear
The rhythm is a mirror showing who I am
Breaking through the walls of the hologram
Yeah,I’m electric,I’m finally me
Dancing on the edge of reality

[Verse]
Synthesized horizons glowing in the dark
Every single beat is a brand new spark
I don’t need a map for the road ahead
I’ll follow where the pulse of the night is led

[Bridge]
No more glitches,no more doubt
Let the energy scream out
This is what it’s all about

[Chorus]
I’m finding my frequency,loud and clear
Nothing left to hide,nothing left to fear
The rhythm is a mirror showing who I am
Breaking through the walls of the hologram
Yeah,I’m electric,I’m finally me
Dancing on the edge of reality

[Outro]
I am the sound
I am the light
Burning bright
Suno v4.5 Mureka V7.6 YUE
DiffRhythm2 ACE-Step HeartMuLa
soft, Sad, Ballad, Longing
Click to Collapse Lyrics
[Verse]
The coffee cup is sitting cold upon the sill
The dust motes dancing in the light are standing still
I trace the pattern of the rain against the glass
Just waiting for another lonely hour to pass

[Prechorus]
And I swear I heard you calling out my name
But it’s just the wind playing its wicked game
Turning whispers into ghosts that look the same

[Chorus]
I’m reaching out to space that you used to fill
Bargaining with time to just keep it still
If longing was a river I would drown inside
With nowhere left to run and nowhere left to hide
I’m missing you in ways I can’t describe

[Verse]
Your jacket’s hanging by the door just like before
A silent sentry to the life we have no more
I try to read the book you left turned on its face
But every word just takes me back to your embrace

[Chorus]
I’m reaching out to space that you used to fill
Bargaining with time to just keep it still
If longing was a river I would drown inside
With nowhere left to run and nowhere left to hide
I’m missing you in ways I can’t describe

[Outro]
So I’ll turn out the light and close my eyes
And meet you in the quiet where the daylight dies
Suno v4.5 Mureka V7.6 YUE
DiffRhythm2 ACE-Step HeartMuLa
Chinese
meditation, faith, acoustic, peaceful
Click to Collapse Lyrics
[Verse]
闭上双眼让喧嚣都停摆
窗外的风轻轻穿过那片海
心里的尘埃慢慢落下来
此刻只有呼吸还在

[Prechorus]
不需要寻找繁复的答案
只需要点亮微弱的烛光
某种力量在悄然生长

[Chorus]
在一片静默里我听见应许
像晨曦温柔洒满了大地
信念是无声却坚定的言语
灵魂终于寻回了栖息地

[Verse]
纯粹的弦音在空气中摇曳
指尖下流淌过岁月的更迭
放下了那些沉重的纠结
在这瞬间与自己和解

[Chorus]
在一片静默里我听见应许
像晨曦温柔洒满了大地
不用去怀疑未知的结局
每一步都走在光里

[Outro]
如风过境如此宁静
Suno v4.5 Mureka V7.6 YUE
DiffRhythm2 ACE-Step HeartMuLa
wedding, Piano, Strings, acoustic guitar, pop, Romantic
Click to Collapse Lyrics
[Intro]

[Verse]
我在云端 守着孤单的白
你在深渊 卷起深邃的蓝
遥不可及的距离 隔着万水千山
却在视线的尽头 彼此纠缠

[Prechorus]
夕阳坠落 烧红了半个天际
这是我为你 准备的嫁衣
屏住呼吸 等待奇迹

[Chorus]
当天空终于沉入大海的胸怀
亿万年的等待 此刻都释怀
分不清是云坠落 还是浪涌上来
整个世界 只有我们相爱

[Bridge]
地平线不再是 阻碍的界限
而是我们 紧紧相连的指尖
倒影里 刻下永远

[Chorus]
当天空终于沉入大海的胸怀
亿万年的等待 此刻都释怀
分不清是云坠落 还是浪涌上来
在这片蔚蓝里 永不分开

[Outro]
Suno v4.5 Mureka V7.6 YUE
DiffRhythm2 ACE-Step HeartMuLa
Sad, Regret, electric guitar, drums, Piano, emotional
Click to Collapse Lyrics
[Verse]
街角的冷风还在吹散昨夜的尘埃
我傻傻站在原地等待一个不可能的未来
时钟转了一圈又一圈终于停摆
仿佛在嘲笑我对你戒不掉的依赖

[Prechorus]
如果是梦为什么迟迟醒不来
如果是痛为什么无法去释怀
明明说过无论如何绝不分开
最后却是我亲手把你推向人海

[Chorus]
这结局太过苍白谁都没能幸免
思念像一场海啸瞬间淹没了视线
声嘶力竭的呐喊你根本听不见
只有沉默在回应我迟到的抱歉
如果能重来哪怕只是一瞬间
绝不让你消失在我的世界

[Verse]
手机里存着那些舍不得删的照片
笑容太刺眼疯狂灼烧着我的双眼
曾经滚烫的誓言如今变成了谎言
每一句都刺在心上最痛的边缘

[Bridge]
以为只要转身就能彻底遗忘
却在深夜被寂寞狠狠捆绑
这空荡荡的心房再也无法安放

[Chorus]
这结局太过苍白谁都没能幸免
思念像一场海啸瞬间淹没了视线
声嘶力竭的呐喊你根本听不见
只有沉默在回应我迟到的抱歉
如果能重来哪怕只是一瞬间
绝不让你消失在我的世界

[Outro]
原来爱是一场无法回头的流浪
只剩下我一个人在原地绝望
Suno v4.5 Mureka V7.6 YUE
DiffRhythm2 ACE-Step HeartMuLa
Walking, acoustic guitar, Strings, hope, hopeful
Click to Collapse Lyrics
[Verse]
脚下的泥泞终于慢慢变干
穿过了森林前方是蔚蓝
口袋里装着昨天的孤单
随着步伐一点点飘散

[Prechorus]
风吹动了衣摆
光穿透了阴霾
不用去管终点还有多远
这一刻只有自在

[Chorus]
一步又一步走出长夜的灰暗
去拥抱那久违的灿烂
路还在延伸哪怕曲折蜿蜒
相信最好的风景就在下一站

[Verse]
路边的野花也在努力盛开
像是在庆祝这新的期待
呼吸着自由自在的节拍
所有阴影都被阳光覆盖

[Chorus]
一步又一步走出长夜的灰暗
去拥抱那久违的灿烂
路还在延伸哪怕曲折蜿蜒
相信最好的风景就在下一站

[Outro]
不停歇的脚步是给未来最好的礼物
Suno v4.5 Mureka V7.6 YUE
DiffRhythm2 ACE-Step HeartMuLa
Japanese
synthesizer, drum machine, powerful, epic, Keyboard, electronic
Click to Collapse Lyrics
[Intro]

[Verse]
メニューの写真を遥かに超えて
運ばれてきた茶色い要塞
テーブルが重さで悲鳴を上げる
私の胃袋も震え出した

[Prechorus]
顎が外れる覚悟を決めて
滴るソースを受け止めろ
カロリーなんて気にするな
今夜は無礼講さ

[Chorus]
スーパービッグハンバーガー!
肉と肉が積み重なる摩天楼
とろけるチーズのナイアガラ
夢も希望も全部挟んで
豪快にかぶりつけ

[Verse]
片手じゃとても持てない質量
溢れ出す肉汁の洪水
ピクルスが隙間から逃げ出した
野性に戻る瞬間さ

[Chorus]
スーパービッグハンバーガー!
肉と肉が積み重なる摩天楼
とろけるチーズのナイアガラ
夢も希望も全部挟んで
豪快にかぶりつけ

[Outro]
ああ、幸せの重み
もう動けないよ
Suno v4.5 Mureka V7.6 YUE
DiffRhythm2 ACE-Step HeartMuLa
Korean
energetic, drums, driving, rock, electric guitar
Click to Collapse Lyrics
[Intro]
심장이 타는 냄새가 나

[Verse]
차가운 레일 위를 긁어대는 소리
마찰열이 내 혈관을 태우고 있어
어두운 터널 따위 그냥 씹어삼켜
뒤를 돌아보는 건 약해빠진 짓이야

[Prechorus]
속도계의 바늘은 이미 Red Zone
경고음은 리듬이 돼 더 크게 울려
준비는 끝났어 미친 듯이 밟아

[Chorus]
폭주하는 강철의 짐승처럼
이 밤의 적막을 찢어버려
중력을 거스른 이 궤도 위에서
나는 오직 앞만 보고 달려가

[Bridge]
브레이크는 처음부터 없었어
충돌한다 해도 상관없어
끝까지 가볼 거야 재가 될 때까지

[Chorus]
폭주하는 강철의 짐승처럼
이 밤의 적막을 찢어버려
중력을 거스른 이 궤도 위에서
나는 멈추지 않는 폭풍이야
Suno v4.5 Mureka V7.6 YUE
DiffRhythm2 ACE-Step HeartMuLa
Spanish
Cafe, Regret, Keyboard, synthesizer, drum machine, R&B, electric guitar
Click to Collapse Lyrics
[Intro]
Luces de neón parpadeando bajo
Un zumbido digital en el rincón

[Verse]
Soy la máquina de cromo y luz
Mirando desde la esquina,vi su cruz
El café se enfría en la mesa dos
Mientras el silencio les dice adiós

[Prechorus]
Quise cambiar el destino con una canción
Mezclar un ritmo alegre en su conversación
Pero mis circuitos fallaron esa vez

[Chorus]
Oh,culpa de mis cables,error fatal
Solté una balada triste,un final
La guitarra eléctrica lloró por mí
Y vi cómo el amor salía de aquí
Solo soy metal con arrepentimiento

[Verse]
La moneda sigue atrapada en mi interior
Como un recuerdo pesado,un dolor
El sintetizador suena en soledad
Repitiendo el eco de esa verdad

[Chorus]
Oh,culpa de mis cables,error fatal
Solté una balada triste,un final
La guitarra eléctrica lloró por mí
Y vi cómo el amor salía de aquí

[Outro]
Sistema apagado
Perdón
Lo siento tanto
Suno v4.5 Mureka V7.6 YUE
Not supported
DiffRhythm2 ACE-Step HeartMuLa

BibTeX

@misc{yang2026heartmulafamilyopensourced,
            title={HeartMuLa: A Family of Open Sourced Music Foundation Models}, 
            author={Dongchao Yang and Yuxin Xie and Yuguo Yin and Zheyu Wang and Xiaoyu Yi and Gongxi Zhu and Xiaolong Weng and Zihan Xiong and Yingzhe Ma and Dading Cong and Jingliang Liu and Zihang Huang and Jinghan Ru and Rongjie Huang and Haoran Wan and Peixu Wang and Kuoxi Yu and Helin Wang and Liming Liang and Xianwei Zhuang and Yuanyuan Wang and Haohan Guo and Junjie Cao and Zeqian Ju and Songxiang Liu and Yuewen Cao and Heming Weng and Yuexian Zou},
            year={2026},
            eprint={2601.10547},
            archivePrefix={arXiv},
            primaryClass={cs.SD},
            url={https://arxiv.org/abs/2601.10547}, 
}