Most free AI music generators give you one lever: type a prompt, paste some lyrics, roll the dice. DiffSynth-Music, released September 14 by ModelScope’s DiffSynth-Studio team, adds the controls a working musician actually wants. Feed it a click track and the song locks to your tempo. Feed it an a cappella vocal and it writes the band around it. Feed it a backing track and it writes a vocal on top.
The weights are Apache 2.0, there is a free browser demo on Hugging Face, and it runs on top of ACE-Step 1.5, the open music model we already liked. The catch is size: the full download is 36 GB. Here is what it does, what the paper does and does not prove, and how to try it today without paying anything.
DiffSynth-Music at a Glance
- What it is: five audio “control modes” (Beats, Vocals, Accompaniment, Prosody, Reference) bolted onto a frozen ACE-Step 1.5 XL music model.
- What’s free: everything. Apache 2.0 weights, Apache 2.0 code, and a free ZeroGPU demo Space.
- What it costs: $0, plus disk space. The full repo is 36.3 GB; the base DiT and each of the three adapters are 8.34 GB apiece.
- Hardware: no official VRAM figure. The model card ships a disk-offload config for low-VRAM GPUs; a comfortable full load needs roughly 20 GB (our estimate).
- The one catch: Python only. No ComfyUI node, no GGUF, no desktop app yet.
What DiffSynth-Music Actually Adds to ACE-Step
DiffSynth-Music does not replace ACE-Step 1.5. It keeps the ACE-Step 1.5 XL SFT generator frozen and adds three “template” models that listen to an audio file and steer the generation. If you read our ACE-Step 1.5 breakdown, the text-to-music quality is the same; what changes is how much of the song you get to decide.
- Beats: generates a click track at a fixed BPM (60–200 in the demo) and forces the downbeats onto it. Useful when a song has to sit on a video edit.
- Vocals: a built-in Demucs separator pulls the vocal out of your file, and the model writes a new accompaniment around it.
- Accompaniment: the reverse. Keep the instrumental, get a new sung vocal from your lyrics.
- Prosody: keeps the timing and pitch of every syllable from a sung guide, but blurs the words, so you can re-lyric a melody you hummed.
- Reference (experimental): borrows style, melody, singing style and timbre from the loudest section of a reference track.
How the KV-Cache Trick Keeps It Fast
According to the paper (Duan, Gao, Zhang and Chen, September 11), each template model is initialized from the backbone’s own DiT. It turns your control audio into layer-by-layer keys and values, which get concatenated into the generator’s attention. The control cache is computed once and reused for all 50 sampling steps, so adding a control costs a few seconds, not double the render time.
The Space’s own code has timing numbers: about 9.5 seconds for a 60-second song at 50 steps on its ZeroGPU hardware, and about 34 seconds for 150 seconds at 100 steps. Vocals and Accompaniment modes add roughly 9 extra seconds for a 150-second track because Demucs runs twice. Because the controls share one audio latent space, the paper says they can be combined. The public examples show one control at a time, though, and all the benchmark numbers were measured one control at a time.
What the Benchmarks Show (and Who Measured Them)
Every number below comes from the authors, on 100 test songs: 50 English tracks from MUSDB18-HQ and 50 Mandarin tracks from a private dataset. Nobody has independently reproduced them yet, and Artificial Analysis does not rank controllable music models.
Beat alignment (Beat-F1) jumps from 0.30 to 0.85, and melody match on prosody (Pitch50) goes from 0.04 to 0.47. That confirms the controls work, but it is not a quality win over rivals: MiniMax Music 3, HeartMuLa and LeVo-2 were scored with no control input either. On overall audio quality (AudioBox-Aesthetics), DiffSynth-Music lands close to ACE-Step, and Vocals mode actually scores lower (7.10 vs 7.37 content enjoyment). LeVo-2-Large beats every DiffSynth mode on raw quality. It is a control tool, not a quality upgrade.
DiffSynth-Music vs Other Free Music Models
| Model | Audio control | License | Download | Free demo |
|---|---|---|---|---|
| DiffSynth-Music | Beats, vocals, accompaniment, prosody, reference | Apache 2.0 | 36.3 GB (all modes) | HF Space |
| ACE-Step 1.5 | Text + lyrics (plus its own edit tools) | MIT | Runs on ~4 GB VRAM | HF Space |
| YuE2-3B | Text + lyrics | CC BY-NC 4.0 (no commercial use) | 7.3 GB | HF Space |
| MiniMax Music 3 | Text + lyrics | See model card | ~9.8 GB VRAM as GGUF | HF Space |
If you only need prompt-to-song, the lighter picks in our YuE2-3B review and the MiniMax Music 3 GGUF guide are cheaper to run. For turning an existing song into a new one, compare it with MuLaCover, which is built for covers.
Try DiffSynth-Music Free Today
Zero-install route (2 minutes): open the DiffSynth-Music demo Space. Pick a mode from the six options (Text to Music, Beats, Vocals, Accompaniment, Prosody, Reference), upload your audio if the mode needs it, add a style prompt and lyrics, and hit generate. Sliders cover duration (20–150 s), BPM (60–200), steps (20–100) and CFG (1–8). It runs on ZeroGPU, so a free Hugging Face login gets you a daily GPU quota; you’ll hit that faster with long, 100-step renders.
Local route: you need an NVIDIA GPU with CUDA, Python, and about 40 GB of free disk. Grab the code from DiffSynth-Studio on GitHub (Apache 2.0):
git clone https://github.com/modelscope/DiffSynth-Studio.git
cd DiffSynth-Studio
pip install -e .Then copy the Python loader from the DiffSynth-Music model card. It pulls the weights automatically through ModelScope’s snapshot_download. Output is 48 kHz stereo, saved with torchaudio.save. On a card under ~20 GB, expand the “Loading the models on low-VRAM GPUs” section: it streams the DiT from disk, keeps the VAE on the CPU, and sets lazy_loading=True so the adapters only load when you use them. It will be slower, but it runs. If you only need one mode, you can skip downloading the adapters you won’t use and save up to 16.7 GB.
Honest Trade-Offs
- It’s heavy. 36 GB on disk for all five modes. ACE-Step 1.5 alone is a fraction of that.
- No creator tooling yet. We found no ComfyUI node or GGUF quant as of today. It’s Python scripts or the Space.
- Quality isn’t better, only more controllable. Vocals mode scored slightly below the plain backbone on the authors’ own aesthetics metrics.
- Half the test set is private. The Mandarin half of the benchmark can’t be checked, and the model card’s example lyrics are in Mandarin. Test English lyrics yourself before you commit.
- Reference mode is labeled experimental, and Apache 2.0 covers the model, not the song you feed it. Using someone else’s track as a reference or vocal source is a copyright question the license doesn’t answer.
Workflow Tips
- Score to picture with Beats mode. Get your edit’s BPM from your NLE’s beat markers, enter the same BPM in the Beats slider, and generate at the exact length of your cut.
- Hum, then re-lyric. Record a rough sung melody on your phone, run Prosody mode, and write proper lyrics. The timing and pitch survive; your placeholder words don’t.
- Keep your own vocal, swap the band. Upload a finished song with your own singing in Vocals mode and prompt for a different genre, like acoustic instead of rock.
- Draft at 20–30 steps. The Space’s timing code shows render time scales with steps × duration. Audition ideas at low steps, then re-run the winner at 50.
- Keep the seed. The examples use seed 42 and CFG 4. Keep the seed fixed while you change one control at a time, so you can tell what each change did.
Want to see how the paid side compares? Our MiniMax Music 3.0 vs Suno 5.5 Pro test covers one of the baselines in this paper, and there’s more AI music testing on the Creative AI Show YouTube channel.
DiffSynth-Music FAQ
Is DiffSynth-Music free?
Yes. The weights and code are Apache 2.0 and the Hugging Face demo Space is free, subject to ZeroGPU’s daily quota.
Can I use DiffSynth-Music commercially?
The Apache 2.0 license allows commercial use, and the ACE-Step backbone is MIT. You are still responsible for the audio you feed in. Don’t use tracks you don’t have rights to in Vocals, Accompaniment or Reference mode.
What are DiffSynth-Music’s system requirements?
You need an NVIDIA CUDA GPU, Python, and about 40 GB of disk for all modes. The team publishes no VRAM minimum. Plan on roughly 20 GB for a full bf16 load (our estimate), or use the model card’s disk-offload config on smaller cards.
Does DiffSynth-Music work in ComfyUI?
Not yet. There is no official or community ComfyUI node as of September 23, 2026. Use the Python pipeline or the demo Space.
How long can DiffSynth-Music songs be?
The demo Space caps songs at 150 seconds. The model card’s local example renders a full 240-second song.
Accurate as of September 23, 2026. Model versions, demo quotas and licenses move fast, so check the model card before you build on it.
No responses yet