Noise2Music: Text-conditioned Music Generation with Diffusion Models

We introduce Noise2Music, where a series of diffusion models is trained togenerate high-quality 30-second music clips from text prompts. Two types ofdiffusion models, a generator model, which generates an intermediaterepresentation conditioned on text, and a cascader model, which generateshigh-fidelity audio conditioned on the intermediate representation and possiblythe text, are trained and utilized in succession to generate high-fidelitymusic. We explore two options for the intermediate representation, one using aspectrogram and the other using audio with lower fidelity. We find that thegenerated audio is not only able to faithfully reflect key elements of the textprompt such as genre, tempo, instruments, mood, and era, but goes beyond toground fine-grained semantics of the prompt. Pretrained large language modelsplay a key role in this story -- they are used to generate paired text for theaudio of the training set and to extract embeddings of the text promptsingested by the diffusion models. Generated examples: https://google-research.github.io/noise2music