Motion to Dance Music Generation using Latent Diffusion Model

Music's role in games and animation, particularly in dance content, is essential for creating immersive experiences. Although recent studies have made strides in generating dance music from videos, their practicality in integrating music into games and animation remains limited. In this context, we present a method capable of generating plausible dance music from 3D motion data and genre labels. Our approach leverages a combination of a UNET based latent diffusion model and a pre-trained VAE model. To evaluate our model's performance, we employed metrics that assess various audio properties, including beat alignment, audio quality, motion-music correlation, and genre score. Quantitative results show that our approach is better than previous methods. Furthermore, we demonstrated that our model can generate audio that seamlessly fits to in-the-wild motion data. This capability enables us to create plausible dance music that complements the dynamic movements of characters and enhances the overall audiovisual experience in interactive media.

At training time, we employ a pre-trained Variational Autoencoder (VAE) to convert the audio mel-spectrogram into a latent code, which is then corrupted using the forward diffusion process to introduce noise. Our U-net based denoising network learns the score of the audio latent code's distribution by conditioning the features from audio-paired motion and genre. Specifically, we concatenate the motion features and genre information and provide them as inputs to all attention modules in our denoising network. During inference, we iteratively generate the audio-latent representation by incorporating the output from our network, which takes the preprocessed motion and genre information as input. This representation is then processed by the VAE decoder to generate the dance motion-aware audio mel-spectrogram. To reconstruct the corresponding audio waveform, we utilize the Griffin-Lim algorithm. This enables us to obtain the final dance music that aligns with the specified motion and genre inputs.

This work was supported by the National Research Foundation of Korea (MSIT) (No. RS-2023-00222383) and Korea Creative Content Agency (MSIT) (Project Name: Development of universal fashion creation platform technology for avatar personality expression, No. RS-2023-00228331) grants funded by the Korea government.

We would like to thank our labmates from the Visual Media Lab (VML) for their support especially the Character team, Chaelin, and Seokhyeon for their invaluable insights. We would also like to extend our gratitude to Joel Casimiro for his assistance in crafting our preview image.