Riffusion

Riffusion is a music generator that creates music from text prompts, allowing users to experiment with different styles, instruments, genres, and sounds to explore the latent space of sound. The platform uses a base for img2img diffusion to keep riffs on beat and impact melodic patterns. Users can adjust the settings to explore and create different musical outcomes, with higher denoising resulting in more creative output, but also potentially causing the music to be off-beat.

Features:

• Generates music from text prompts
• Supports a wide range of styles, instruments, modifiers, genres, and sounds
• Can generate infinite variations of a prompt by varying the seed
• Image-to-image conditioning allows for modifying sounds while preserving the structure of an original clip
• Smoothly interpolates between prompts and seeds in the latent space of the model
• Interactive web app with real-time generation and 3D visualization of spectrogram timeline

Technical Details:

• Built on Stable Diffusion, the open-source AI model that generates images from text
• Fine-tuned on images of spectrograms paired with text to generate audio clips
• Short-time Fourier transform (STFT) used to compute the spectrogram from audio
• Griffin-Lim algorithm used to approximate the phase when reconstructing the audio clip
• Torchaudio used for efficient audio processing on the GPU

Cost and Availability:

• The code is available on GitHub: https://github.com/riffusion/
• The model checkpoint is available on Hugging Face: https://huggingface.co/riffusion/riffusion-model-v1
• The interactive web app is available at https://www.riffusion.com/
• GPU-powered inference is provided by Baseten, which offers auto-scaling and observability
• If you have a GPU powerful enough to generate stable diffusion results in under five seconds, you can run the experience locally using their test flask server

‍

‍