Almost a year ago, developers Seth Forsgren and Hayk Martiros launched a fascinating hobby project called Riffusion. This innovative platform generates music not from audio but from images representing sound. While this method may seem unconventional, it has proven effective—my colleague Devin Coldewey has the details here.
Despite some inherent limitations, Riffusion attracted considerable attention—a phenomenon not surprising given the growing interest (and debate) surrounding AI-generated music technology. According to Forsgren, millions have explored Riffusion, which has even been referenced in research papers from major tech companies like Meta, Google, and TikTok's parent company, ByteDance.
This interest has also caught the eye of investors. This year, Forsgren and Martiros decided to take Riffusion to the next level. They have partnered with the renowned musical duo, The Chainsmokers, and successfully closed a $4 million seed funding round led by Greycroft, with support from South Park Commons and Sky9.
Additionally, Riffusion is set to unveil a new, user-friendly app—an enhanced version of last year’s platform—that empowers users to input lyrics and select a musical style to create shareable "riffs."
“[The new Riffusion] allows anyone to produce original music through short, shareable audio clips,” Forsgren shared in an email interview. “Users just describe the lyrics and the desired music style, and our model generates riffs, complete with vocals and custom artwork, in seconds. This tool acts as a bridge for creators, whether that’s inspiring musicians or just sending a cheerful ‘good morning’ to a friend. Riffs offer a fresh and accessible form of expression that lowers the barriers to making music.”
Forsgren and Martiros crossed paths while studying at Princeton, where they spent a decade making music together in an amateur band. Forsgren has previously founded two venture-backed tech companies, Hardline and Yodel. Meanwhile, Martiros was among the first employees at the drone startup, Skydio.
Forsgren noted that the pandemic inspired them to expand Riffusion, as it highlighted the profound connection music can foster during isolating times. “The pandemic gave us all more time at home, which led me to learn piano,” Forsgren recalled. “Music has a unique ability to connect us during lonely moments. Generative AI is a rapidly evolving landscape, and Riffusion is designed to harness this technology, providing a joyful new instrument that encourages everyone to create music throughout their lives.”
The upgraded Riffusion is powered by an audio model that the team—comprised of six members, including Forsgren and Martiros—developed from the ground up. Similar to its predecessor, the new model is refined on spectrograms, which are visual representations of sound that display the intensity of various frequencies over time.
Forsgren and Martiros generated spectrograms of different music styles and tagged the images with descriptors such as “blues guitar” and “jazz piano.” This process taught the model how specific sounds “look” and how to recreate or blend them based on text prompts like “lo-fi beat for the holidays” or “folk blues from the Mississippi Delta.”
“Users can describe musical qualities through natural language or even record their own voice to prompt the model, resulting in unique outputs,” Forsgren explained. “We believe this product will empower music producers and audio engineers to explore new creative avenues and find inspiration in a completely fresh way.”
In terms of copyright concerns, some may wonder about the implications of using generative AI to create tracks that resemble existing styles. Recently, a Discord community released an entire album featuring an AI-generated imitation of Travis Scott’s voice, prompting legal scrutiny from his label.
Music labels have swiftly flagged AI-generated tracks using similar sounds to streaming platforms like Spotify and SoundCloud, often citing intellectual property issues—and they have generally succeeded. However, questions remain regarding whether "deepfake" music infringes on the copyrights of musicians, record labels, and other rights holders.
Forsgren emphasized that the improved Riffusion does not use famous artist names or songs for training and is not designed to replicate them. “The product isn't intended to generate deepfakes nor does it recognize popular artists' names as prompts,” he clarified. “Instead, it allows users to craft personalized messages and catchy hooks within the app. It’s not uncommon for a riff you create to linger in your mind, leading you to hum along to it throughout the day.”
Looking ahead, there isn’t a definitive monetization strategy in place yet. Currently, Forsgren and Martiros are focused on expanding the Riffusion team and developing complementary generative AI products.
However, Forsgren hinted at potential collaborations with artists like The Chainsmokers to explore how this technology could enhance their creative processes. “It’s still early in the realm of generative music,” Forsgren said. “Models like Google’s MusicLM, Facebook’s MusicGen, and Stability’s Stable Audio are exciting developments in this field. Nevertheless, Riffusion differentiates itself by enabling users to generate lyrics in their music through an enjoyable and accessible online platform.”
Ranking
