AI music: beyond the buzzword

Over a million users tried out OpenAI’s then latest model within five days of its launch.

When Google themselves announced at the start of this year they had developed a similar system which could create music instead of text, ‘AI Music’ almost immediately reached peak interest on their search engine - the concept entering many of our minds for the first time.

Fast forward to now, and elder statesmen of pop music are lining up one after another to proclaim the AI doomsday that is facing the music industry, while a select group of others such as David Guetta champion the tech as ‘the future of music’.

Meanwhile AI voice covers are flooding social media, particularly on TikTok, where you can hear Freddie Mercury sing ‘Hey Jude’, or even Barack Obama take on ‘Let it Go’.

There was also much confusion recently when Paul McCartney announced a final Beatles song would be released with the help of artificial intelligence (it was actually used to help isolate the vocals of a John Lennon demo, rather than create anything original).

It seems like AI and its use in music has at once become both sensational and polarising. It is the latest ‘buzzword’ dominating conversation, both inside music but also outside at times. 

“There's a reason there are tech buzzwords because once in a while they're real…It's going to change everyone's life. It's going to be like electricity.”

Airbnb CEO Brian Chesky talking recently to Yahoo News

While AI voice cloning is still most at home on social media and with home users, a glimpse of a possible future emerged when David Guetta recently released a song with artificially generated Eminem vocals. 

The French DJ said he would ‘obviously’ not release the song commercially, but it hasn’t stopped some from trying to make money through the technology in other ways.

There are a number of ‘freemium’ sites that allow users to upload their own tracks which the site then converts, so they don't have to build their own model and manually do this.

Many TikTok accounts that post ‘voice cloning’ videos put links to these sites - such as Voicify, VoiceDub, and previously Uberduck.

These companies often have their own profiles too which share covers made using their software.

The latter of these has recently removed almost all copyrighted voice models, such as famous musicians or actors, even if they were generated by users themselves. 

Uberduck faced copyright claims from voice actors Jim Cummings and Tara Strong, leading them to announce they are focusing on ‘commercial friendly’ voices.

It’s still incredibly easy to generate these kind of covers on other platforms however.

For £8.99 a month, Voicify will let you do 25 voice ‘conversions’ a month - and it is as simple as picking a voice and pasting in a YouTube link.

It’s not entirely clear whether the company uses a licensing model (after declining to comment), but it seems they are trying to distance themselves from any possible infringement. 

The Terms of Use state: “It is important to clarify that these models are completely artificial creations…They do not incorporate or utilize any proprietary recordings, performances, or unique voice samples of these public figures.”

Voicify further ‘prohibit’ users from infringing intellectual property rights, and also state they must ‘indemnify’ the company from any claims resulting from its use.

The fact that people are using the technology in this way, which researchers such as Prof. Bryan-Kinns have worked on for the last five years, is something he says they did not anticipate. 

It utilises a similar form of ‘deep learning’ that language models such as Chat GPT use, except it is trained on sound. 

“This one is generating voice, so you will train a deep learning model on a lot of recordings of somebody's voice. And then you match that up to say, the lyrics, or the words that they were saying, as the training input. 

“Then you can ask it to generate either new speech from text. So based on what you've taught it, or you can just use it to transpose from one style to another, or one person's voice to another.

“What’s happening now is people are starting to use existing models, so you have a base model of, say, a male person from England, and then you customise it with your own voice. That way you need much less training.”

The researcher feels that although there will be a decline in interest, the AI behind this vocal cloning is going to have a big impact both in music, but in speech and voice technology more widely. 

“The sort of novelty effect of training your own voice, singing like somebody else, that'll wear off. But it's going to start getting embedded in all the technologies that we're using…fairly soon in Zoom for instance, you’ll be able to have real time translation of my voice into a different language. It'll be a straightforward thing to do.

“The TikTok mashups seem like a flash in the pan. But it is a different kind of genre that's already emerged due to AI. I think other different styles or genres will start to emerge, and you can't really predict more about that, because it's going to be something that somehow has a feel to it or something that resonates with a certain segment of society - that really makes them want to listen to it more.”

This changing of one voice to another is also known as 'Timbre Transfer", and is not only limited to voices. It can also be used to make instruments sound like other instruments, or voices like instruments.

“For Damon Albarn and Blur, there’s probably moments where it sounds very cockney, and so, if you put Kurt Cobain on top of that, it doesn't necessarily change the way certain vowels are used. It still sounds cockney.”

Dr Tom Collins, Assistant Professor in Music Technology at the University of York, tries to imagine how the voice technology could be used to convert accents too, something it can be said to sometimes struggle with at the moment. 

“I think it could become part of a creative workflow where there's some middle person who's still there, like a human in the loop at that stage that kind of glues these two things together. Or, you have to redesign the way that the algorithm is working.” 

Dr Collins however, uses artificial intelligence in music in quite a different way himself, having represented Britain along with his spouse in the first annual AI Song Contest. 

Inspired by Eurovision, the competition aims to promote ‘the possibilities of human-AI co-creativity’, and held its first competition in 2021.

“I made it relatively quickly with my spouse, because I had been working with Imogen Heap on the song, but when lock down came in while we were trying to finish it off, it didn't happen.

“So we had AI generated starting materials, and we had to do it within a matter of days before the deadline. It was around 20 to 30 melodies, baselines, drum beats, and chord sequences that had been generated by a Markov model. The lyrics come from an AI Lyric generator site.

“They provided us with something like 200 previous Eurovision entries in MIDI format - so it was very tied to the competition.” 

Mark does touch upon an area where he feels there may be a change caused by generative AI.

“I wonder how much TV production companies will cut costs, by thinking, we're not going to employ a specific composer, or buy licensed music, We're going to go onto an AI thing, put in variables and create music that way. I think that will happen, some composers are concerned about that.”

Dr Robert Laidlow, a composer and researcher at The University of Oxford, also feels this could be a case where technology may be used as a cost cutting measure - so that a human composer is not needed.

“What we haven't really seen before is just how much power some of these big technological corporations might have over consumerist music in the sense that there are certain musical genres where we're just going to say that's fine, we don't need a human to do that. I'm happy with AI generating it. Advert music is surely going to be like this.

“Where's the line for people accepting AI music? Film music? I don't think you're going to get something like Oppenheimer with an AI generated score, but you probably will get some lower budget ones.”

Even a quick Google search reveals there are already many companies offering AI generated music that seemingly fit into the ‘advert’ or ‘stock music’ mould. 

But the success of some suggest they shouldn’t all be dismissed as merely tools to accompany low-budget video production. 

Australian company Splash Music is one of these, and is a platform which allows users to type a prompt to generate music, but also rapping or singing.

Splash, which was founded in 2017, launched an AI-powered music game within the extremely popular online platform Roblox', which was played over 128 million times within the first year of its launch in 2020, and still gets over 95,000 unique users making and performing music per month.

Michelle Dintner, Chief Strategy Officer, says while the music generation works differently in Splash Pro, she sees the popularity of the game is evidence that younger generations have a different relationship with music.

“There is a demand amongst young people to have a more interactive relationship with music. They are no longer just passive listeners, they want to make the music that they want to hear.” 

The model that powers the software is trained by a dedicated music team, and Michelle says the music industry is simply following developments that have already happened in art and video production.

When asked about the concern around AI and it taking the ‘human element’ out of music, she stressed they saw AI music as an addition not a replacement, and stressed the potential for ‘new’ kinds of music instead of the erosion of familiar forms.

“We see AI as being purely additive and overall increasing the size of the music market that benefits everyone including the commercial music industry and traditional artists.

“We hope to see more people making music than ever before, and to see new genres emerge and a new type of artists emerge that may not have been empowered to make music before AI tools existed.

"We’d also like to see established artists using AI to change how they promote their music, grow and interact with their fans in new and exciting ways that were previously not possible.”