Researchers at Georgia Tech have used artificial intelligence and deep learning technology to teach a robot to compose.

For years robots have been able to play musical instruments, but now thanks to artificial intelligence and deep learning technology, developers at Georgie Tech have taught a robot to compose. A four-armed marimbist robot named Shimon, capable of wielding eight mallets at a time, has been gigging alongside human players for years – and was even featured on The Colbert Report – but now the virtuoso performer is in the running to become the next Bothoven.

^{Shimon, the marimba playing robot composer. Photo © Georgia Institute of Technology}

Researchers at Georgia Tech have taken the robot’s musical skills to new heights, Georgia Tech News reports, feeding the robot datasets including nearly 5000 complete pieces of music and over two million shorter fragments and motifs – from artists as diverse as Beethoven, Miles Davis and Lady Gaga.

“We’ve trained the neuron network with the database that consists of a wide variety of music encompassing everything from jazz, classical to pop music. What [Shimon] is generating is almost like a fusion of these different styles,” explains Mason Bretan – a PhD candidate in Music Technology at the Georgia Institute of Technology – in a YouTube video featuring one of Shimon’s compositions. “What you’re hearing is the result of what it’s learned about music in general, so because it has learned all of these concepts, the crossover between genres, the output is a mixture of all of these different things.”

The composer robot has already debuted two new works online and will premiere its first works live at the Aspen Ideas Festival in Colorado in June. While Bretan has worked with Shimon for seven years, during which time the robot has learned to “listen” to human players and improvise over chord progressions, it is only more recently that the robot has become a composer in its own right, generating both melody and harmony unaided – an initial “seed” of four bars is used as a starting point, but from then on no humans are involved in the compositions Shimon performs.

“Once Shimon learns the four measures we provide, it creates its own sequence of concepts and composes its own piece,” Bretan told Georgia Tech News. “Shimon’s compositions represent how music sounds and looks when a robot uses deep neural networks to learn everything it knows about music from millions of human-made segments.”

The robot is also now capable of playing harmonies and chords – a dramatic improvement from the robot’s days improvising single melody lines. “Shimon is now coming up with higher-level musical semantics. Rather than thinking note by note, it has a larger idea of what it wants to play as a whole,” Bretan said.

Shimon produces completely different music based on the different seed fragments it is fed as a starting point. “They sound like a fusion of jazz and classical,” said Bretan. “I definitely hear more classical, especially in the harmony. But then I hear chromatic moving steps in the first piece — that’s definitely something you hear in jazz.”

“This is a leap in Shimon’s musical quality because it’s using deep learning to create a more structured and coherent composition,” said Gil Weinberg, director of Georgie Tech’s Center for Music Technology and the original creator of Shimon. “We want to explore whether robots could become musically creative and generate new music that we humans could find beautiful, inspiring and strange.”