Developer spotlight: ACE Studio
Type lyrics, draw or play a melody, and hear it sung. That is ACE Studio: a singing-synthesis app that generates realistic vocals from notes and words, with a choice of voices and languages, built for producers who need a topline demo, backing vocals or a choir part without booking a singer.
A tool like this sounds like the work of a large AI lab. It began as an experiment between friends. This is the story of how ACE Studio came to be, told by its co-founder Sean Zhao, and where the team wants to take it.
ACE Studio on MuseHub
In this series we look at the people and ideas behind the software on MuseHub — from apps that sing to producers crafting tomorrow’s hits.
Finding a calling: Sean Zhao’s journey
When Sean Zhao was a kid, there were no musical instruments at home. There were still things that made noise. He discovered he could play melodies on the family’s touchtone phone, which produced a different note for each key, and he transcribed the melodies into number sequences. One day he heard a song with a melody like one he had “composed” earlier. “I felt incredibly happy when I heard it; this experience planted the seed of my love for music,” Sean recalls.
He took piano lessons in middle school, but it was an electronic dictionary — with a built-in code compiler and a few games — that lit the other half of his interests. “I wrote my first lines of code on that little device; a simple for-loop that printed numbers from 1 to 100,” he says.
“I chose software engineering as my major in university and obtained a master’s degree in computer game engineering. My first job was as a game developer, creating mobile games and working on game engine development at Tencent.”
Music never went away, and in early 2016 Sean saw the musical potential in early neural networks — systems that learn patterns from examples rather than following explicit rules, and the foundation of today’s AI.
“In late 2016, an AI named DeepBach emerged, capable of generating music in Bach’s style,” Sean explains. “I was shocked by this technology; it reignited my childhood dream that I had once thought unattainable. I realized that perhaps I could achieve my dream through a different path — by enabling anyone, regardless of skill level, to create their own music.”
He met two like-minded dreamers, Joe and Conger. “Joe has deep insights, and Conger is a musician with high artistic taste.” Together they formed a company and started building.
The idea was to make music creation accessible to everyone. Non-professionals often feel awkward asking singers to perform their songs, or get stuck in endless revisions. Singing synthesis, the team reasoned, could lower that barrier.
The progress in AI made a once near-impossible task look achievable. “We started with a simple concatenative singing synthesis engine. Although the results were not initially great, users still created impressive works. That was when we realized people needed this product.”
Try, try… and try again
The team doubled down, and ACE Studio took shape. For Sean, Joe and Conger, the goal is to give producers a way to express an artistic vision and experiment with their style. “We’d love it if one of our customers has a hit song that was made possible by ACE Studio.”
AI is a live debate among creative audiences, with real concern about its effect on the arts. Sean acknowledges it: “Thanks to the development of large language models (LLMs) and many pre-trained self-supervised models, generative models in other modalities, especially video and audio, have been developing very rapidly. These models have significantly lowered the barriers to content creation, and we can expect to see many unexpected and innovative works emerge. However, as AI-generated content becomes increasingly indistinguishable from real content, there is a growing need to further refine laws and regulations to better govern AI-generated content and to protect human copyrights and other rights.”
On whether musicians will be replaced, the ACE Studio team is positive:
“AI music is developing rapidly, and there are already powerful AI systems like Suno and Udio that can generate high-quality music on demand. However, we believe that musicians and composers will never be completely replaced. Music creators need to express their emotions and inspirations, and directly generating music with AI or describing it in words can never precisely produce the results they envision.
We should expect that much commercial music, such as music for advertisements or background music for bars and restaurants, could be generated by AI. However, music for enjoyment needs a human touch to infuse it with soul, and often it is more than just the audio waveform itself. AI can only learn patterns from existing songs; it cannot create new genres, and there is no human soul or story to listen to as part of the music it creates. For example, if AI music had existed before the advent of jazz, AI would never have been able to emulate jazz.”
ACE Studio is AI in service of a composer’s own vision: the notes and words are yours, the voice is generated. So what is next?
“Currently, we have integrated many AI tools into ACE Studio, such as converting vocals to MIDI and lyrics, and extracting vocals, accompaniment, and various instruments from audio. In the future, we will introduce more AI tools, such as suggesting accompaniment based on vocals and making it easier for users to generate choirs. Additionally, we plan to gradually add more built-in vocal effects like reverb, equalizer, and compression, and we are also considering supporting third-party plugins.”
Next steps
ACE Studio is one of the AI apps on MuseHub. For the wider field, the best AI tools for musicians and the best AI plugins cover what else is worth installing, and the best vocal plugins covers what to do with a vocal once you have one. Browse the AI tools on MuseHub, or get ACE Studio and everything else in one place — download MuseHub free.