Researchers at Amazon have developed the largest text-to-speech model to date, claiming it possesses “emergent” qualities that enhance its ability to articulate even complex sentences naturally. This innovative technology could help it transcend the uncanny valley often associated with synthetic voices.
While improvements in these models were anticipated, Amazon's researchers specifically aimed for a significant leap in capabilities, similar to the advancements seen with larger language models (LLMs). Historically, LLMs demonstrate greatly enhanced robustness and versatility once they surpass a certain threshold in size.
It’s important to clarify that this doesn’t imply these models are gaining sentience. Instead, they exhibit a marked improvement in performance for specific conversational AI tasks. The Amazon AGI team, perhaps revealing their broader ambitions, theorized that the same trend could apply to text-to-speech models, and their findings support this idea.
The newly introduced model is named Big Adaptive Streamable TTS with Emergent Abilities, abbreviated as BASE TTS. The largest version utilizes an impressive 100,000 hours of public domain speech data, predominantly in English, with smaller portions in German, Dutch, and Spanish.
With 980 million parameters, BASE TTS appears to be the largest model in this category. The researchers also tested 400M- and 150M-parameter models trained on 10,000 and 1,000 hours of audio, respectively. This varied approach allows them to identify where emergent behaviors may begin to manifest.
Interestingly, the medium-sized model exhibited the capability jump the team sought. While it does not significantly outperform its predecessors in overall speech quality, it has shown noteworthy emergent abilities. The researchers highlighted several challenging text examples from their study, including:
- Compound nouns: “The Beckhams decided to rent a charming stone-built quaint countryside holiday cottage.”
- Emotions: “Oh my gosh! Are we really going to the Maldives? That’s unbelievable!” Jennie squealed, bouncing on her toes with uncontained glee.
- Foreign words: “Mr. Henry, renowned for his mise en place, orchestrated a seven-course meal, each dish a pièce de résistance.”
- Paralinguistics: “Shh, Lucy, shhh, we mustn’t wake your baby brother,” Tom whispered as they tiptoed past the nursery.
- Punctuations: She received an odd text from her brother: “Emergency @ home; call ASAP! Mom & Dad are worried…#familymatters.”
- Questions: But the Brexit question remains: After all the trials and tribulations, will the ministers find the answers in time?
- Syntactic complexities: The movie that De Moya, who was recently awarded the lifetime achievement award, starred in 2022 was a box-office hit, despite the mixed reviews.
As the authors state, “These sentences are designed to contain challenging tasks—parsing garden-path sentences, placing phrasal stress on long-winded compound nouns, producing emotional or whispered speech, or generating the correct phonemes for foreign words like 'qi' or punctuations like '@'—none of which BASE TTS is explicitly trained to perform.”
Typically, such features can confuse text-to-speech engines, leading to mispronunciations, skipped words, incorrect intonations, or other errors. While BASE TTS still encounters some challenges, it performs significantly better than contemporaries like Tortoise and VALL-E.
The researchers have provided several examples of difficult texts articulated naturally by the new model, available on their demonstration site. Although selected to showcase the model's strengths, these examples are undeniably impressive. A couple of demonstrations include:
-
-
-
Given that the three BASE TTS models share a common architecture, it appears that the model's size and extensive training data contribute significantly to its ability to navigate the complexities outlined earlier. However, it’s crucial to remember that this is still an experimental model, not a commercial product. Future studies will aim to pinpoint the inflection point for emergent capabilities and determine efficient training and deployment methods.
Leo Zao, a representative from Amazon AI and not a co-author of the study, emphasized that the team does not claim exclusive emergent properties for this model. “We think it’s premature to conclude that such emergence won’t appear in other models. Our proposed emergent abilities test set is one way to quantify this emergence, and it’s possible that applying this test set to other models could yield similar results. This is partly why we decided to release the test set publicly,” he explained via email. “It is still early days for establishing a ‘Scaling Law’ for TTS, and we look forward to further research on this topic.”
Notably, this model is “streamable,” as indicated by its name, meaning it generates speech moment-to-moment at a relatively low bitrate rather than producing entire sentences all at once. The team has also worked on creating a separate, low-bandwidth stream that packages speech metadata, such as emotionality and prosody.
With advancements like BASE TTS, text-to-speech technology may experience a breakthrough in 2024, potentially aiding various sectors, including accessibility. However, the team has opted not to publicly release the model's source data due to concerns about misuse. Nonetheless, the details will likely emerge eventually.
Ranking
