00:00:01 · John Collison, Mati Staniszewski
[00:00:01] S0: Mati Staniszewski cofounded ElevenLabs in 2022 and has since scaled it to the $11,000,000,000 leader in AI audio. He's credited with capturing the humanness of speech through realistic emotional inflection, inflection, and they're now expanding into everything from agentic workflows to music. Thanks for [00:00:17] S1: doing this. [00:00:18] S0: Thanks for having me. Maybe a place to start is describe to me how... Like, I know how an LLM works at a high level. Describe to me how an audio model works. Like, if we were Karpathy style looking to, you know, build a toy one from scratch, how does it work?
00:00:18 · John Collison, Mati Staniszewski
[00:00:18] S0: Thanks for having me. Maybe a place to start is describe to me how... Like, I know how an LLM works at a high level. Describe to me how an audio model works. Like, if we were Karpathy style looking to, you know, build a toy one from scratch, how does it work? [00:00:36] S1: In the early days, you try to replicate it exactly how you would replicate it with with with the human body. So you try to completely try to reproduce a machine, analog machine, that will create a vocal tract effectively. Then that progressed into trying to create effectively, like, a digital signals for speech. Bell Labs was one of the first to try to, like, create a structured set of signals that will represent the speech, and that is a a first precursor to what we would do today. Then you would try to stitch in phonemes, effectively different sounds of of of how we would speak humans, and then try to concatenate them together. It's another important part in equation where you would, based on the most probabilistic approach of the next word, you would effectively try to bring the phonemes from your library of phonemes and bring them together. And then down to the modern history where now we effectively do similar, like, neural nets in in other domains. So you predict the next sound Yes. Based on on, of course, the context of the previous sounds. Yeah. If it's a streaming speech. If it's, let's say, a context of audio, you will use combination of predicting of the phonemes, but you also use the contextual text element [00:01:45] S0: Yep.
00:01:45 · John Collison, Mati Staniszewski
[00:01:45] S0: Yep. [00:01:45] S1: Of of that work. And here, credit to my cofounder, Piotr, who effectively came with that new idea of of how you can now create voice models, which are both reliable, high quality, quick, where you would bring a lot of the ideas from transformer models, from diffusion models into the speech space. Yes. So that prediction of the next token on the phoneme space [00:02:05] S0: Yes. [00:02:05] S1: Wasn't something that was possible. You might be... You you always talk briefly about this of, like, how you kind of operate on the text, on the waveform space. There's also a MEL spectrogram space. So, like, usually, you do text, MEL spectrogram, waveform. Sorry. What's the [00:02:16] S0: spectrogram space? [00:02:17] S1: It's like a visual representation of how the speech sounds across pitch, across energy, and then you transform that into waveform. [00:02:24] S0: Got
00:02:24 · John Collison, Mati Staniszewski
[00:02:24] S0: Got [00:02:24] S1: it. So, like, when WaveNet came came along and and tachotron models, they would effectively use text to MEL spectrogram, so that visual representation Yeah. And then how you decode and encode that into the waveform to to bring it across. And Piotr figured out how to, like, abstract some of those steps and decode and encode them a lot a lot better so that predicting of the next phoneme was one of the big piece. And second big piece was how do you bring that context into the equation? So what I mean by context is if a voice actor was reading a textual copy, you would know that, okay, this is a dialogue sequence. I need to produce a dialogue. If it's a happy sentence, I might need to pronounce it as a happy sentence. Mhmm. But kind of the... What happens before and after [00:03:04] S0: Yes. [00:03:04] S1: Comes into the equation, and you need to bring that across. And then there's a last big piece. So voice model has sound of how you intonate the given fragment, but the second big part is the voice itself of the characteristics of accents, of style, prosody across that voice. So when you actually try to vocalize something, when you create that voice model, you turn text into audio, you need the text, you also need the voice reference of how you want it to be spoken. So here was kind of the second big innovation. So apart from context, it's how you decode and encode those features. So when Bell Labs came with their initial representation of speech, the big piece there was you would have effectively hard coded parameters for that speech.
00:03:04 · Mati Staniszewski, John Collison
[00:03:04] S1: Comes into the equation, and you need to bring that across. And then there's a last big piece. So voice model has sound of how you intonate the given fragment, but the second big part is the voice itself of the characteristics of accents, of style, prosody across that voice. So when you actually try to vocalize something, when you create that voice model, you turn text into audio, you need the text, you also need the voice reference of how you want it to be spoken. So here was kind of the second big innovation. So apart from context, it's how you decode and encode those features. So when Bell Labs came with their initial representation of speech, the big piece there was you would have effectively hard coded parameters for that speech. [00:03:48] S0: With ElevenLabs models... Hard coded parameters for enthusiastic speaker British accents. [00:03:53] S1: Exactly. Exactly. That kind of stuff, like the set of pitch elements that you can select, set of energy, spectrograms you can select from. And in our approach, effectively, you would give the the model open ended ability to select what those parameters should be. So it's not gonna be British, Polish, Spanish, English speaker, but the model will deduce them themselves, the same for for other set of parameters that are not hardcoded, [00:04:18] S0: whether it's [00:04:19] S1: the enthusiasm, whether it's the sadness, etcetera. [00:04:21] S0: You're saying kind of Britishness is an emergent property Exactly. In your voice models.
00:04:21 · John Collison, Mati Staniszewski
[00:04:21] S0: You're saying kind of Britishness is an emergent property Exactly. In your voice models. [00:04:26] S1: Exactly. But... Yeah. And those kind of... Those two big parts, so it's encoding and decoding of how you create the voice. Yep. Super hard problem before and figured out too, how you then construct that in the sentence, how you get the context across so you can predict the next phonemes. [00:04:40] S0: Yes. [00:04:40] S1: So how you bring them together in a reliable and stable way while doing it quick, And these were kind of the two first big innovations in the voice models that continue to today. [00:04:48] S0: But... Okay. So if LLMs reason about text and, you know, words, subparts, tokens, as the way they think about the world, what is the equivalent of a token in the voice model? You mentioned phonemes a bunch. Like, what is that representation? [00:05:04] S1: So we do... We we store we store the voice embedding effectively for the speaker. Uh-huh. So you need that reference when you produce and create the speech. Yes. Of course, in the input to the voice model, you still get the text [00:05:16] S0: Mhmm. [00:05:16] S1: And you bring the the speaker and and and coding. And then when you produce speech, you do operate on the waveform or on the... Or effectively on the phoneme level of of that of that speech. And then when we kind of go the opposite, so of course, when... [00:05:31] S0: I'm sorry. What is a phoneme? Fill in my understanding.