00:03:04 · Mati Staniszewski, John Collison
[00:03:04] S1: Comes into the equation, and you need to bring that across. And then there's a last big piece. So voice model has sound of how you intonate the given fragment, but the second big part is the voice itself of the characteristics of accents, of style, prosody across that voice. So when you actually try to vocalize something, when you create that voice model, you turn text into audio, you need the text, you also need the voice reference of how you want it to be spoken. So here was kind of the second big innovation. So apart from context, it's how you decode and encode those features. So when Bell Labs came with their initial representation of speech, the big piece there was you would have effectively hard coded parameters for that speech. [00:03:48] S0: With ElevenLabs models... Hard coded parameters for enthusiastic speaker British accents. [00:03:53] S1: Exactly. Exactly. That kind of stuff, like the set of pitch elements that you can select, set of energy, spectrograms you can select from. And in our approach, effectively, you would give the the model open ended ability to select what those parameters should be. So it's not gonna be British, Polish, Spanish, English speaker, but the model will deduce them themselves, the same for for other set of parameters that are not hardcoded, [00:04:18] S0: whether it's [00:04:19] S1: the enthusiasm, whether it's the sadness, etcetera. [00:04:21] S0: You're saying kind of Britishness is an emergent property Exactly. In your voice models.
00:04:21 · John Collison, Mati Staniszewski
[00:04:21] S0: You're saying kind of Britishness is an emergent property Exactly. In your voice models. [00:04:26] S1: Exactly. But... Yeah. And those kind of... Those two big parts, so it's encoding and decoding of how you create the voice. Yep. Super hard problem before and figured out too, how you then construct that in the sentence, how you get the context across so you can predict the next phonemes. [00:04:40] S0: Yes. [00:04:40] S1: So how you bring them together in a reliable and stable way while doing it quick, And these were kind of the two first big innovations in the voice models that continue to today. [00:04:48] S0: But... Okay. So if LLMs reason about text and, you know, words, subparts, tokens, as the way they think about the world, what is the equivalent of a token in the voice model? You mentioned phonemes a bunch. Like, what is that representation? [00:05:04] S1: So we do... We we store we store the voice embedding effectively for the speaker. Uh-huh. So you need that reference when you produce and create the speech. Yes. Of course, in the input to the voice model, you still get the text [00:05:16] S0: Mhmm. [00:05:16] S1: And you bring the the speaker and and and coding. And then when you produce speech, you do operate on the waveform or on the... Or effectively on the phoneme level of of that of that speech. And then when we kind of go the opposite, so of course, when... [00:05:31] S0: I'm sorry. What is a phoneme? Fill in my understanding.
00:05:31 · John Collison, Mati Staniszewski
[00:05:31] S0: I'm sorry. What is a phoneme? Fill in my understanding. [00:05:33] S1: It's like a a syllable deconstructed even to smaller elements. Okay. And these are effectively like the human sounds you can produce. [00:05:40] S0: Got [00:05:40] S1: it. So this would be like the most the close to that representation. But of course, in our models, now it's going to be a combination of not only operating on phoneme level, you also operate on the text level. You operate kind of in both in sync because when you are predicting the context, you need to understand Yes. How that sentence will get constructed, and especially if it's more of a streaming real time use case and, like, a voice agent setting, you need both parts to to work across. So it's similar to how you would operate on a tuck in level on the on the tech side. We operate on the tuck in level on the audio side. [00:06:12] S0: It feels like a big part of the magic of 11 was your voices were much more human sounding. How did you accomplish that?
00:06:12 · John Collison, Mati Staniszewski
[00:06:12] S0: It feels like a big part of the magic of 11 was your voices were much more human sounding. How did you accomplish that? [00:06:21] S1: So I'll kind of give you a quick quick synopsis of what... How we think about the models on the text to speech side today. In any model, you need you need architecture, you need compute, you need data. So architecture innovations were one thing. The data part was the second big thing. With audio, you will have you will have a lot of audio data available, but frequently, you will not have annotated in the right way. You won't have which speaker is speaking when. Some of the what is annotated, but the how isn't. So, like, as we are speaking now, what's the emotions that we use? What are the accents that we use? So we would invest a lot internally on effectively creating our own data labelers, our own team to be able to create those datasets that will be better. And that was a combination of, of course, like semiautomatic techniques and then manual techniques. Actually, a lot of the models that we did afterwards actually spun out from a lot of that research too. Speech to text model initially was a model we did for ourselves because the models on the market just weren't good to annotate that data. Then another brilliant researcher on our team was being able to construct it so we span it out as a model that we brought to the customers. [00:07:30] S0: So you've just been doing useful stuff in voice, and that has emerged with a whole bunch of products that you might not have expected because you find you're building useful stuff.
00:07:30 · John Collison, Mati Staniszewski
[00:07:30] S0: So you've just been doing useful stuff in voice, and that has emerged with a whole bunch of products that you might not have expected because you find you're building useful stuff. [00:07:39] S1: Exactly. Exactly. And that's kind of combination of data of being able to do it automatically, create a team that's coached on voice on how to describe it because most of the labelers out there just aren't as well versed on understanding the audio and voice, helped us a lot to to bring that back. And then, of course, deploying those models in production, seeing how customers interact with them, having them annotate all of the data helped us refine those models over time. A very interesting thing on the side, so we spoke about kind of the speech representation. The first guy who created the speech representation is a guy called Kempelen, Von Kempelen. So he created those analog machine that would represent effectively a human vocal vocal tract and try to produce that sound. He had to spend decades on that, and that kind of started producing vowels. But that's the same person that created a a chess machine, the first viral, let's say, chess machine that that would kind of simulate playing chess. [00:08:35] S0: Is this a mechanical Turk? [00:08:36] S1: It was called Turk. Yeah. Yeah. But the... But... Exactly. But the kind of crazy thing behind it, it was operated by a human. [00:08:42] S0: Yeah.
00:08:42 · John Collison, Mati Staniszewski
[00:08:42] S0: Yeah. [00:08:42] S1: Yeah. And it was all a fluke. Yes. And that's where the mechanical Turk... Which actually we use in that kind of data labeling production to to make that that that work there. [00:08:52] S0: Yeah. Yeah. And sorry. We we kinda jumped right in. But if you describe the Eleven business today, people think of you as the speech company. What... How should they actually think of your business to the extent you can describe the the big areas Text to speech, speech to text, voice agents, just, like... Break down the business for us.