00:04:21 · John Collison, Mati Staniszewski
[00:04:21] S0: You're saying kind of Britishness is an emergent property Exactly. In your voice models. [00:04:26] S1: Exactly. But... Yeah. And those kind of... Those two big parts, so it's encoding and decoding of how you create the voice. Yep. Super hard problem before and figured out too, how you then construct that in the sentence, how you get the context across so you can predict the next phonemes. [00:04:40] S0: Yes. [00:04:40] S1: So how you bring them together in a reliable and stable way while doing it quick, And these were kind of the two first big innovations in the voice models that continue to today. [00:04:48] S0: But... Okay. So if LLMs reason about text and, you know, words, subparts, tokens, as the way they think about the world, what is the equivalent of a token in the voice model? You mentioned phonemes a bunch. Like, what is that representation? [00:05:04] S1: So we do... We we store we store the voice embedding effectively for the speaker. Uh-huh. So you need that reference when you produce and create the speech. Yes. Of course, in the input to the voice model, you still get the text [00:05:16] S0: Mhmm. [00:05:16] S1: And you bring the the speaker and and and coding. And then when you produce speech, you do operate on the waveform or on the... Or effectively on the phoneme level of of that of that speech. And then when we kind of go the opposite, so of course, when... [00:05:31] S0: I'm sorry. What is a phoneme? Fill in my understanding.
00:05:31 · John Collison, Mati Staniszewski
[00:05:31] S0: I'm sorry. What is a phoneme? Fill in my understanding. [00:05:33] S1: It's like a a syllable deconstructed even to smaller elements. Okay. And these are effectively like the human sounds you can produce. [00:05:40] S0: Got [00:05:40] S1: it. So this would be like the most the close to that representation. But of course, in our models, now it's going to be a combination of not only operating on phoneme level, you also operate on the text level. You operate kind of in both in sync because when you are predicting the context, you need to understand Yes. How that sentence will get constructed, and especially if it's more of a streaming real time use case and, like, a voice agent setting, you need both parts to to work across. So it's similar to how you would operate on a tuck in level on the on the tech side. We operate on the tuck in level on the audio side. [00:06:12] S0: It feels like a big part of the magic of 11 was your voices were much more human sounding. How did you accomplish that?
00:06:12 · John Collison, Mati Staniszewski
[00:06:12] S0: It feels like a big part of the magic of 11 was your voices were much more human sounding. How did you accomplish that? [00:06:21] S1: So I'll kind of give you a quick quick synopsis of what... How we think about the models on the text to speech side today. In any model, you need you need architecture, you need compute, you need data. So architecture innovations were one thing. The data part was the second big thing. With audio, you will have you will have a lot of audio data available, but frequently, you will not have annotated in the right way. You won't have which speaker is speaking when. Some of the what is annotated, but the how isn't. So, like, as we are speaking now, what's the emotions that we use? What are the accents that we use? So we would invest a lot internally on effectively creating our own data labelers, our own team to be able to create those datasets that will be better. And that was a combination of, of course, like semiautomatic techniques and then manual techniques. Actually, a lot of the models that we did afterwards actually spun out from a lot of that research too. Speech to text model initially was a model we did for ourselves because the models on the market just weren't good to annotate that data. Then another brilliant researcher on our team was being able to construct it so we span it out as a model that we brought to the customers. [00:07:30] S0: So you've just been doing useful stuff in voice, and that has emerged with a whole bunch of products that you might not have expected because you find you're building useful stuff.
00:07:30 · John Collison, Mati Staniszewski
[00:07:30] S0: So you've just been doing useful stuff in voice, and that has emerged with a whole bunch of products that you might not have expected because you find you're building useful stuff. [00:07:39] S1: Exactly. Exactly. And that's kind of combination of data of being able to do it automatically, create a team that's coached on voice on how to describe it because most of the labelers out there just aren't as well versed on understanding the audio and voice, helped us a lot to to bring that back. And then, of course, deploying those models in production, seeing how customers interact with them, having them annotate all of the data helped us refine those models over time. A very interesting thing on the side, so we spoke about kind of the speech representation. The first guy who created the speech representation is a guy called Kempelen, Von Kempelen. So he created those analog machine that would represent effectively a human vocal vocal tract and try to produce that sound. He had to spend decades on that, and that kind of started producing vowels. But that's the same person that created a a chess machine, the first viral, let's say, chess machine that that would kind of simulate playing chess. [00:08:35] S0: Is this a mechanical Turk? [00:08:36] S1: It was called Turk. Yeah. Yeah. But the... But... Exactly. But the kind of crazy thing behind it, it was operated by a human. [00:08:42] S0: Yeah.
00:08:42 · John Collison, Mati Staniszewski
[00:08:42] S0: Yeah. [00:08:42] S1: Yeah. And it was all a fluke. Yes. And that's where the mechanical Turk... Which actually we use in that kind of data labeling production to to make that that that work there. [00:08:52] S0: Yeah. Yeah. And sorry. We we kinda jumped right in. But if you describe the Eleven business today, people think of you as the speech company. What... How should they actually think of your business to the extent you can describe the the big areas Text to speech, speech to text, voice agents, just, like... Break down the business for us.
00:09:08 · Mati Staniszewski
[00:09:08] S1: Cool. So in, like, the nutshell I'll describe, ElevenLabs is a research and product deployment company. We built foundational audio and voice models and then build a platform for businesses to transform how they communicate with their customers, with their employees, and that will apply through Yep. AI agents, from customer support, sales, hiring training, all the way through to marketing and storytelling through our creative tools. And in that set, we've created all types of foundational audio models, so text to speech models for producing speech, speech to text models that work over a 100 languages and happily beat others on benchmarks, all the way through to conversational models of how you loop them together Mhmm. To music, to other domains of audio. And then, of course, beyond the models, when you actually bring them to production, that's where the second level of the platform comes in, where that meets their businesses on the specific use case. So on the agent specific example, it would be how you now connect those models to the knowledge base, to telephony, to to the integrations that you need to perform the actions, how you evaluate and monitor the agent that behaves in the right way, how you build the right save rates. On the on the creative side, on the marketing side, it's how do you create a good ad so you can create a good video voice over for for one of the campaigns, how you create an article that's narrated with a specific voice that represents the brand in a good way. So that's where we combine kind of the models and understanding of the customers we work with into into one policy platform.