00:06:12 · John Collison, Mati Staniszewski
[00:06:12] S0: It feels like a big part of the magic of 11 was your voices were much more human sounding. How did you accomplish that? [00:06:21] S1: So I'll kind of give you a quick quick synopsis of what... How we think about the models on the text to speech side today. In any model, you need you need architecture, you need compute, you need data. So architecture innovations were one thing. The data part was the second big thing. With audio, you will have you will have a lot of audio data available, but frequently, you will not have annotated in the right way. You won't have which speaker is speaking when. Some of the what is annotated, but the how isn't. So, like, as we are speaking now, what's the emotions that we use? What are the accents that we use? So we would invest a lot internally on effectively creating our own data labelers, our own team to be able to create those datasets that will be better. And that was a combination of, of course, like semiautomatic techniques and then manual techniques. Actually, a lot of the models that we did afterwards actually spun out from a lot of that research too. Speech to text model initially was a model we did for ourselves because the models on the market just weren't good to annotate that data. Then another brilliant researcher on our team was being able to construct it so we span it out as a model that we brought to the customers. [00:07:30] S0: So you've just been doing useful stuff in voice, and that has emerged with a whole bunch of products that you might not have expected because you find you're building useful stuff.
00:07:30 · John Collison, Mati Staniszewski
[00:07:30] S0: So you've just been doing useful stuff in voice, and that has emerged with a whole bunch of products that you might not have expected because you find you're building useful stuff. [00:07:39] S1: Exactly. Exactly. And that's kind of combination of data of being able to do it automatically, create a team that's coached on voice on how to describe it because most of the labelers out there just aren't as well versed on understanding the audio and voice, helped us a lot to to bring that back. And then, of course, deploying those models in production, seeing how customers interact with them, having them annotate all of the data helped us refine those models over time. A very interesting thing on the side, so we spoke about kind of the speech representation. The first guy who created the speech representation is a guy called Kempelen, Von Kempelen. So he created those analog machine that would represent effectively a human vocal vocal tract and try to produce that sound. He had to spend decades on that, and that kind of started producing vowels. But that's the same person that created a a chess machine, the first viral, let's say, chess machine that that would kind of simulate playing chess. [00:08:35] S0: Is this a mechanical Turk? [00:08:36] S1: It was called Turk. Yeah. Yeah. But the... But... Exactly. But the kind of crazy thing behind it, it was operated by a human. [00:08:42] S0: Yeah.
00:08:42 · John Collison, Mati Staniszewski
[00:08:42] S0: Yeah. [00:08:42] S1: Yeah. And it was all a fluke. Yes. And that's where the mechanical Turk... Which actually we use in that kind of data labeling production to to make that that that work there. [00:08:52] S0: Yeah. Yeah. And sorry. We we kinda jumped right in. But if you describe the Eleven business today, people think of you as the speech company. What... How should they actually think of your business to the extent you can describe the the big areas Text to speech, speech to text, voice agents, just, like... Break down the business for us.
00:09:08 · Mati Staniszewski
[00:09:08] S1: Cool. So in, like, the nutshell I'll describe, ElevenLabs is a research and product deployment company. We built foundational audio and voice models and then build a platform for businesses to transform how they communicate with their customers, with their employees, and that will apply through Yep. AI agents, from customer support, sales, hiring training, all the way through to marketing and storytelling through our creative tools. And in that set, we've created all types of foundational audio models, so text to speech models for producing speech, speech to text models that work over a 100 languages and happily beat others on benchmarks, all the way through to conversational models of how you loop them together Mhmm. To music, to other domains of audio. And then, of course, beyond the models, when you actually bring them to production, that's where the second level of the platform comes in, where that meets their businesses on the specific use case. So on the agent specific example, it would be how you now connect those models to the knowledge base, to telephony, to to the integrations that you need to perform the actions, how you evaluate and monitor the agent that behaves in the right way, how you build the right save rates. On the on the creative side, on the marketing side, it's how do you create a good ad so you can create a good video voice over for for one of the campaigns, how you create an article that's narrated with a specific voice that represents the brand in a good way. So that's where we combine kind of the models and understanding of the customers we work with into into one policy platform.
00:10:44 · John Collison, Mati Staniszewski
[00:10:44] S0: Every platform company has this question about how far they go into applications. So how do you think about where you go horizontal and power the whole ecosystem versus where you develop application? Because, like, you can imagine there being a whole ecosystem of closed captioning tools that grow up that, again, are built on the ElevenLabs tech. Like, it's not necessarily a space that you would have to go after yourself. [00:11:06] S1: I think the big difference between... And your kind of question, today, we we we we see ourselves as a platform where if you're building a horizontal use case in your business, a great place to come. If you have a lot of domain specificity, that's where I see a lot of kind of the application companies forming over time, where they will, where where that's the specifically not the spaces we will go into. [00:11:28] S0: And I think it also is interesting when the tech is moving as quickly as it is here. It's one thing, you know, with SaaS where you get these, like, vertical specific providers, but I would imagine one of the biggest risks for you guys in being intermediated is if there is, you know, like in this example, a closed captioning service that is on a two versions... Old version of ElevenLabs and hasn't upgraded, that's a problem because you want people to be using the latest and greatest model that you've developed, and you'll be kind of deploying new capabilities every week. And I presume that's part of your thinking, is that just when it's moving that quickly, you need to go direct in a of cases.
00:11:28 · John Collison, Mati Staniszewski
[00:11:28] S0: And I think it also is interesting when the tech is moving as quickly as it is here. It's one thing, you know, with SaaS where you get these, like, vertical specific providers, but I would imagine one of the biggest risks for you guys in being intermediated is if there is, you know, like in this example, a closed captioning service that is on a two versions... Old version of ElevenLabs and hasn't upgraded, that's a problem because you want people to be using the latest and greatest model that you've developed, and you'll be kind of deploying new capabilities every week. And I presume that's part of your thinking, is that just when it's moving that quickly, you need to go direct in a of cases. [00:12:04] S1: That's right. In closed captioning, like, you know, here, already now, we know that our services is going to be able to tackle, like, 99.9% of the cases that customers have. And then there's, like, added benefit of we work with health care customers where we will create custom models for those customers where it will get that transcription perfectly. [00:12:24] S0: The context is the tricky thing in closed captions where, like, we talk a lot about a lot of technical stuff on on this. [00:12:30] S1: Yeah. For sure. And that's where you need, like, effectively, like, a dictionary of words that do the tech before beforehand, which which, as we work with the businesses, we know we need to we need to embed in that creation process.