00:04:47 · Mati Staniszewski, Andrew Reed
We try to healthy on the margins so we can continue investing with the assumption that it's better for us to figure out that stream and be able to be independent in that development. But then as the ambitions grew, we knew that we needed to train models. So we, of course, brought a lot of money externally as well. And I think, like, projecting to today, one thing that's clear for us is there's still so many of those niches that people don't tackle that you can start with and then step by step start opening them up. [00:05:17] S0: I think a lot of customers see ElevenLabs through their narrow needs, right? Maybe take a zoomed out view, like what is the suite of models that ElevenLabs works on? How do you prioritize them? How do you organize R and D, etcetera?
00:05:32 · Mati Staniszewski
[00:05:32] S1: So we started we started with the first text to speech model, so the model that could finally understand the context of what's being written. And based on that context understanding, get the right emotion, the right intonation from text. So it was a happy sentence, you get that happiness out. If it's a it's a dialogue, it can pronounce the dialogue out. And then continuously started adding that. So it started with the problem of of breaking down language barriers. The things you need to solve dubbing is transcription, so understanding, then the the translation, and then text to speech. So you first solve text to speech. Then we knew we needed to add a data component, is speech to text and being able to transcribe content in a in a great way. Then how we combine those models together. So that's kind of what the first three models in the first couple of years. And then, of course, the other things started happening across the space, which is that a lot of the reasoning models started becoming quick enough and smart enough at the same time where you could imagine those interactive experiences being possible. And that's where we started launching our more of the real time streaming models across audio and then combining those into conversational experiences. So added effectively all the stack, all the turn taking and orchestration to create a voice engine for a voice for a voice agent. And then on the other side, as we realized that the emotionality is something we can solve, we added some of the hardest modality in in in audio, which is music and being able to produce music. So today, we span entirety of the research of audio, whether it's text to speech, speech to text, combining those models together in both localization with dubbing, with orchestration with voice agent... Voice engine, and then being able to do that across music as well.
00:07:14 · Andrew Reed
[00:07:14] S0: And what's the... All those things and all that interesting development work, was there any, oh, wow moment in terms of what these products are capable of that you can remember?
00:07:24 · Mati Staniszewski
[00:07:24] S1: You know, there's so many, and it's kind of the bar changes for all of us. The first moment for us was... Well, first moment for us, they always use my voice as a testing voice because he has this weird accent. And and the first time was, like, when when we could replicate my voice based on a good sample, that was, a first wow moment to to myself. And you always go for this moment, like, this is not how my voice sounds like. And then you listen to yourself side by side, and it's, like, definitely how it sounds like, unfortunately. Then the the second moment was where we first got it to laugh, and people were like, okay. This is actually the thing that that makes the the whole experience more human. The the laughter, the pauses, the ums, the ums, the imperfections. So we started getting those out, and that was the moment for us because we made it to the top of hacker news with the first AI that can laugh model, which was a very proud moment for us. And and then, of course, for the years, kind of that extended where you might remember in 2023, 2024, there was a Javier Milei speech that went viral where you could speak other languages. So it was translated from... Into English. And and it was the first time where you could still hear his voice out there. So that's the kind of continuous wow moment that was was something that's completely impossible. And then we saw that happen time and time again with Narendra Modi, with president Zelensky, all the way to recently one of the, I feel like, pinnacles of the voice performance, Matthew McConaughey giving his newsletter and these iconic lines in in in Spanish and Portuguese, where for the first time, his family who speaks that language could hear him speak those languages too. But for more recent pieces, the two two ones that we are excited about bringing to production, I think the first one is finally figuring out the emotional intelligence in that interactive experience. So in the voice agent experience where it doesn't only get the right intonation and emotion, but can understand the other side. So if somebody is stressed, it gets and delivers that soothing, reassuring emotion. If someone is excited, maybe it matches that. If someone speaks slowly, it makes sure to slow down.
00:09:37 · Mati Staniszewski, Andrew Reed
And that emotional intelligence is something that we are finally seeing internally a path to solving, which will be just a continuous step change to what's possible. And then the second one, which will apply there but also apply into general audio spaces, audio general intelligence, where you can combine audio models together in one stream. So you could theoretically have a model that narrates, then pauses, and let's say, starts singing with that same continuous voice. And that's something that's extremely hard to combine today and and something that would be would be possible, I think, very, very soon. [00:10:13] S0: And voice, you mentioned voice agents. And it seems like everybody is, at least on the customer side, everyone's buying a voice agent. And I think intuitively, you think customer support, the old phone tree replacement. What's actually going on in the world of voice agents? What do think are the most interesting overlooked opportunities, spots where startup founders should focus?
00:10:36 · Mati Staniszewski
[00:10:36] S1: Yeah, of course, the customer support is probably the one that everybody heard and knows about very, very well. I think the second thing and the second thread we are seeing is increasing shift to revenue generating opportunities where voice agents can act in sales, whether it's inbound or outbound set... Of of sales. It doesn't replace the entire experience but takes and amplifies part of that experience. Maybe a good example is Deliveroo, where Deliveroo will have voice agents that contact the restaurants to capture their opening times. And based on their opening times, they can update the riders and drivers and, of course, the people ordering on when to get to to that work all the way through to the inbound sales where increasingly people that's a good example of Deutsche Telekom will be contacting to inquire about the service, inquire to buy a product. And instead of going for the dropdown, instead of going for the form, you can speak with the voice agent to leave that information. We do it ourselves too, we have a good metrics of an understanding of what's happening there. One, of course, so much simpler and quicker to go through instead of going through that form. But the second thing that started happening in that inbound sales flow is we we had a lot more information that people started leaving because they would speak about the use case they are coming with, but then where it's not working, where it's working, some of the other use cases that they are evaluating, which we can combine and then just deliver such a much better experience afterwards. On the overlooked side, I think my favorite example is the citizen support education and health care will completely change. On the citizen support, all of us would benefit from just generally better government access, whether that's understanding how to fill in the taxes that I think many of you went through earlier this month, all the way through to just learning how... What is the policy for travel abroad and and and and how that might affect the the space. We've recently seen that work deployed in government of Ukraine, who we think is, like, one of the most advanced governments on that front. We traveled to Ukraine working with their team, and what they are trying to solve is they they have a a government app, which every citizen can access and get information about what's happening.