00:02:36 · Mati Staniszewski
[00:02:36] S1: Yeah. That goes... I think that continues that great lack in timing because we started in 2022. For those of you working in the domain at the time, that that was the year of crypto and metaverse. Nobody was still working on the AI side. Even further, people were starting to work, of course, on the text models, on the visual models, but audio as a domain was still considered a big niche. There's so few researchers in the space working on on that work. So for us, that was a good part of picking that domain where, a, we were excited about where that future is called. We felt the people around just didn't realize the value of that domain. But three, the requirements of what you needed to solve were very different. The audio models were smaller, so you don't need as much compute as you need for some of the other sister domains. The data needs are big, but while there's a lot of audio data, we knew that the thing to actually get that audio working, you will need to figure out how to transcribe a lot of that data and annotate all of that data, which we knew we can do. And then ultimately, it all boiled down to architectural side of can we can we solve that part in a good way. And here, my co founder is one of the smartest people I know and a great researcher and has been able to assemble some of the best people in audio to help us. And we took a slightly untraditional approach at the time. We started in London. We had a lot of people between London and Warsaw and started a company in remote... Completely remote way. So we wanted to hire the best researchers wherever they were. We were going for the classic GitHub scraping and and and trying to reach people based on their work instead of based on their presence. And based on that work, we would reach out to those people. We would always share our samples and try to get them to join the team. And that's how we assembled the first the first set of of of people who we think are some of the best researchers in that audio domain. And through the years, they still help us crank a lot of those models in... Into production. Then we launched the product. I think the slightly different approach we took was monetizing very quickly. So trying to get some of the revenue stream back so we can fund a lot of the work in the models.
00:04:47 · Mati Staniszewski, Andrew Reed
We try to healthy on the margins so we can continue investing with the assumption that it's better for us to figure out that stream and be able to be independent in that development. But then as the ambitions grew, we knew that we needed to train models. So we, of course, brought a lot of money externally as well. And I think, like, projecting to today, one thing that's clear for us is there's still so many of those niches that people don't tackle that you can start with and then step by step start opening them up. [00:05:17] S0: I think a lot of customers see ElevenLabs through their narrow needs, right? Maybe take a zoomed out view, like what is the suite of models that ElevenLabs works on? How do you prioritize them? How do you organize R and D, etcetera?
00:05:32 · Mati Staniszewski
[00:05:32] S1: So we started we started with the first text to speech model, so the model that could finally understand the context of what's being written. And based on that context understanding, get the right emotion, the right intonation from text. So it was a happy sentence, you get that happiness out. If it's a it's a dialogue, it can pronounce the dialogue out. And then continuously started adding that. So it started with the problem of of breaking down language barriers. The things you need to solve dubbing is transcription, so understanding, then the the translation, and then text to speech. So you first solve text to speech. Then we knew we needed to add a data component, is speech to text and being able to transcribe content in a in a great way. Then how we combine those models together. So that's kind of what the first three models in the first couple of years. And then, of course, the other things started happening across the space, which is that a lot of the reasoning models started becoming quick enough and smart enough at the same time where you could imagine those interactive experiences being possible. And that's where we started launching our more of the real time streaming models across audio and then combining those into conversational experiences. So added effectively all the stack, all the turn taking and orchestration to create a voice engine for a voice for a voice agent. And then on the other side, as we realized that the emotionality is something we can solve, we added some of the hardest modality in in in audio, which is music and being able to produce music. So today, we span entirety of the research of audio, whether it's text to speech, speech to text, combining those models together in both localization with dubbing, with orchestration with voice agent... Voice engine, and then being able to do that across music as well.
00:07:14 · Andrew Reed
[00:07:14] S0: And what's the... All those things and all that interesting development work, was there any, oh, wow moment in terms of what these products are capable of that you can remember?
00:07:24 · Mati Staniszewski
[00:07:24] S1: You know, there's so many, and it's kind of the bar changes for all of us. The first moment for us was... Well, first moment for us, they always use my voice as a testing voice because he has this weird accent. And and the first time was, like, when when we could replicate my voice based on a good sample, that was, a first wow moment to to myself. And you always go for this moment, like, this is not how my voice sounds like. And then you listen to yourself side by side, and it's, like, definitely how it sounds like, unfortunately. Then the the second moment was where we first got it to laugh, and people were like, okay. This is actually the thing that that makes the the whole experience more human. The the laughter, the pauses, the ums, the ums, the imperfections. So we started getting those out, and that was the moment for us because we made it to the top of hacker news with the first AI that can laugh model, which was a very proud moment for us. And and then, of course, for the years, kind of that extended where you might remember in 2023, 2024, there was a Javier Milei speech that went viral where you could speak other languages. So it was translated from... Into English. And and it was the first time where you could still hear his voice out there. So that's the kind of continuous wow moment that was was something that's completely impossible. And then we saw that happen time and time again with Narendra Modi, with president Zelensky, all the way to recently one of the, I feel like, pinnacles of the voice performance, Matthew McConaughey giving his newsletter and these iconic lines in in in Spanish and Portuguese, where for the first time, his family who speaks that language could hear him speak those languages too. But for more recent pieces, the two two ones that we are excited about bringing to production, I think the first one is finally figuring out the emotional intelligence in that interactive experience. So in the voice agent experience where it doesn't only get the right intonation and emotion, but can understand the other side. So if somebody is stressed, it gets and delivers that soothing, reassuring emotion. If someone is excited, maybe it matches that. If someone speaks slowly, it makes sure to slow down.
00:09:37 · Mati Staniszewski, Andrew Reed
And that emotional intelligence is something that we are finally seeing internally a path to solving, which will be just a continuous step change to what's possible. And then the second one, which will apply there but also apply into general audio spaces, audio general intelligence, where you can combine audio models together in one stream. So you could theoretically have a model that narrates, then pauses, and let's say, starts singing with that same continuous voice. And that's something that's extremely hard to combine today and and something that would be would be possible, I think, very, very soon. [00:10:13] S0: And voice, you mentioned voice agents. And it seems like everybody is, at least on the customer side, everyone's buying a voice agent. And I think intuitively, you think customer support, the old phone tree replacement. What's actually going on in the world of voice agents? What do think are the most interesting overlooked opportunities, spots where startup founders should focus?