00:01:05 · Mati Staniszewski
[00:01:05] S1: We did. It was still a year when the topics of the day were crypto and metaverse. So 2022 was still a year where everybody was obsessed about those two. So it was a perfect time because we could actually focus and build a lot on the AI side. What were you doing before you founded this company? I was at Palantir. So I was helping build optimization models and bring optimization models to the customers. So NHS during the COVID response on how you distribute vaccines across UK or working with oil and gas industry and figuring out how to optimize their energy work. A lot of optimization models and then working with customers on actually figuring out how you bring that into their production. My co founder was at Google and he did a lot of the text models for knowledge graph. Before that, did research at university around image visual models. Incredible brain, the smartest person I know for developing a lot of the research work. I was happily on that intersection of building the product and bringing it out to the customers. So that was before Palantir's BlackRock, building risk models, bringing it out to customers and by background, study mathematics. Good intersection of the things I liked. And now at ElevenLabs, that's also what I do. That's also from the company lens. And when we started, that was the whole goal of like, can we combine research and product deployment under one hood? On the research side, built all the audio models, starting with model to produce speech, text to speech model, and that was that 2022, the first model that could finally cross that human like quality. Then over time, now it's entirety of audio models, transcription models, localization models, orchestration for voice conversations with agents? And then on the product side, can we unify that as one platform that helps people, businesses transform how they would communicate with their audience?
00:02:55 · David Senra, Mati Staniszewski
[00:02:55] S0: Wait, so the first idea, you started with audio. [00:02:58] S1: We started with audio. Why? The actual trigger point comes from where you're from, from Poland. Very peculiar thing. If you watch a movie in Polish, all the voices, whether it's a male voice or whether it's a female voice, get narrated with one single character. So you have one voice narrating the whole movie. All the emotional intonation disappears. It's still... So we grew up with this. Everybody in Palantir groups have with it. It's like, you know, one person dabbing. And then in 2021, we retested or re experienced this still happening in that content. Then second thing happened, was [00:03:34] S0: Wait, it was still happening in movies in 2021, but you'd watch in Poland? Yes. Same thing that was happening when [00:03:39] S1: you were [00:03:39] S0: a kid.
00:03:39 · David Senra, Mati Staniszewski
[00:03:39] S0: a kid. [00:03:39] S1: Exactly. It's crazy because it's, of course, it's cheaper, easier to do, quicker timeline, but the quality is poor. As you can imagine, it's a pretty terrible experience. That was like a trigger point. In the future, that experience will be completely different. You will have original voice, original emotions, original intonation, and actually be able to experience that in that incredible way, which is very timely because just a week ago, released finally a model that is able to do that extremely well and finally bring the content from one language to another. But maybe more broadly, that was also our eye opening of how we interact with technology, how you can interact with food technology will change. And that will apply the language barrier that will break, but also just the general conversations with devices, digital world will happen differently across different modalities, across different channels. And we wanted to [00:04:32] S0: This is what you thought back in 2022 or what do think today? [00:04:34] S1: In 2022, we knew that you will you will need to unlock the stories, the content across the modalities and channels. We didn't yet know how quickly the shift from static to interactive will happen. We hope this will happen, but we didn't know how quickly. [00:04:48] S0: So the first idea was dub static content that I was watching. I'm watching a movie in Poland. I want this to actually feel like I'm watching in my native language. [00:04:58] S1: Exactly.
00:04:58 · Mati Staniszewski, David Senra
[00:04:58] S1: Exactly. [00:04:59] S0: That had to appear like a tiny business though, right, at the time? Like if that's your initial idea, you weren't going into it thinking this is going be a giant business, like you're one of the fastest growing startups today.
00:05:10 · Mati Staniszewski, David Senra
[00:05:10] S1: We actually thought it was a huge business at the time, too. Thought it was like if we think about all the content, all the stories out there, how incredible if they were available in audio and whether it's some of the biggest streaming companies, whether that's TV, whether that's the conversations, all of those could be actually delivered in the local language. We thought it's actually huge and I still think it's huge. And then if you shift this to to the end of the conversation we are having now, could this be in the future, this version where I speak Polish and you understand me in English, or I speak English and you understand me in any language that you want? In Hitchhiker's Guide to Galaxy, there's this idea of a bubble fish that you put next to the ear, and you can understand everything around you regardless of the language they speak. So, we knew that we will get there. But initially, it was like dubbing. So, I'll give you like a full way of how kind of it progressed. Initially, it was dubbing. And then as we started diving into what do we need to do to solve dubbing, we realized there are three steps in dabbing process. There is transcription step, then a translation step to another language, and then you need to regenerate that in another language. But the research that existed at the time for each of those steps wasn't very good. [00:06:18] S0: Which companies were doing the research? [00:06:21] S1: Those are good, NVIDIA models, open source models. [00:06:24] S0: But wait, it's just an independent researcher, so not any of the big companies?
00:06:27 · Mati Staniszewski
[00:06:27] S1: No, everything was poor. Everything was robotic. Everything was pretty robotic. It was still like you immediately tell that it was like a robotic voice and nothing really crossed that uncanny valley yet. There was one good open source paper, an open source repo that I'm trying to recall that was pretty good, but very unstable and still took so much time to generate. That was like the best. NVIDIA had some good research on some of those components for, especially on the speech to text side. Translation was okay ish. DeepL at the time was doing incredibly well. Then you had the kind of Google translate version of that. But all of the components to do dub were not good enough yet. So that's part one. The research wasn't there. And part two, when we started testing the dabbing idea with a lot of initiate creators, the message we are getting back from the creators is like, Great, I would love to get dabbing one day, but today I have different problems. My problems are I want to post produce and change the line that was recorded in the wrong way. Or before I record my video, want to be able to narrate the script and see how it sounds. Or instead of me speaking over a video, could I just have AI speak over that? So before I even think about dubbing, can you give me that? And that was, for us, was like, okay, before we can solve dubbing, let's solve the research component to generate speech and make it sound great. And on the product side, let's actually deliver for people to be able to just narrate content. So like, let's pan down and ignore for a second the language shift that just help bringing content alive with audio with a high quality.