00:15:19 · Nikhil Kamath, Mati Staniszewski
[00:15:19] S1: Mhmm. [00:15:20] S0: I think they will they will they will continue proving that we can be ahead. But then 100%, you know, we cannot just rely on research. We need to have more both defense points, but also how we deliver value effectively to get customers, so it's kind of independent of the research work. And here, from our perspective, we have kind of two main things. One is we have a creative platform offering and then an agents platform offering. Creative platform is helping with narrations, voice overs Mhmm. Localization from individuals, so from creators, podcasters Mhmm. News articles, through to some media entertainment companies like Star. Mhmm. And then on agents, we help automate voice agents and and and customer experience, training, education. So Meesho would be a great example. [00:16:09] S1: Can you can you describe to us what ElevenLabs does overall, like, in a in a simplified manner that anybody could understand? [00:16:18] S0: We built foundational audio AI technology, so we built models to make speech sound incredible and speech be understood, and then we built products on top of that research that help you create voice agents that can elevate your customer experience, can elevate and train people internally or externally. And then we help on the creative side with customers and how they can communicate with with the user base. [00:16:47] S1: So 50% of your business is creator and 50% is enterprise in a way? [00:16:52] S0: That's right. [00:16:53] S1: And say I'm a creator. How do I use ElevenLabs?
00:16:53 · Nikhil Kamath, Mati Staniszewski
[00:16:53] S1: And say I'm a creator. How do I use ElevenLabs? [00:16:58] S0: The... Let's take this podcast as an example. Yeah. The... You know, after after we we we we we've finalized our conversation, something might have not been recorded, you want to add a line Mhmm. You can post process that with ElevenLabs, make sure that we effectively recreate my voice, we smoothen that line out. Mhmm. That's a very simple part. Then let's take a second step. You wanna take this podcast and you want to localize it outside of English, you wanna bring it to Spanish or French or Hindi. Mhmm. And here, you go to our interface, and you can both automatically dab the entire podcast and then go for any corrections that you need. So when we did a podcast with... Between Lex Fridman and prime minister Narendra Modi, that was effectively going through that workflow. [00:17:47] S1: Did you dub that? Yeah. Mhmm. [00:17:49] S0: And that was, like, going through the Human on the Loop, making sure the corrections are there, and that was, like, well, like, an like an very long conversation, incredible test of the of the software, but then you can enjoy that in in a number of other languages. [00:18:01] S1: How much does it cost, say, example, how much did Lex and our prime minister... How much did it cost to dub?
00:18:01 · Nikhil Kamath, Mati Staniszewski
[00:18:01] S1: How much does it cost, say, example, how much did Lex and our prime minister... How much did it cost to dub? [00:18:08] S0: You know, this one was harder. In in general, like, if you just wanted to automatically dub, then, like, an hour podcast will be few dollars, up to few dollars. Mhmm. But with with any conversation like like the one of the prime minister, we did employ a wider set of of people that would actually check every translation, with working with the Lex team and the other side of the team to to make sure that the content is preserved, like, we we know... It's like calibre of the conversation where every word matters. So that was maybe a more unique on that side. [00:18:42] S1: We have this use case. We want to dub all the content we make into regional languages, but we've been struggling with solutions really. More than the cost of it, I feel like the emotion behind what is being said is always lost and everything sounds a bit robotic. How do you change that? I've not tried ElevenLabs.
00:18:42 · Nikhil Kamath, Mati Staniszewski
[00:18:42] S1: We have this use case. We want to dub all the content we make into regional languages, but we've been struggling with solutions really. More than the cost of it, I feel like the emotion behind what is being said is always lost and everything sounds a bit robotic. How do you change that? I've not tried ElevenLabs. [00:19:07] S0: Then perfect. It's like almost a cue to show you. No. I think I think that's like one of the big parts that I think we we now can bring to the to the equation that was lost. Like, how do you both preserve the voice, preserve intonation, preserve emotion? And the way, you know, we do it today is we we, of course, detect your speech, we then detect your speakers, we we translate it. But then, with the moment we generate the speech, it actually captures the contextual information from the previous response and tries to recreate that in the in the new target language. Like, effectively, our model, the way we've... We created that for the localization use case, has been trying to match pairs of languages that we have and how would you deliver that in professional setting and how you can reapply that in the automatic DAB so I think it works really well The the the hard things is, like, you know, of course, when you have, like, amplitude of emotions, like Mhmm. Laughter, scream, and you then shift the ordering of the sentence, then it's hard. [00:20:08] S1: Like, see, course. Say, for example, Mati is smiling while you're saying a certain thing. Mati is the right way to say it? [00:20:14] S0: Yeah. Yeah. Yeah. Yeah.
00:20:14 · Mati Staniszewski, Nikhil Kamath
[00:20:14] S0: Yeah. Yeah. Yeah. Yeah. [00:20:15] S1: When you smile when you say a certain thing in English, isn't it hard to take that emotion into Hindi, for example? [00:20:24] S0: The... It's, you know, it's not easy, a 100%. And then there's, like, kind of two... Like, there's the present and the future in the current way, you would still translate it, you would of course dub it. I think you can preserve the emotion, but of course my movement of my lips when I smile will... On the language structure it might be in a different part of the sentence so like maybe an easy example is German from English to German, German usually will have noun at the end so like yes that will exactly happen you will like smile at a different part than the conversation you can do some tricks to try to to still to smoothen it out but yes, there will be some discrepancy so that's present can still work in the future, you know, there are some limitations of the current delivery softwares you'll likely do a lip animation too or lip reanimation to move the lips tiny bit to apply to where you would smile in that part of the sentence so I think that will happen a 100% I hope this happens this year [00:21:30] S1: is that something you would do? [00:21:32] S0: We would want to do it I think like as you think about anything that can amplify the audio experience we want to bring that into our flow. So on that creative side like today if you were to localize hopefully in the next few months you will be able to animate lips straight away as well.
00:21:32 · Mati Staniszewski, Nikhil Kamath
[00:21:32] S0: We would want to do it I think like as you think about anything that can amplify the audio experience we want to bring that into our flow. So on that creative side like today if you were to localize hopefully in the next few months you will be able to animate lips straight away as well. [00:21:48] S1: Could a tiny company be built just around this? Just around changing how the lip moves when something is being dubbed?