WTF Online
00:16:53 · Nikhil Kamath, Mati Staniszewski
[00:16:53] S1: And say I'm a creator. How do I use ElevenLabs? [00:16:58] S0: The... Let's take this podcast as an example. Yeah. The... You know, after after we we we we we've finalized our conversation, something might have not been recorded, you want to add a line Mhmm. You can post process that with ElevenLabs, make sure that we effectively recreate my voice, we smoothen that line out. Mhmm. That's a very simple part. Then let's take a second step. You wanna take this podcast and you want to localize it outside of English, you wanna bring it to Spanish or French or Hindi. Mhmm. And here, you go to our interface, and you can both automatically dab the entire podcast and then go for any corrections that you need. So when we did a podcast with... Between Lex Fridman and prime minister Narendra Modi, that was effectively going through that workflow. [00:17:47] S1: Did you dub that? Yeah. Mhmm. [00:17:49] S0: And that was, like, going through the Human on the Loop, making sure the corrections are there, and that was, like, well, like, an like an very long conversation, incredible test of the of the software, but then you can enjoy that in in a number of other languages. [00:18:01] S1: How much does it cost, say, example, how much did Lex and our prime minister... How much did it cost to dub?
00:18:01 · Nikhil Kamath, Mati Staniszewski
[00:18:01] S1: How much does it cost, say, example, how much did Lex and our prime minister... How much did it cost to dub? [00:18:08] S0: You know, this one was harder. In in general, like, if you just wanted to automatically dub, then, like, an hour podcast will be few dollars, up to few dollars. Mhmm. But with with any conversation like like the one of the prime minister, we did employ a wider set of of people that would actually check every translation, with working with the Lex team and the other side of the team to to make sure that the content is preserved, like, we we know... It's like calibre of the conversation where every word matters. So that was maybe a more unique on that side. [00:18:42] S1: We have this use case. We want to dub all the content we make into regional languages, but we've been struggling with solutions really. More than the cost of it, I feel like the emotion behind what is being said is always lost and everything sounds a bit robotic. How do you change that? I've not tried ElevenLabs.
00:18:42 · Nikhil Kamath, Mati Staniszewski
[00:18:42] S1: We have this use case. We want to dub all the content we make into regional languages, but we've been struggling with solutions really. More than the cost of it, I feel like the emotion behind what is being said is always lost and everything sounds a bit robotic. How do you change that? I've not tried ElevenLabs. [00:19:07] S0: Then perfect. It's like almost a cue to show you. No. I think I think that's like one of the big parts that I think we we now can bring to the to the equation that was lost. Like, how do you both preserve the voice, preserve intonation, preserve emotion? And the way, you know, we do it today is we we, of course, detect your speech, we then detect your speakers, we we translate it. But then, with the moment we generate the speech, it actually captures the contextual information from the previous response and tries to recreate that in the in the new target language. Like, effectively, our model, the way we've... We created that for the localization use case, has been trying to match pairs of languages that we have and how would you deliver that in professional setting and how you can reapply that in the automatic DAB so I think it works really well The the the hard things is, like, you know, of course, when you have, like, amplitude of emotions, like Mhmm. Laughter, scream, and you then shift the ordering of the sentence, then it's hard. [00:20:08] S1: Like, see, course. Say, for example, Mati is smiling while you're saying a certain thing. Mati is the right way to say it? [00:20:14] S0: Yeah. Yeah. Yeah. Yeah.
00:20:14 · Mati Staniszewski, Nikhil Kamath
[00:20:14] S0: Yeah. Yeah. Yeah. Yeah. [00:20:15] S1: When you smile when you say a certain thing in English, isn't it hard to take that emotion into Hindi, for example? [00:20:24] S0: The... It's, you know, it's not easy, a 100%. And then there's, like, kind of two... Like, there's the present and the future in the current way, you would still translate it, you would of course dub it. I think you can preserve the emotion, but of course my movement of my lips when I smile will... On the language structure it might be in a different part of the sentence so like maybe an easy example is German from English to German, German usually will have noun at the end so like yes that will exactly happen you will like smile at a different part than the conversation you can do some tricks to try to to still to smoothen it out but yes, there will be some discrepancy so that's present can still work in the future, you know, there are some limitations of the current delivery softwares you'll likely do a lip animation too or lip reanimation to move the lips tiny bit to apply to where you would smile in that part of the sentence so I think that will happen a 100% I hope this happens this year [00:21:30] S1: is that something you would do? [00:21:32] S0: We would want to do it I think like as you think about anything that can amplify the audio experience we want to bring that into our flow. So on that creative side like today if you were to localize hopefully in the next few months you will be able to animate lips straight away as well.
00:21:32 · Mati Staniszewski, Nikhil Kamath
[00:21:32] S0: We would want to do it I think like as you think about anything that can amplify the audio experience we want to bring that into our flow. So on that creative side like today if you were to localize hopefully in the next few months you will be able to animate lips straight away as well. [00:21:48] S1: Could a tiny company be built just around this? Just around changing how the lip moves when something is being dubbed?
00:21:48 · Nikhil Kamath, Mati Staniszewski
[00:21:48] S1: Could a tiny company be built just around this? Just around changing how the lip moves when something is being dubbed? [00:21:58] S0: It could. There's there's actually a good set of companies that are, like, crashing it and going very quickly. Right. You know, of course, think this is an entry point, not the end point. Mhmm. They will likely try to add more of that functionality, but but 100%, you know, it's like... And the the... Of course, now we speak about the static part Mhmm. But the moment you start moving into the real time part, those tiny companies now can be huge companies. And that's kinda like, you know, we spoke about the creative side. If you think about the agent side, and I think there are some use cases here for for for for both the lip animation and even for for for your... For you Mhmm. I have a I have a one idea for you. So, you know, we spoke about the creative side. On the agent side, effectively, what we do is is is allow you to create a voice experience, bring any knowledge that that that you want inside of that experience, and and then deploy that in in in different settings while monitoring, evaluating how it performs. I don't know if you're familiar with Masterclass. Yeah. So Masterclass works with us where they would recreate and work with people like Gordon Ramsay [00:23:07] S1: Mhmm.