AI Tinkerers - "One-Shot"
00:32:34 · Joe Heitzeberg, Luke Harries
[00:32:34] S1: Do you do diarization, which is Yeah. For those who don't know, delineating of the different speakers? [00:32:38] S0: Yeah. So it does diarize... Excellent diarization. It also does character level time stamps. Cool. So that you can like Wow. [00:32:46] S1: Character sync and you can do apps on that. [00:32:48] S0: Yeah. You can do character level, word level. So if you're thinking like Joe's podcast, let's do an automated pipeline to make caption style videos, like, it's fantastic for [00:32:58] S1: doing that. Can I give it a bag of terms like custom brand names and technical terms to hint to [00:33:02] S0: the Yep? [00:33:03] S1: The AI? [00:33:03] S0: Yeah. Yeah. Cool. Actually, it does. It gets words like ElevenLabs. Right? One of my colleagues is called Thor. His full name is Thorsten. It gets words like that straight out of Cool. We have to definitely try out with this transcript. [00:33:15] S1: What about in the diarization, do you... Can you use the LLM to... In natively in in the one shot context, figure out the names of the speakers in context? [00:33:24] S0: I think that's done after. I think it just says the speaking name. I need to double [00:33:30] S1: check number and ID. Be good. Hi, Luke. And then you speak, and it knows that you're Luke. [00:33:35] S0: Right? I'm I'm not sure we have that built. Maybe we should. We We most likely should. [00:33:40] S1: It's a convenience thing for developers. [00:33:42] S0: Yeah. Maybe built into the SDKs could be a pretty cool one. [00:33:45] S1: Character level timestamps and it's accurate.
00:33:45 · Joe Heitzeberg, Luke Harries
[00:33:45] S1: Character level timestamps and it's accurate. [00:33:47] S0: That's Yeah. It's and super accurate. That's amazing. The other cool thing is it does audio events. So it does like laugh. Does It [00:33:53] S1: laugh. [00:33:53] S0: It does body laugh. So say you want to, like, alteredly... Automatically extract any coughs, Yeah. You [00:33:59] S1: Oh, that's really cool. Yeah. Oh, wow. To get rid of them. Right? Yeah. So you could do jump cutting essentially at that level. Yeah. I've noticed a lot of the the time stamping and some of the others aren't that accurate, and so your subtitles won't exactly match the words, and therefore, can't do automated jump cuts. [00:34:13] S0: The other cool thing about this was, the way we trained it, like, actually, lots of... We speak English. We're fluent in English. We're very lucky to be in English because if you're a different language, like a more niche language, lots of transcription just doesn't support you. So any sort of conversational AI, any sorts of transcription, medical transcription, none of those will work. And when we built this model, we were the first one to make transcription actually feasible in massive languages like Gujarati or Malay or Serbian. And it's like suddenly, if you were this, say, medical transcription app, you're like, oh, wow. My TAM's got bigger. I can now, like, launch an entire Serbian sales team. Yeah. And we're the only people to actually have Serbian support.
00:34:13 · Luke Harries, Joe Heitzeberg
[00:34:13] S0: The other cool thing about this was, the way we trained it, like, actually, lots of... We speak English. We're fluent in English. We're very lucky to be in English because if you're a different language, like a more niche language, lots of transcription just doesn't support you. So any sort of conversational AI, any sorts of transcription, medical transcription, none of those will work. And when we built this model, we were the first one to make transcription actually feasible in massive languages like Gujarati or Malay or Serbian. And it's like suddenly, if you were this, say, medical transcription app, you're like, oh, wow. My TAM's got bigger. I can now, like, launch an entire Serbian sales team. Yeah. And we're the only people to actually have Serbian support. [00:34:58] S1: Yeah. Cool. I'd love you to show us the conversation AI Yes. API stuff. I don't think most people are familiar with that, and voice agent stuff is really blowing up right now. It's super exciting. I think the the people just haven't connected the dots that, like, actually, no. The best text to speech that's best and fast as well, natural, etcetera, is actually... You can compose this into useful apps today. Yeah. With actually no coding. No. But there's also an interface to that. Could you walk us through
00:34:58 · Joe Heitzeberg, Luke Harries
[00:34:58] S1: Yeah. Cool. I'd love you to show us the conversation AI Yes. API stuff. I don't think most people are familiar with that, and voice agent stuff is really blowing up right now. It's super exciting. I think the the people just haven't connected the dots that, like, actually, no. The best text to speech that's best and fast as well, natural, etcetera, is actually... You can compose this into useful apps today. Yeah. With actually no coding. No. But there's also an interface to that. Could you walk us through [00:35:28] S0: the show? It's amazing. And if you're interested for a fun tidbit, when I first started ElevenLabs, Matty laid out this big vision. Yeah. And he's like, yeah. One of the things we're gonna do is conversational AI or voice AI agents. And I was like, never work. The latency [00:35:42] S1: is too high. It's like, [00:35:43] S0: mommy, I think of... Let's just give it a few months. And, yeah, I was completely wrong. [00:35:48] S1: Interesting. Yeah. It makes a huge difference. What is the what is the number of milliseconds of delay that you need to achieve to get a reply back that feels human? [00:35:56] S0: Yeah. What's interesting about this? Well, I think there's like the concrete numbers. It's like, it needs to be really less than a second. Sure. But if you think about what customers actually want, like the job to be done of certain calls, there's certain... There's different types of calls with companion apps. Super important at its low latency, super, like, emotional emergency. And, like, we're doing lots of work on that side.
00:35:56 · Luke Harries, Joe Heitzeberg
[00:35:56] S0: Yeah. What's interesting about this? Well, I think there's like the concrete numbers. It's like, it needs to be really less than a second. Sure. But if you think about what customers actually want, like the job to be done of certain calls, there's certain... There's different types of calls with companion apps. Super important at its low latency, super, like, emotional emergency. And, like, we're doing lots of work on that side. [00:36:23] S1: Cool. And... Which we haven't seen out yet. [00:36:26] S0: No. No. We've got new models in the works. Research timelines are always hard, but, yeah, yeah, there's there's more coming. But, I mean, we're already powering, some pretty big some pretty big companion apps. [00:36:39] S1: So latency, realism, emotion Yes. Are all important for conversational and and interactive... Trust building trust. [00:36:45] S0: Yeah. But the other angle as well is, like, loads of people use it in a business context. So say customer support or sales. And, actually, what you want is normally, like, a quick reply, which is, like, let me look into that for you. Yeah. But then it's really almost about, like, the rag bit. Yes. And, actually, normally, with a customer support call, what you're trying to do is, like, solve the user's problem. Yep. And there, what you care about is, like, overall time to solve their problem more so than that first. You still need Yeah.
00:36:45 · Luke Harries, Joe Heitzeberg
[00:36:45] S0: Yeah. But the other angle as well is, like, loads of people use it in a business context. So say customer support or sales. And, actually, what you want is normally, like, a quick reply, which is, like, let me look into that for you. Yeah. But then it's really almost about, like, the rag bit. Yes. And, actually, normally, with a customer support call, what you're trying to do is, like, solve the user's problem. Yep. And there, what you care about is, like, overall time to solve their problem more so than that first. You still need Yeah. [00:37:13] S1: If I'm gonna do a really good quality answer Yes. I don't expect a human to have the answer instantaneously. Go off and do a little thinking. We look some stuff up, you know, and then come back in a multi agent rag retrieval. Yeah. We'll do, you know, obviously, multiplex and a judgment and a report gen and then Yeah. Summarization, and that's the part that's spoken. And maybe ask a question, then give me the rest interactively. It's almost like there has to be a few threads going. And one is, like, the I'm in the moment with you, conversing with you, has to be attentive, quick, friendly, you know, while also background processing Yes. The retrieval. So are you... Do you do you... First of the system that you've built, does it expose that to me? Oh, okay. Yeah. I can system prompt what this is first level interaction will be like, and I can go kick off some background stuff, come back and push it into the complex.