AI Tinkerers - "One-Shot"
00:31:24 · Joe Heitzeberg, Luke Harries
[00:31:24] S1: That's cool. [00:31:25] S0: And it's super cheap because it's just raw API usage. [00:31:27] S1: Yeah. Love it. I think... I don't think people are familiar with the speech to text at ElevenLabs. That's rather new. Right? [00:31:34] S0: Yes. Yeah. [00:31:34] S1: Yeah. I have a I have a quick curious question. I admit I haven't tried it yet. I've done long transcriptions on videos and diarization and all the bits. So just... I wonder if you could just quickly go through. So some of the limitations of just using something open source like Whisper Yep. Are actually, like, context length Yes. Or Gemini. Gemini is great, on, input tokens Yeah. Not output. So you can't just say, here's a two hour video. Give me the whole transcript. It'll stop halfway. Yeah. Do you break through that? [00:32:02] S0: Yeah. 100%. And the the interesting thing particularly for something like this with podcast transcripts is small errors compound dramatically Yeah. Because it will shift the output such that, like, it will end up... If it makes a mistake early on Yeah. Often it'll be misaligned then for the entire podcast. Yeah. And you have to, like, go through. Sure. Yeah, we built the world's best speech to text model called Scribe. [00:32:27] S1: Mhmm. [00:32:28] S0: And and it still beats, I think, OpenAI released a new model recently, still beats that. Beats Gemini. [00:32:34] S1: Do you do diarization, which is Yeah. For those who don't know, delineating of the different speakers?
00:32:34 · Joe Heitzeberg, Luke Harries
[00:32:34] S1: Do you do diarization, which is Yeah. For those who don't know, delineating of the different speakers? [00:32:38] S0: Yeah. So it does diarize... Excellent diarization. It also does character level time stamps. Cool. So that you can like Wow. [00:32:46] S1: Character sync and you can do apps on that. [00:32:48] S0: Yeah. You can do character level, word level. So if you're thinking like Joe's podcast, let's do an automated pipeline to make caption style videos, like, it's fantastic for [00:32:58] S1: doing that. Can I give it a bag of terms like custom brand names and technical terms to hint to [00:33:02] S0: the Yep? [00:33:03] S1: The AI? [00:33:03] S0: Yeah. Yeah. Cool. Actually, it does. It gets words like ElevenLabs. Right? One of my colleagues is called Thor. His full name is Thorsten. It gets words like that straight out of Cool. We have to definitely try out with this transcript. [00:33:15] S1: What about in the diarization, do you... Can you use the LLM to... In natively in in the one shot context, figure out the names of the speakers in context? [00:33:24] S0: I think that's done after. I think it just says the speaking name. I need to double [00:33:30] S1: check number and ID. Be good. Hi, Luke. And then you speak, and it knows that you're Luke. [00:33:35] S0: Right? I'm I'm not sure we have that built. Maybe we should. We We most likely should. [00:33:40] S1: It's a convenience thing for developers. [00:33:42] S0: Yeah. Maybe built into the SDKs could be a pretty cool one. [00:33:45] S1: Character level timestamps and it's accurate.
00:33:45 · Joe Heitzeberg, Luke Harries
[00:33:45] S1: Character level timestamps and it's accurate. [00:33:47] S0: That's Yeah. It's and super accurate. That's amazing. The other cool thing is it does audio events. So it does like laugh. Does It [00:33:53] S1: laugh. [00:33:53] S0: It does body laugh. So say you want to, like, alteredly... Automatically extract any coughs, Yeah. You [00:33:59] S1: Oh, that's really cool. Yeah. Oh, wow. To get rid of them. Right? Yeah. So you could do jump cutting essentially at that level. Yeah. I've noticed a lot of the the time stamping and some of the others aren't that accurate, and so your subtitles won't exactly match the words, and therefore, can't do automated jump cuts. [00:34:13] S0: The other cool thing about this was, the way we trained it, like, actually, lots of... We speak English. We're fluent in English. We're very lucky to be in English because if you're a different language, like a more niche language, lots of transcription just doesn't support you. So any sort of conversational AI, any sorts of transcription, medical transcription, none of those will work. And when we built this model, we were the first one to make transcription actually feasible in massive languages like Gujarati or Malay or Serbian. And it's like suddenly, if you were this, say, medical transcription app, you're like, oh, wow. My TAM's got bigger. I can now, like, launch an entire Serbian sales team. Yeah. And we're the only people to actually have Serbian support.
00:34:13 · Luke Harries, Joe Heitzeberg
[00:34:13] S0: The other cool thing about this was, the way we trained it, like, actually, lots of... We speak English. We're fluent in English. We're very lucky to be in English because if you're a different language, like a more niche language, lots of transcription just doesn't support you. So any sort of conversational AI, any sorts of transcription, medical transcription, none of those will work. And when we built this model, we were the first one to make transcription actually feasible in massive languages like Gujarati or Malay or Serbian. And it's like suddenly, if you were this, say, medical transcription app, you're like, oh, wow. My TAM's got bigger. I can now, like, launch an entire Serbian sales team. Yeah. And we're the only people to actually have Serbian support. [00:34:58] S1: Yeah. Cool. I'd love you to show us the conversation AI Yes. API stuff. I don't think most people are familiar with that, and voice agent stuff is really blowing up right now. It's super exciting. I think the the people just haven't connected the dots that, like, actually, no. The best text to speech that's best and fast as well, natural, etcetera, is actually... You can compose this into useful apps today. Yeah. With actually no coding. No. But there's also an interface to that. Could you walk us through
00:34:58 · Joe Heitzeberg, Luke Harries
[00:34:58] S1: Yeah. Cool. I'd love you to show us the conversation AI Yes. API stuff. I don't think most people are familiar with that, and voice agent stuff is really blowing up right now. It's super exciting. I think the the people just haven't connected the dots that, like, actually, no. The best text to speech that's best and fast as well, natural, etcetera, is actually... You can compose this into useful apps today. Yeah. With actually no coding. No. But there's also an interface to that. Could you walk us through [00:35:28] S0: the show? It's amazing. And if you're interested for a fun tidbit, when I first started ElevenLabs, Matty laid out this big vision. Yeah. And he's like, yeah. One of the things we're gonna do is conversational AI or voice AI agents. And I was like, never work. The latency [00:35:42] S1: is too high. It's like, [00:35:43] S0: mommy, I think of... Let's just give it a few months. And, yeah, I was completely wrong. [00:35:48] S1: Interesting. Yeah. It makes a huge difference. What is the what is the number of milliseconds of delay that you need to achieve to get a reply back that feels human? [00:35:56] S0: Yeah. What's interesting about this? Well, I think there's like the concrete numbers. It's like, it needs to be really less than a second. Sure. But if you think about what customers actually want, like the job to be done of certain calls, there's certain... There's different types of calls with companion apps. Super important at its low latency, super, like, emotional emergency. And, like, we're doing lots of work on that side.
00:35:56 · Luke Harries, Joe Heitzeberg
[00:35:56] S0: Yeah. What's interesting about this? Well, I think there's like the concrete numbers. It's like, it needs to be really less than a second. Sure. But if you think about what customers actually want, like the job to be done of certain calls, there's certain... There's different types of calls with companion apps. Super important at its low latency, super, like, emotional emergency. And, like, we're doing lots of work on that side. [00:36:23] S1: Cool. And... Which we haven't seen out yet. [00:36:26] S0: No. No. We've got new models in the works. Research timelines are always hard, but, yeah, yeah, there's there's more coming. But, I mean, we're already powering, some pretty big some pretty big companion apps. [00:36:39] S1: So latency, realism, emotion Yes. Are all important for conversational and and interactive... Trust building trust. [00:36:45] S0: Yeah. But the other angle as well is, like, loads of people use it in a business context. So say customer support or sales. And, actually, what you want is normally, like, a quick reply, which is, like, let me look into that for you. Yeah. But then it's really almost about, like, the rag bit. Yes. And, actually, normally, with a customer support call, what you're trying to do is, like, solve the user's problem. Yep. And there, what you care about is, like, overall time to solve their problem more so than that first. You still need Yeah.