AI Tinkerers - "One-Shot"
00:30:17 · Luke Harries, Joe Heitzeberg
[00:30:17] S0: We're chatting. You're like, I need it to draft a reply. You just hold it down, takes the screenshot Yeah. Does the reply for you, paste it back in. [00:30:25] S1: You know those apps... Know those apps that'll do a long screenshot of a scroll all the way? Yes. So imagine that. Now I'm giving the LLM the context, and then I'm the voice of instruction. [00:30:33] S0: Yeah. Yeah. [00:30:33] S1: Hey. Take this page. Make an email to this. Send it to Luke. Ask him a couple questions about the blog post. That'd be very [00:30:39] S0: good. Maybe we should hack that Let's [00:30:41] S1: do it. [00:30:41] S0: Ten minutes after. But honestly, you you could... The only trick... I don't know if you can get that long screenshot in most apps. I only seen some apps actually. [00:30:50] S1: Yeah. Yeah. [00:30:51] S0: But even the stuff on the screen is probably enough. [00:30:53] S1: And probably enough for most cases. You could always zoom out or Yeah. Pinch to zoom. [00:30:56] S0: One one other side point is I worked out that ChatGPT can actually generate Apple Shortcuts. Yeah. And and there's there's an annoying bit whereby you have to sign it. So you have to generate it on your Mac. Mhmm. You then have to run a sign command. Mhmm. But then if you click the link, you then have it instantly loaded Yeah. In your Mac and in your phone. [00:31:17] S1: Yeah. Nice. [00:31:18] S0: And what's cool about Apple Shortcuts as well is that exact workflow I showed you on my phone. [00:31:23] S1: It works on [00:31:23] S0: your Mac. [00:31:24] S1: That's cool.
00:31:24 · Joe Heitzeberg, Luke Harries
[00:31:24] S1: That's cool. [00:31:25] S0: And it's super cheap because it's just raw API usage. [00:31:27] S1: Yeah. Love it. I think... I don't think people are familiar with the speech to text at ElevenLabs. That's rather new. Right? [00:31:34] S0: Yes. Yeah. [00:31:34] S1: Yeah. I have a I have a quick curious question. I admit I haven't tried it yet. I've done long transcriptions on videos and diarization and all the bits. So just... I wonder if you could just quickly go through. So some of the limitations of just using something open source like Whisper Yep. Are actually, like, context length Yes. Or Gemini. Gemini is great, on, input tokens Yeah. Not output. So you can't just say, here's a two hour video. Give me the whole transcript. It'll stop halfway. Yeah. Do you break through that? [00:32:02] S0: Yeah. 100%. And the the interesting thing particularly for something like this with podcast transcripts is small errors compound dramatically Yeah. Because it will shift the output such that, like, it will end up... If it makes a mistake early on Yeah. Often it'll be misaligned then for the entire podcast. Yeah. And you have to, like, go through. Sure. Yeah, we built the world's best speech to text model called Scribe. [00:32:27] S1: Mhmm. [00:32:28] S0: And and it still beats, I think, OpenAI released a new model recently, still beats that. Beats Gemini. [00:32:34] S1: Do you do diarization, which is Yeah. For those who don't know, delineating of the different speakers?
00:32:34 · Joe Heitzeberg, Luke Harries
[00:32:34] S1: Do you do diarization, which is Yeah. For those who don't know, delineating of the different speakers? [00:32:38] S0: Yeah. So it does diarize... Excellent diarization. It also does character level time stamps. Cool. So that you can like Wow. [00:32:46] S1: Character sync and you can do apps on that. [00:32:48] S0: Yeah. You can do character level, word level. So if you're thinking like Joe's podcast, let's do an automated pipeline to make caption style videos, like, it's fantastic for [00:32:58] S1: doing that. Can I give it a bag of terms like custom brand names and technical terms to hint to [00:33:02] S0: the Yep? [00:33:03] S1: The AI? [00:33:03] S0: Yeah. Yeah. Cool. Actually, it does. It gets words like ElevenLabs. Right? One of my colleagues is called Thor. His full name is Thorsten. It gets words like that straight out of Cool. We have to definitely try out with this transcript. [00:33:15] S1: What about in the diarization, do you... Can you use the LLM to... In natively in in the one shot context, figure out the names of the speakers in context? [00:33:24] S0: I think that's done after. I think it just says the speaking name. I need to double [00:33:30] S1: check number and ID. Be good. Hi, Luke. And then you speak, and it knows that you're Luke. [00:33:35] S0: Right? I'm I'm not sure we have that built. Maybe we should. We We most likely should. [00:33:40] S1: It's a convenience thing for developers. [00:33:42] S0: Yeah. Maybe built into the SDKs could be a pretty cool one. [00:33:45] S1: Character level timestamps and it's accurate.
00:33:45 · Joe Heitzeberg, Luke Harries
[00:33:45] S1: Character level timestamps and it's accurate. [00:33:47] S0: That's Yeah. It's and super accurate. That's amazing. The other cool thing is it does audio events. So it does like laugh. Does It [00:33:53] S1: laugh. [00:33:53] S0: It does body laugh. So say you want to, like, alteredly... Automatically extract any coughs, Yeah. You [00:33:59] S1: Oh, that's really cool. Yeah. Oh, wow. To get rid of them. Right? Yeah. So you could do jump cutting essentially at that level. Yeah. I've noticed a lot of the the time stamping and some of the others aren't that accurate, and so your subtitles won't exactly match the words, and therefore, can't do automated jump cuts. [00:34:13] S0: The other cool thing about this was, the way we trained it, like, actually, lots of... We speak English. We're fluent in English. We're very lucky to be in English because if you're a different language, like a more niche language, lots of transcription just doesn't support you. So any sort of conversational AI, any sorts of transcription, medical transcription, none of those will work. And when we built this model, we were the first one to make transcription actually feasible in massive languages like Gujarati or Malay or Serbian. And it's like suddenly, if you were this, say, medical transcription app, you're like, oh, wow. My TAM's got bigger. I can now, like, launch an entire Serbian sales team. Yeah. And we're the only people to actually have Serbian support.
00:34:13 · Luke Harries, Joe Heitzeberg
[00:34:13] S0: The other cool thing about this was, the way we trained it, like, actually, lots of... We speak English. We're fluent in English. We're very lucky to be in English because if you're a different language, like a more niche language, lots of transcription just doesn't support you. So any sort of conversational AI, any sorts of transcription, medical transcription, none of those will work. And when we built this model, we were the first one to make transcription actually feasible in massive languages like Gujarati or Malay or Serbian. And it's like suddenly, if you were this, say, medical transcription app, you're like, oh, wow. My TAM's got bigger. I can now, like, launch an entire Serbian sales team. Yeah. And we're the only people to actually have Serbian support. [00:34:58] S1: Yeah. Cool. I'd love you to show us the conversation AI Yes. API stuff. I don't think most people are familiar with that, and voice agent stuff is really blowing up right now. It's super exciting. I think the the people just haven't connected the dots that, like, actually, no. The best text to speech that's best and fast as well, natural, etcetera, is actually... You can compose this into useful apps today. Yeah. With actually no coding. No. But there's also an interface to that. Could you walk us through
00:34:58 · Joe Heitzeberg, Luke Harries
[00:34:58] S1: Yeah. Cool. I'd love you to show us the conversation AI Yes. API stuff. I don't think most people are familiar with that, and voice agent stuff is really blowing up right now. It's super exciting. I think the the people just haven't connected the dots that, like, actually, no. The best text to speech that's best and fast as well, natural, etcetera, is actually... You can compose this into useful apps today. Yeah. With actually no coding. No. But there's also an interface to that. Could you walk us through [00:35:28] S0: the show? It's amazing. And if you're interested for a fun tidbit, when I first started ElevenLabs, Matty laid out this big vision. Yeah. And he's like, yeah. One of the things we're gonna do is conversational AI or voice AI agents. And I was like, never work. The latency [00:35:42] S1: is too high. It's like, [00:35:43] S0: mommy, I think of... Let's just give it a few months. And, yeah, I was completely wrong. [00:35:48] S1: Interesting. Yeah. It makes a huge difference. What is the what is the number of milliseconds of delay that you need to achieve to get a reply back that feels human? [00:35:56] S0: Yeah. What's interesting about this? Well, I think there's like the concrete numbers. It's like, it needs to be really less than a second. Sure. But if you think about what customers actually want, like the job to be done of certain calls, there's certain... There's different types of calls with companion apps. Super important at its low latency, super, like, emotional emergency. And, like, we're doing lots of work on that side.