AI Tinkerers - "One-Shot"
00:34:13 · Luke Harries, Joe Heitzeberg
[00:34:13] S0: The other cool thing about this was, the way we trained it, like, actually, lots of... We speak English. We're fluent in English. We're very lucky to be in English because if you're a different language, like a more niche language, lots of transcription just doesn't support you. So any sort of conversational AI, any sorts of transcription, medical transcription, none of those will work. And when we built this model, we were the first one to make transcription actually feasible in massive languages like Gujarati or Malay or Serbian. And it's like suddenly, if you were this, say, medical transcription app, you're like, oh, wow. My TAM's got bigger. I can now, like, launch an entire Serbian sales team. Yeah. And we're the only people to actually have Serbian support. [00:34:58] S1: Yeah. Cool. I'd love you to show us the conversation AI Yes. API stuff. I don't think most people are familiar with that, and voice agent stuff is really blowing up right now. It's super exciting. I think the the people just haven't connected the dots that, like, actually, no. The best text to speech that's best and fast as well, natural, etcetera, is actually... You can compose this into useful apps today. Yeah. With actually no coding. No. But there's also an interface to that. Could you walk us through
00:34:58 · Joe Heitzeberg, Luke Harries
[00:34:58] S1: Yeah. Cool. I'd love you to show us the conversation AI Yes. API stuff. I don't think most people are familiar with that, and voice agent stuff is really blowing up right now. It's super exciting. I think the the people just haven't connected the dots that, like, actually, no. The best text to speech that's best and fast as well, natural, etcetera, is actually... You can compose this into useful apps today. Yeah. With actually no coding. No. But there's also an interface to that. Could you walk us through [00:35:28] S0: the show? It's amazing. And if you're interested for a fun tidbit, when I first started ElevenLabs, Matty laid out this big vision. Yeah. And he's like, yeah. One of the things we're gonna do is conversational AI or voice AI agents. And I was like, never work. The latency [00:35:42] S1: is too high. It's like, [00:35:43] S0: mommy, I think of... Let's just give it a few months. And, yeah, I was completely wrong. [00:35:48] S1: Interesting. Yeah. It makes a huge difference. What is the what is the number of milliseconds of delay that you need to achieve to get a reply back that feels human? [00:35:56] S0: Yeah. What's interesting about this? Well, I think there's like the concrete numbers. It's like, it needs to be really less than a second. Sure. But if you think about what customers actually want, like the job to be done of certain calls, there's certain... There's different types of calls with companion apps. Super important at its low latency, super, like, emotional emergency. And, like, we're doing lots of work on that side.
00:35:56 · Luke Harries, Joe Heitzeberg
[00:35:56] S0: Yeah. What's interesting about this? Well, I think there's like the concrete numbers. It's like, it needs to be really less than a second. Sure. But if you think about what customers actually want, like the job to be done of certain calls, there's certain... There's different types of calls with companion apps. Super important at its low latency, super, like, emotional emergency. And, like, we're doing lots of work on that side. [00:36:23] S1: Cool. And... Which we haven't seen out yet. [00:36:26] S0: No. No. We've got new models in the works. Research timelines are always hard, but, yeah, yeah, there's there's more coming. But, I mean, we're already powering, some pretty big some pretty big companion apps. [00:36:39] S1: So latency, realism, emotion Yes. Are all important for conversational and and interactive... Trust building trust. [00:36:45] S0: Yeah. But the other angle as well is, like, loads of people use it in a business context. So say customer support or sales. And, actually, what you want is normally, like, a quick reply, which is, like, let me look into that for you. Yeah. But then it's really almost about, like, the rag bit. Yes. And, actually, normally, with a customer support call, what you're trying to do is, like, solve the user's problem. Yep. And there, what you care about is, like, overall time to solve their problem more so than that first. You still need Yeah.
00:36:45 · Luke Harries, Joe Heitzeberg
[00:36:45] S0: Yeah. But the other angle as well is, like, loads of people use it in a business context. So say customer support or sales. And, actually, what you want is normally, like, a quick reply, which is, like, let me look into that for you. Yeah. But then it's really almost about, like, the rag bit. Yes. And, actually, normally, with a customer support call, what you're trying to do is, like, solve the user's problem. Yep. And there, what you care about is, like, overall time to solve their problem more so than that first. You still need Yeah. [00:37:13] S1: If I'm gonna do a really good quality answer Yes. I don't expect a human to have the answer instantaneously. Go off and do a little thinking. We look some stuff up, you know, and then come back in a multi agent rag retrieval. Yeah. We'll do, you know, obviously, multiplex and a judgment and a report gen and then Yeah. Summarization, and that's the part that's spoken. And maybe ask a question, then give me the rest interactively. It's almost like there has to be a few threads going. And one is, like, the I'm in the moment with you, conversing with you, has to be attentive, quick, friendly, you know, while also background processing Yes. The retrieval. So are you... Do you do you... First of the system that you've built, does it expose that to me? Oh, okay. Yeah. I can system prompt what this is first level interaction will be like, and I can go kick off some background stuff, come back and push it into the complex.
00:38:02 · Luke Harries, Joe Heitzeberg
[00:38:02] S0: Have a fully integrated rag pipeline Okay. Where you can actually, like, chuck in, you know, huge number of documents and then do all the rag stuff for you. But kind of zooming out what... Why we built this is we end up... We work with, like, lots of small customers as well as some of the biggest enterprises in the world, like Perplexity or Time Magazine. Yeah. And we literally had about 20 customers over the space of two months saying, great. We're taking your text to speech and we're taking speech to text. Sure. We're merging those together with term making model. Like, have you got any tips for latency and blood? And about the twentieth time of helping people implement this, we were like, okay. Let's like [00:38:39] S1: Put it together. [00:38:40] S0: Put together. And the the bit we're trying to do is we we have like two modes you can use it. Yeah. One which is like hardcore engineer mode Yeah. Which is, like Cool. I just want ElevenLabs to do all the voice stuff. [00:38:52] S1: Mhmm. [00:38:53] S0: So we do voice in turn taking, but you're the brain. And it's literally just, like, web hooks or web sockets [00:39:01] S1: Mhmm.
00:39:01 · Joe Heitzeberg, Luke Harries
[00:39:01] S1: Mhmm. [00:39:01] S0: To handle the, like, I mean, it abstracts it all to text in text out. Yep. So let's say you're a big enterprise, you spend... You have a thousand per cent engineering team building out all your text based solutions, you can very quickly wrap this and suddenly you have a voice system. And then the other bit is we have the much more feature complete version where you can do proper rag and we have evals. But, yeah, I can show you... Talk you through some of that. [00:39:24] S1: Yeah. Almost think of that other case, the simpler one is like something that SMB might use. [00:39:29] S0: Yes. [00:39:29] S1: Like, I've... Yeah. We've got a WordPress blog and [00:39:31] S0: Yes. [00:39:32] S1: A knowledge base and and like your tools makes it super easy just to kinda, like, load up a rag Yes. Personality define an agent and then actually ship it with JavaScript on the website. Yep. Right? All of it. Yeah. Yeah. Pretty cool. I built I built a couple last night. [00:39:45] S0: Oh, nice. Yeah. What did you build? [00:39:47] S1: Well, I built an AI Tinkerers frequently asked questions. You know? And it was... I have to admit it. There's a little bit of tuning Yes. Needed, you know, in the system prompt and that I didn't quite... Because I I I put out our frequently asked questions. I don't know if you've read. It's very specific. It's very opinionated. [00:40:01] S0: Yes. [00:40:02] S1: You know? And there's a question [00:40:03] S0: That's all the best community