AI Tinkerers - "One-Shot"
00:36:45 · Luke Harries, Joe Heitzeberg
[00:36:45] S0: Yeah. But the other angle as well is, like, loads of people use it in a business context. So say customer support or sales. And, actually, what you want is normally, like, a quick reply, which is, like, let me look into that for you. Yeah. But then it's really almost about, like, the rag bit. Yes. And, actually, normally, with a customer support call, what you're trying to do is, like, solve the user's problem. Yep. And there, what you care about is, like, overall time to solve their problem more so than that first. You still need Yeah. [00:37:13] S1: If I'm gonna do a really good quality answer Yes. I don't expect a human to have the answer instantaneously. Go off and do a little thinking. We look some stuff up, you know, and then come back in a multi agent rag retrieval. Yeah. We'll do, you know, obviously, multiplex and a judgment and a report gen and then Yeah. Summarization, and that's the part that's spoken. And maybe ask a question, then give me the rest interactively. It's almost like there has to be a few threads going. And one is, like, the I'm in the moment with you, conversing with you, has to be attentive, quick, friendly, you know, while also background processing Yes. The retrieval. So are you... Do you do you... First of the system that you've built, does it expose that to me? Oh, okay. Yeah. I can system prompt what this is first level interaction will be like, and I can go kick off some background stuff, come back and push it into the complex.
00:38:02 · Luke Harries, Joe Heitzeberg
[00:38:02] S0: Have a fully integrated rag pipeline Okay. Where you can actually, like, chuck in, you know, huge number of documents and then do all the rag stuff for you. But kind of zooming out what... Why we built this is we end up... We work with, like, lots of small customers as well as some of the biggest enterprises in the world, like Perplexity or Time Magazine. Yeah. And we literally had about 20 customers over the space of two months saying, great. We're taking your text to speech and we're taking speech to text. Sure. We're merging those together with term making model. Like, have you got any tips for latency and blood? And about the twentieth time of helping people implement this, we were like, okay. Let's like [00:38:39] S1: Put it together. [00:38:40] S0: Put together. And the the bit we're trying to do is we we have like two modes you can use it. Yeah. One which is like hardcore engineer mode Yeah. Which is, like Cool. I just want ElevenLabs to do all the voice stuff. [00:38:52] S1: Mhmm. [00:38:53] S0: So we do voice in turn taking, but you're the brain. And it's literally just, like, web hooks or web sockets [00:39:01] S1: Mhmm.
00:39:01 · Joe Heitzeberg, Luke Harries
[00:39:01] S1: Mhmm. [00:39:01] S0: To handle the, like, I mean, it abstracts it all to text in text out. Yep. So let's say you're a big enterprise, you spend... You have a thousand per cent engineering team building out all your text based solutions, you can very quickly wrap this and suddenly you have a voice system. And then the other bit is we have the much more feature complete version where you can do proper rag and we have evals. But, yeah, I can show you... Talk you through some of that. [00:39:24] S1: Yeah. Almost think of that other case, the simpler one is like something that SMB might use. [00:39:29] S0: Yes. [00:39:29] S1: Like, I've... Yeah. We've got a WordPress blog and [00:39:31] S0: Yes. [00:39:32] S1: A knowledge base and and like your tools makes it super easy just to kinda, like, load up a rag Yes. Personality define an agent and then actually ship it with JavaScript on the website. Yep. Right? All of it. Yeah. Yeah. Pretty cool. I built I built a couple last night. [00:39:45] S0: Oh, nice. Yeah. What did you build? [00:39:47] S1: Well, I built an AI Tinkerers frequently asked questions. You know? And it was... I have to admit it. There's a little bit of tuning Yes. Needed, you know, in the system prompt and that I didn't quite... Because I I I put out our frequently asked questions. I don't know if you've read. It's very specific. It's very opinionated. [00:40:01] S0: Yes. [00:40:02] S1: You know? And there's a question [00:40:03] S0: That's all the best community
00:40:03 · Luke Harries, Joe Heitzeberg
[00:40:03] S0: That's all the best community [00:40:03] S1: to Questions like, you know, literally, like, you know, I'm a Harvard MBA. I sold my company for 500,000,000. I'm not technical, but I've got a groundbreaking idea, and I would be a real asset to the community. I'm super well connected. I just wanna go, and you should make an exception for me. And it's like, no. [00:40:20] S0: Yes. And then... [00:40:21] S1: So I built an agent with that whole repo of, like, questions, and and I asked that question or something similar. And it was like, well, maybe you might be of value to us. Let's see. Why don't you... Why don't we try once? And I was like, no. You know? And so it made me feel like there's some rag tuning [00:40:37] S0: Yes. [00:40:38] S1: That either I have to do or or And you have [00:40:40] S0: full access to the system prompt to do that. Yeah. That's... Did you... By the way, did you integrate client tools as well? [00:40:47] S1: I I looked at the webhook for that and I I thought about... And I think if I did that, I'd get exactly what I want. [00:40:53] S0: Well, there's... Yeah. Well, actually, I was thinking... So I think you should be able to solve your problem just through the system prompts because it's a... You can select any model. You could do Anthropic, you can do Gemini, you can do OpenAI.
00:40:53 · Luke Harries, Joe Heitzeberg
[00:40:53] S0: Well, there's... Yeah. Well, actually, I was thinking... So I think you should be able to solve your problem just through the system prompts because it's a... You can select any model. You could do Anthropic, you can do Gemini, you can do OpenAI. [00:41:05] S1: You know what I did with the system prompts? So rather than actually solve the problem I described is I was like, okay. Now I want you to spell every word in in a Boston accent. I use I use at ElevenLabs, like, gruff voice, and I tried to make it sound. [00:41:18] S0: Voices remain. [00:41:19] S1: Just as fun. [00:41:19] S0: Finding a voice. Well, Joe, you should cloak your I don't know. Genuinely. And use that for... We can actually demo. I've got my voice in this panel. [00:41:26] S1: Well, you know, I have a lot of emails for the organizers that are that are me. Yes. Aren't me. [00:41:31] S0: Yeah. I I I am sure we're getting to it. But the... To show you the, it's a pretty cool demo. I'll just show you if you want, which is, and then we can jump into how you build these after. Yeah. Is there's actually two modes of tools. We have, like, server side tools [00:41:46] S1: Yeah. [00:41:47] S0: And... But we also have client side tools. [00:41:49] S1: Mhmm.
00:41:49 · Joe Heitzeberg, Luke Harries
[00:41:49] S1: Mhmm. [00:41:49] S0: And so this can run, execute JavaScript on your behalf Mhmm. Or like cool. Well, it just returns which tool choose and the args. Yeah. But like, you can have it do stuff like switch the page. Mhmm. So you can be like, oh, I'd love to go to a meetup in Warsaw, and they will like take you immediately to the page. And then it goes great, like give me a quick summary of yourself. Yeah. And then they could like pre fill the entire thing for you. Yeah. Which could be pretty cool. But just to show people that one. Yeah. Hey there. I'm Alexis from ElevenLabs support. How can I help you today? Yeah. Hey, Alexis. Can you tell me more about conversational AI? Sure thing. ElevenLabs Conversational AI is a platform that allows you to deploy customized conversational voice agents. It combines speech to text language models Alexa. Text to speech, and Can you tell me a bit more? Can you show me the the client side tools? Okay. I can tell you more about client side tools. Client tools enable your assistant execute. Perfect. I'm now now actually take me to the page. Or sending... Oh, take me to the page. Do you wanna be taken to the documentation page about client side tools? [00:43:03] S1: Yeah. Is this part baked in with the JavaScript widget, the the navigation aspect? Or do I have to build that?