AI Tinkerers - "One-Shot"
00:26:03 · Luke Harries, Joe Heitzeberg
[00:26:03] S0: Yes. [00:26:03] S1: Was a a person, from a... Tanya, from a person... A company called TinFoil. They do this. [00:26:09] S0: Oh, really? [00:26:10] S1: Secure end and enclaves, cryptographically secured to the hardware. Yeah. [00:26:15] S0: That's very cool. [00:26:16] S1: And with a cofounder from MIT, he's a world expert in that and knows everyone in that niche within PhD computer science, and they're building the company on that. They were a few months away from... Later, I caught up with them. We actually did an episode of One Shot with Tanya because it was so good to hear about this, and that that seems to be the itch that they're scratching. Mean, I exactly what you described. Yeah. And then, we sat down to do the interview, they're like, we just got into YC yesterday. It was a great timing. So pulling them full circle on a lot of these things. Yeah. That's an interesting and a strange thing. [00:26:46] S0: I think either that or, like, could we just get running running on smartphones? And actually, the company I'd love to see. Yeah. [00:26:53] S1: Yeah. Of course. The company [00:26:54] S0: I'd love to see the most just doing this is Apple. Yeah. Because I think Apple has, like, everything set up for them, but we're we're not there just yet. [00:27:04] S1: It's a laugh. I mean, I am frustrated using wonderful tools like Whisper and Super Whisper on my Mac to transcribe, and I'm addicted to that. [00:27:13] S0: Well, that's [00:27:13] S1: I use it for everything. [00:27:14] S0: That maybe takes us nicely into the second hack I'd love to show.
00:27:14 · Luke Harries, Joe Heitzeberg
[00:27:14] S0: That maybe takes us nicely into the second hack I'd love to show. [00:27:17] S1: Well, now we get... ElevenLabs get in there too, [00:27:19] S0: but Yes. [00:27:20] S1: I... And we will, but, I'm so addicted to using transcription in that modality now that I can't use Apple. It's terrible. No. Apple cannot spell the word hackathon. [00:27:29] S0: Yes. [00:27:29] S1: Apple will take the word there and use the wrong version of it in a sentence. I know. It just... An LLM can't even make that mistake if you try to get [00:27:39] S0: it more time, like, tapping around in the message [00:27:41] S1: to try and read the script. Yeah. It's terrible. Now let's talk about then... Let's go to the next topic. Yeah. We're talking about voice
00:27:41 · Joe Heitzeberg, Luke Harries
[00:27:41] S1: to try and read the script. Yeah. It's terrible. Now let's talk about then... Let's go to the next topic. Yeah. We're talking about voice [00:27:48] S0: now. So I was literally ran into the exact issue we're talking about, and I haven't actually demoed this to Joe just yet. So ElevenLabs, we have... We were initially famous for our text to speech models where we were the first company to cross this threshold of human like speech. Mhmm. We more recently shipped the best speech to text model, which is the harsh quality translation. Mhmm. And so what I... I was actually having lunch with my wife, and we're waiting for the bill, and I was like, how is iPhone dictation so bad? Mhmm. And I was like, surely there's a way where I can just speak into it. I can transcribe it with ElevenLabs. Mhmm. I can then just like clean up the raw transcript Mhmm. And then paste it back in. Mhmm. And so what I thought was interesting, I was like, oh, okay. How could I do that? Do I need to build an app? [00:28:39] S1: Do I [00:28:39] S0: have it? I was like, well, Apple Shortcuts is the thing. Oh, yeah. How far can I push Apple Shortcuts? Yeah. So the way I've set it up is all you do is you hold down the action button [00:28:47] S1: Wow. You assign that to be a shortcut. Yeah. Okay. Yeah. [00:28:50] S0: Cool. So you hold down the action button. Hey, Joe. Really excited to do our podcast called two shot... Sorry, one shot, this coming week. Can't wait to see you on Thursday. Friday. Can't wait to see you on Friday. Best wishes, wishes, Luke. [00:29:08] S1: 10AM, not 9AM.
00:29:08 · Joe Heitzeberg, Luke Harries
[00:29:08] S1: 10AM, not 9AM. [00:29:10] S0: And the way it works, you can see in the top right here, it's now processing that shortcut. Cool. It then sends it to ElevenLabs, sends it to to OpenAI to clean up the transcript, and then you can just paste it in. And here we've got, hey, Joe. Really excited to to do our podcast called OneShot this coming week. Can't wait to see you on Friday. Best wishes, Luke. And it's all cleaned up. Great communication. You can call and send. [00:29:35] S1: Well, I'm gonna definitely get that set up. [00:29:37] S0: Yes. Because [00:29:37] S1: Siri is driving me crazy. [00:29:39] S0: The one... I mean, this is maybe a brief tangent, but, like, Apple Shortcuts is so powerful. Yeah. And the the latest hack I've been trying to do is like, I've realized, well, one is like, if you can just hold down the button, do a voice note and then execute actions. Wow. That's powerful for so many follow on Yep. Tasks. The other really cool one though is the vision models have got so good. Like Mhmm. Why not hold it down? That triggers a shortcut, which takes a screenshot Mhmm. Sends that to an LLM to work out the next step. Mhmm. So you basically have this, like, full context, super smart AI assistant. [00:30:16] S1: That's a good idea. [00:30:16] S0: So you could do, like [00:30:17] S1: Good idea. [00:30:17] S0: We're chatting. You're like, I need it to draft a reply. You just hold it down, takes the screenshot Yeah. Does the reply for you, paste it back in.
00:30:17 · Luke Harries, Joe Heitzeberg
[00:30:17] S0: We're chatting. You're like, I need it to draft a reply. You just hold it down, takes the screenshot Yeah. Does the reply for you, paste it back in. [00:30:25] S1: You know those apps... Know those apps that'll do a long screenshot of a scroll all the way? Yes. So imagine that. Now I'm giving the LLM the context, and then I'm the voice of instruction. [00:30:33] S0: Yeah. Yeah. [00:30:33] S1: Hey. Take this page. Make an email to this. Send it to Luke. Ask him a couple questions about the blog post. That'd be very [00:30:39] S0: good. Maybe we should hack that Let's [00:30:41] S1: do it. [00:30:41] S0: Ten minutes after. But honestly, you you could... The only trick... I don't know if you can get that long screenshot in most apps. I only seen some apps actually. [00:30:50] S1: Yeah. Yeah. [00:30:51] S0: But even the stuff on the screen is probably enough. [00:30:53] S1: And probably enough for most cases. You could always zoom out or Yeah. Pinch to zoom. [00:30:56] S0: One one other side point is I worked out that ChatGPT can actually generate Apple Shortcuts. Yeah. And and there's there's an annoying bit whereby you have to sign it. So you have to generate it on your Mac. Mhmm. You then have to run a sign command. Mhmm. But then if you click the link, you then have it instantly loaded Yeah. In your Mac and in your phone. [00:31:17] S1: Yeah. Nice. [00:31:18] S0: And what's cool about Apple Shortcuts as well is that exact workflow I showed you on my phone. [00:31:23] S1: It works on [00:31:23] S0: your Mac. [00:31:24] S1: That's cool.
00:31:24 · Joe Heitzeberg, Luke Harries
[00:31:24] S1: That's cool. [00:31:25] S0: And it's super cheap because it's just raw API usage. [00:31:27] S1: Yeah. Love it. I think... I don't think people are familiar with the speech to text at ElevenLabs. That's rather new. Right? [00:31:34] S0: Yes. Yeah. [00:31:34] S1: Yeah. I have a I have a quick curious question. I admit I haven't tried it yet. I've done long transcriptions on videos and diarization and all the bits. So just... I wonder if you could just quickly go through. So some of the limitations of just using something open source like Whisper Yep. Are actually, like, context length Yes. Or Gemini. Gemini is great, on, input tokens Yeah. Not output. So you can't just say, here's a two hour video. Give me the whole transcript. It'll stop halfway. Yeah. Do you break through that? [00:32:02] S0: Yeah. 100%. And the the interesting thing particularly for something like this with podcast transcripts is small errors compound dramatically Yeah. Because it will shift the output such that, like, it will end up... If it makes a mistake early on Yeah. Often it'll be misaligned then for the entire podcast. Yeah. And you have to, like, go through. Sure. Yeah, we built the world's best speech to text model called Scribe. [00:32:27] S1: Mhmm. [00:32:28] S0: And and it still beats, I think, OpenAI released a new model recently, still beats that. Beats Gemini. [00:32:34] S1: Do you do diarization, which is Yeah. For those who don't know, delineating of the different speakers?