This Week in Startups
00:00:00 · Jonathan Siddarth, Alex Wilhelm, Lon Harris
[00:00:00] S0: It's really important to benchmark LLMs to figure out where they're good at, where are gaps in the models so that you can generate data that could help the LLMs get better at those specific tasks. One big challenge, Alex, that I see today in the overall evaluation and benchmarking market is that a lot of the evaluations are somewhat academic and somewhat synthetic and don't connect to real world applications or real world use. Ideally, you want AGI to progress in a way where the models get better at useful tasks. [00:00:37] S1: This Week in Startups is brought to you by Northwest Registered Agent. Starting your business should be simple. With Northwest Registered Agent, you can form your entire business identity in just 10 clicks and ten minutes. From LLCs to trademarks, domains to custom websites, they've got you covered. Get more privacy, more options, and more done. Visit Northwest Registered Agent dot com slash Twist today. Dot tech. Say it without saying it. Head to get.tech/twist or your favorite registrar to get a clean, sharp .tech domain today. And AWS Activate. AWS Activate helps startups bring their ideas to life. As you build and scale your business, Activate credits grow with you to support your changing needs. Apply to AWS Activate today and receive up to $100,000 in credits. Visit aws.amazon.com/startups/credits. [00:01:42] S2: Hey. Welcome to this week at startups. You may notice I am not Jason Kalakanis. He is on the road this week. We're gonna do a quick show without him. I am your host, Lon Harris. Joining me as always, Alex Wilhelm.
00:01:42 · Lon Harris, Alex Wilhelm
[00:01:42] S2: Hey. Welcome to this week at startups. You may notice I am not Jason Kalakanis. He is on the road this week. We're gonna do a quick show without him. I am your host, Lon Harris. Joining me as always, Alex Wilhelm. [00:01:53] S1: Hello. Hello. [00:01:54] S2: We have three incredible, amazing Twist 500 interviews coming up right now that Alex did all week. Don't move from that seat. But first, I do think we've gotta talk about GPT five a little bit, Alex.
00:01:54 · Lon Harris, Alex Wilhelm, Jonathan Siddarth
[00:01:54] S2: We have three incredible, amazing Twist 500 interviews coming up right now that Alex did all week. Don't move from that seat. But first, I do think we've gotta talk about GPT five a little bit, Alex. [00:02:05] S1: Yes. Yes. GPT five. So this has gotten a lot of play, but Sam Altman, one of the founders of OpenAI and kind of what you might call the face of AI from the software side, Jensen being the hardware face, if you will, went on a podcast with a man named Theo Vaughn, and they were having a a pretty candid conversation about what they're building. And Sam says, you know, I was playing with GPT five, and I gave it a challenge, and it did it very quickly. And he talks about how it set him back and, like, made him think about what they're building. And the quote that really went viral was he was like, you know, there are moments in time, in science, when people sit back and go, what have we done? And, you know, on one hand, Lon, both... I have three things here. First of all, Sam Altman loves to hype things up and then get mad at everyone when they get a little overhyped. He's a master at PR. So for part one. Two, if you go back in time to GPT-three, even maybe even GPT-two, there was a lot of worries that as this technology advanced and progressed and got better, it would be misused. And we're seeing that now when people are using voice cloning to trick elders out of [00:03:07] S0: their money, and AI is being used in more and more nefarious circumstances.
00:03:07 · Jonathan Siddarth, Alex Wilhelm
[00:03:07] S0: their money, and AI is being used in more and more nefarious circumstances. [00:03:12] S1: But we also, at the time, had a lot of people thinking that there was gonna be a lot of improvement and things were gonna get much better more quickly. So when I when I think about his GPT five comments, they don't actually feel that different to me than what we've seen with GPT four, GPT three, GPT two. So that... That's... To me, this probably does feel that impressive to him as he felt blown away before. So to me, I don't think he's being cynical. I don't think he's overhyping here. I think we're just seeing improvement, and I don't think we know exactly where it's all going.
00:03:12 · Alex Wilhelm, Lon Harris
[00:03:12] S1: But we also, at the time, had a lot of people thinking that there was gonna be a lot of improvement and things were gonna get much better more quickly. So when I when I think about his GPT five comments, they don't actually feel that different to me than what we've seen with GPT four, GPT three, GPT two. So that... That's... To me, this probably does feel that impressive to him as he felt blown away before. So to me, I don't think he's being cynical. I don't think he's overhyping here. I think we're just seeing improvement, and I don't think we know exactly where it's all going. [00:03:43] S2: I mean, I I think that sounds reasonable enough. And and look. I... I'm sure he's playing around with GPT five and is genuinely impressed at what it can do. Like, I think there's two levels here. Like, even... Like, I was very cynical about AI early on. Even I have had experiences like that where I'm impressed by what it can do, and it sort of raises the bar on what I thought was possible. I'm not discounting that experience at all. I'm sure he does have that experience. But I also think this is a weird and unfortunate narrative that we've sort of fallen in. Like... And Jason has talked about this many times that, like, real science and real technology kind of chases sci fi. And I feel like sci fi introduced this idea that AI is scary and that AI is gonna be thinking for itself and all of those things, like, from Skynet in Terminator to Scarlett Johansson and her evolving beyond Joaquin Phoenix and wanting to go off on her own. Like, I think that's
00:03:43 · Lon Harris, Alex Wilhelm
[00:03:43] S2: I mean, I I think that sounds reasonable enough. And and look. I... I'm sure he's playing around with GPT five and is genuinely impressed at what it can do. Like, I think there's two levels here. Like, even... Like, I was very cynical about AI early on. Even I have had experiences like that where I'm impressed by what it can do, and it sort of raises the bar on what I thought was possible. I'm not discounting that experience at all. I'm sure he does have that experience. But I also think this is a weird and unfortunate narrative that we've sort of fallen in. Like... And Jason has talked about this many times that, like, real science and real technology kind of chases sci fi. And I feel like sci fi introduced this idea that AI is scary and that AI is gonna be thinking for itself and all of those things, like, from Skynet in Terminator to Scarlett Johansson and her evolving beyond Joaquin Phoenix and wanting to go off on her own. Like, I think that's [00:04:38] S1: always Spoilers. [00:04:39] S2: So spoilers for her. Sorry, everybody. I feel like that's always the fictional narrative around AI, and I do feel like that has an impact on real world AI CEOs, and they're trying to be part of that narrative and play into that narrative because it's a helpful narrative for them. We're creating iconic, legendary, history shaping technology that's akin to the atomic bomb or the discovery of electricity. And I mean, like, it plays into a friendly narrative if you're the CEO of an AI company.