00:15:41 · Sebastian Johnson
[00:15:41] S0: What do you think about the LLMs at the moment in general? We're seeing a plateauing of the improvements across all models, we're seeing increased level of competitiveness, seeing like Gemini is now often talked about as the better model compared to OpenAI, it's managed to catch up. What's your view? Do you do you see a world in which the models become commoditized?
00:15:59 · Eric Landau, Sebastian Johnson
[00:15:59] S1: Yeah. So I I think I have a different frame on it. One is, like, where does the scaling law actually, like, manifest? Mhmm. And one, kind of metric that I I track and a bunch of other people track is, from the METR Institute in terms of the average length of task that an AI system can, successfully complete. And if you look ten years ago, the average length of task would be something like recognizing a cup, which takes a human, like, a a millisecond to do. So an AI could do something that takes a human a millisecond to do. And then a couple years later, it might be to, write an email, without actually, having the the context lost in the email, and that might take, like, five minutes. Now we're at the place where an AI system can successfully successfully do something like... That takes a human two hours, and this number doubles every seven months. So now these AI systems are doing things that humans take two hours. In seven months, it'll be four hours. Once it hits to the level that it's actually two weeks of time, that's the typical kind of cadence cycle of what we do in our company, like, two week sprints. So that's where you'll have an AI, like, as a as a coworker, as a colleague, as an employee, and that will look like a very different world. And that... This, exponential, this doubling every two months, that hasn't slowed down at all. So while it's easy... Like, you know, in certain metrics, there's saturation. It's hard to see, like, the the deltas of improvement. [00:17:23] S0: Yeah.
00:17:23 · Sebastian Johnson, Eric Landau
[00:17:23] S0: Yeah. [00:17:24] S1: There are places where the AIs are just kind of continuously improving at the rate or faster than the rate than than we expected. So while, yeah, there's gonna be a lot of competition, the progress of the models is not slowing down at all. And I think that's, like, a misconception that some some people have. [00:17:38] S0: Do you have a peg for who might come out on top, or are you able to look... Looking at those metrics? Are you able to see differences in the providers?
00:17:45 · Eric Landau, Sebastian Johnson
[00:17:45] S1: I think there can be multiple winners because there's there's not just one end metric that matters. Mhmm. And, I think, like, one also misconception that people have is that the intelligence of the model is the only thing that matters. And the the analogy that I give is that, you actually don't need very high intelligence for a lot of use cases. You don't want Einstein to be bagging your groceries. Like, I don't I don't need, like, a PhD level physicist to be the one that just kind of recognizes the carrot from the milk and puts them in the in the right place in the bag. Right? I need someone that... I need, like, a a model that's, like, sufficiently good. Yeah. So you can have model providers that actually do some set of tasks, like, really, really well. And and now Anthropic is doing really well in, like, focusing on enterprise and coding, and they're really winning in that space. And OpenAI is focusing on consumer use cases and the regular chatbot and, like, being more of a personal assistant. There is a world where... Like, there's more than one winner. Mhmm. At the end of the day, if I was to be a, like, purely betting man, I would say that probably the company that's attached to the biggest cash cash machine will win, which is, Google. I mean, they're, you know, they have a huge ATM in their pocket that's constantly being fed, to to data centers. They can continuously create models, but I think it won't just come down to a singular, player at the end of the day. [00:19:03] S0: Okay. Interesting. Multimodels but focused in different niches.
00:19:03 · Sebastian Johnson, Eric Landau
[00:19:03] S0: Okay. Interesting. Multimodels but focused in different niches. [00:19:07] S1: Different niches. Yeah. Models are... They have personality. Right? They're like people. Right? And they... They're gonna be good at things, and we're gonna have different, preferential, kind of treatment and, interactions with with different types of models. So there's no reason why... Yeah. You you know, you'll want a model that's, like, super friendly and, like, this is your friend model, and this is your coworker model Yeah. And this is, like, your, you know, how... Your servant model or whatever. Those can all exist, like, in in tandem with each other. They don't have to be one, one, end model. [00:19:41] S0: How how do you think about leveraging AI in your business internally? We've seen things like, OpenClaw. You know, these are major regenerative tools be... Know, that felt like another ChatGPT moment when had this new technology come out. We're seeing a wide range of tools across marketing, go to market. How are you getting up to speed with the tools and implementing them across your own business?
00:20:00 · Eric Landau
[00:20:00] S1: Yeah. It's it's a good question, and I don't know if I have, like, a great answer. We've done things like integrate, Claude into all of our different, kind of knowledge based tools into Slack and and Notion and, get people the right answer very quickly just by tagging it, and so it can pull all the, you know, the the relevant data. We did a, like, a a cursor slash, Claude Code hackathon in our... So, like, who can build the coolest thing without actually touching a line of code, within the the engineering team. And so we have, like, these different things that we're trying to enable the the the tools, but I don't think anyone has really settled on, you know, what the equilibrium state of using AI within your, like, your company context because these things are changing rapidly as well. And, I asked, like, a bunch of... Like, my co... My cofounder and I, like, are asking different companies, like, how are you using AI? And no one really has something which is, like, a very killer, hey. This completely changed everything. It's more like, yeah. We've, we use Claude here, we're doing more coding here. I think that will just continue to, like, evolve on a, like, a more piecemeal basis. Yeah. And that... That's been