Zero-Shot Learning
00:38:28 · Nancy Wang, Dev Tagare
[00:38:28] S3: Yeah. Wasn't that one of the first wave of AI companies? Yeah. Companions. [00:38:32] S1: Oh, yeah. Yeah. Sure. Character AI. Yeah. One of the things I was thinking about was you... Earlier in the pod, you mentioned, like, you wanna bring agents to every surface. Right? Like, where work happens, you want it to be there. And stuff like this, because you're going where work happens is super contextualized. It's very, like, in the moment, you know, last one day, two day, five days, etcetera. All of that context comes about, and then you reconcile that with your APEX benchmarks. Now you're talking about, like, what is the actual productivity gain for a given individual, for a given enterprise, so on and so forth. When you think about this over time, do you think we're moving away from, like, a token based model effectively, then to a value added model now to a outcomes oriented model, like the outcomes economy versus the token economy, if you will?
00:39:26 · Adarsh Hiremath
[00:39:26] S2: Absolutely. Yeah. I think that the reason that people are so fixated on token consumption today is because outcomes are pretty hard to measure. And also, agents aren't able to operate completely autonomously and produce work output without a level of steering, such as how do you attribute the value to the agent versus the person who steered it. In a lot of cases, like, that can be a little bit tricky. But I think that the value at stake model will become more and more of the norm when AI agents fully take on the role of employees at companies. Right? Where you would judge how well a person does based on their work output, and similarly, you will judge how well an agent does by the agent output. And it just feels inevitable that we're we're gonna go down this way. Now the the last thing is the most exciting applications to me are applications that just on net increase productivity as opposed to to cost savings. Mhmm. Right? Because I think of the the the AI wave as a massive lever on the productivity of society. And it's super, super exciting to me what people will be able to achieve, net new things, like more, what abundance will be created as a result of it, opposed to just like incremental cost savings as important as as they are.
00:40:36 · Nancy Wang
[00:40:36] S3: And so, doing work, and maybe bringing this back to your benchmark, right, what kind of stuck out to us when we're reviewing Apex Wee was the fact that most benchmarks really kind of focus on things that are easy to measure, right, that are very straightforward. And when we talk about doing actual work that a human would do, right, there's all these edge cases, all these nuances, right? And also, the fact that, look, we may not make the same decision, right, on the same amount of data just tomorrow, or even yesterday, Right? It's really in that moment. And so, how did you think about balancing simplicity for your benchmark versus reality?
00:41:09 · Adarsh Hiremath, Nancy Wang
[00:41:09] S2: I would say it's it's definitely skewed towards the reality side. Right? If you look at these benchmarks, you can tell that they're simple because they're saturated, actually. And you can tell that they're not realistic because we aren't seeing the adoption across all of these different companies. Right? Like, to me, it's it's one of... It's a trillion dollar problem to solve that these agents are so smart, smarter than me at, like, a a bunch of different things Mhmm. And smarter than a bunch of the smartest people I know, but they're not doing a lot of knowledge work tasks that I know that they're capable of doing, in part because they don't have the enterprise context, enterprise tool connections, and feedback loop Mhmm. For what good looks like. Right? So the real way that this all plays out is these are brilliant employees being dropped into a company with no onboarding. Yep. And solving that problem is a multi trillion dollar problem because it will enable agents to operate at high throughput and high sophistication across a bunch of different use cases and increase the productivity of humanity. Like, think that problem space is a multi trillion dollar problem. [00:42:04] S3: And so, maybe bringing it home a little bit as well, right? So, what result would you be looking for from this benchmark? So to your point, we can get more mass adoption across builders?
00:42:15 · Adarsh Hiremath
[00:42:15] S2: The benchmark as it stands today, the frontier labs are actively looking at the data samples, hill climbing the benchmark. And I... At at a base model level, I'm very, very excited that this benchmark will make the models more capable at doing real engineering tasks. In the context of Apex Suite, the two categories we're excited by are about observability. Right? There's a production failure. Yeah. You have to go through all these different systems to figure it out, and then you wanna actually resolve it. And the second is integration. So not just a a single system, but a set of different systems. So I'm really, really excited that that intelligence is gonna be present in the base models, and hopefully all of us will feel it when we're using the next generation of coding models. Now the second thing is to make it accessible and useful to a broader set of people. There, there's a a little bit of work in taking the base dataset and making contextual to an enterprise's context. Right? Every enterprise is idiosyncratic. If that was not the case, then then those companies would not exist. Right? Because every single company is doing something specially, or at least believes that they are. And a benchmark that reflects the nuances of that enterprise and the way that they work and tools that they use is gonna be really, really important at making coding agents, for example, the best at that, company's SDLC.
00:43:22 · Dev Tagare
[00:43:22] S1: Mhmm. That's that's a great point. So switching gears a little bit, like, Merkur sits at a very unique spot between, the the human network and now, like, let's call it the agents' forms or, like, transitioning into the AI network and and evolving into more of a operating system for agents overall at this point. Breaking this down, how much of of a mode do you think is infrastructure versus the product surfaces versus user experience, the data flywheel, so on and so forth? Usually, it's a combination of things, but relatively speaking, let's say six months out, a year out, two years out, where do you think that categorization is landing?