Zero-Shot Learning
00:44:05 · Adarsh Hiremath
[00:44:05] S2: Yeah. The way that I think about it is the system of record for specifying Asian behavior, right? You want to be able to answer two questions the best that you can, which is what should the agent do? It's granted the company context. Is it doing it well or not? Which is the evals that identify whether there's a failure, and then the continual learning loop to make sure that that failure never happens again. If we solve those problems really, really well, then, you know, there's there's a reason for for companies to to keep using us. There's also this last point, which is I really, really believe that there are certain models that are really good for specific use cases. Mhmm. And as the spend on inference, a thousand x's, right, picking the right model for the right use case is gonna become more and more important. And Eval's are the answer to that. Right? So a representative Eval set allows you to make those optimizations, even if you're just focused on cutting down your cloud bill because you can't pay it. Right? Yeah. You you need to have a quantitative way to understand which models to use, and that starts with an Eval investment. So to me, it really, comes down to evals. I believe every company will create an eval set that is representative of all the different jobs that their people are doing, all of the different departments, all of the tasks to be automated, and there will be a whole set of good things that come out of that.
00:45:19 · Nancy Wang, Adarsh Hiremath
[00:45:19] S3: And so, to your point, right, as people use these sort of purpose built models more and more, right, they're going to generate more eval sets, which are going to further fine tune these models. I think the enterprises, they're thinking about, Okay, how much context should I actually need to give in order for this to be super purpose built for my environment? So, how do you counter questions from, let's say, your enterprise customers? They're like, Hey, maybe I don't want to annotate those data sets, or I don't want it to index that data store. [00:45:50] S2: Yeah. Well, I mean, it depends on the specific appetite of that company. Right? There are some companies that will happily use a model like Gemini, and have faith in the security of the of that model and the retention policies. And then they say, we wanna stuff everything in there. Right? Because they recognize that if you enable a really smart model with the right context to to do more, that by itself, it's pretty good across the company, might, like, triple the market cap for the company, right, because you're able to do so much with an AI native company. There are other companies that rightfully so have very, very specific privacy, you know, trust security requirements, right? In which case they might opt to use an open source model, either from one of the labs or neo labs, and have everything local. So so they'll still put in that context, but it won't be leaving the the company's premises at all.
00:45:50 · Adarsh Hiremath, Dev Tagare, Nancy Wang
[00:45:50] S2: Yeah. Well, I mean, it depends on the specific appetite of that company. Right? There are some companies that will happily use a model like Gemini, and have faith in the security of the of that model and the retention policies. And then they say, we wanna stuff everything in there. Right? Because they recognize that if you enable a really smart model with the right context to to do more, that by itself, it's pretty good across the company, might, like, triple the market cap for the company, right, because you're able to do so much with an AI native company. There are other companies that rightfully so have very, very specific privacy, you know, trust security requirements, right? In which case they might opt to use an open source model, either from one of the labs or neo labs, and have everything local. So so they'll still put in that context, but it won't be leaving the the company's premises at all. [00:46:41] S1: Do you think in in this space, like, enterprise context actually becomes the moat? As the model gets more intelligent over time, what you really control is the harness and and the context that goes with the harness. That if if Merkur is is effectively building the harness and and curating the context and the agent trajectories with the evals, that then becomes a repeatable sort of offering agnostic of the models that are getting [00:47:08] S3: That becomes deployed potentially your platform offering. [00:47:10] S2: Yes. Right. And how do
00:47:10 · Adarsh Hiremath, Nancy Wang, Dev Tagare
[00:47:10] S2: Yes. Right. And how do [00:47:13] S3: you share that then safely across? Let's say, there might be some context that, let's say, you glean from 1Password that might be applicable across different security or infrastructure security companies. How would you securely share that? [00:47:26] S2: Well, the answer is we don't share it. [00:47:28] S3: Okay. [00:47:28] S2: Right? Are... Not shared. There are silos both at an infrastructure level and an operational level. Infrastructure meaning, you know, there's... Every tenant is a different one, and separated that way, and treated treated as such. And the second thing is from an operational one, of course, you know, trust with with our customers is everything, and you'd not want FTEs working with a specific company to also be working with a competitor. Right? Like, that's just not the the way to do it. So we take both of those components pretty seriously. [00:47:56] S1: You wanna do the builder's mind take? [00:47:58] S3: Yeah. Let's do it. [00:47:58] S1: Let's do it. [00:47:59] S3: Yeah. So maybe, with the builder's mindset, we like to kinda end the, episode on some spicy hot takes. My spicy hot takes is... And we've talked about this at length, is actually, are agents gonna exist in the next twelve to eighteen months? [00:48:13] S2: Agents can exist in twelve to eighteen months. Think, yeah, definitely. Yeah. It sounds like your take is that they're not going to? [00:48:21] S3: Actually, would say yours.
00:48:21 · Nancy Wang, Dev Tagare, Adarsh Hiremath
[00:48:21] S3: Actually, would say yours. [00:48:22] S1: My take is, I think agents will become more of our orchestrators, like leaner and leaner over time, and the action will shift partly towards more more of the harness and more to the models. As the models are becoming more intelligent, what we think of agents today will will be like a dispatcher or an orchestrator effectively. [00:48:46] S2: Okay. But then the answer would be yes, agents will still exist. It's just they'll be doing more orchestration? [00:48:50] S1: I think they'll purely just be doing orchestration. Everything else will look more like traditional software engineering outside of back to the models kinda thing. [00:48:59] S2: Mhmm. But an agent does have to do the software engineering? [00:49:01] S1: A little bit. Like... Yeah. I mean, it's it's becoming... So today, we were beginning with, like, goal driven agents, and then it was like a long iteration flow. Mhmm. So the number of tokens you would need to get to the outcomes you want would be, like, super high, and and the feedback loop was kinda latent. My my thesis is that because you will begin with evals first, and the models can do deep retrievals simultaneously, we'll do it more and more of it. I I think you will go from a largely goal driven system Yeah. To a trajectory driven system governed by, like, this thin orchestration layer. [00:49:36] S3: So what would that actually look like? Just a set of instruction sets?
00:49:36 · Nancy Wang, Dev Tagare, Adarsh Hiremath
[00:49:36] S3: So what would that actually look like? Just a set of instruction sets? [00:49:40] S1: Or A set of skills and tools effectively. Yeah. And then, like, a general arbiter of picking skills and tools and bootstrapping what we think... What what we call agents today effectively. But, like, there's no random goal driven thing anymore. It's all evals driven. Evals inform trajectories. Models are able to keep the task on a trajectory. [00:50:02] S2: I mean, I would say that I largely agree with that, where even if you just take the agent to agent interaction out of the picture, I think one of the things that people told about agents that was kind of incorrect is that it's a unidirectional thing between humans and agents. You give the agent the task, it just does it. It's very much bidirectional. Right? The human gives the agent the task. It needs the human feedback to understand what good looks like, not make mistakes, all the things that we've talked about. And I think the natural progression of that is the bidirectional, you know, collaboration model agents, where an agent will need to talk to another agent, it will need to orchestrate with another agent, it will need to oversee the output of another agent's work. And I think that's certainly something that'll happen in twelve to eighteen months. [00:50:41] S1: I mean, like, it's it's effectively becoming a three way handshake between the the human set of agents and maybe the model as a separate category. And the models are more a constellation of models versus a single model.