Zero-Shot Learning
00:32:33 · Dev Tagare, Adarsh Hiremath
[00:32:33] S1: So one of the the themes that comes out of the paper is, like, strong performance seems to come from, you know, epitomistic discipline, knowing that you've verified something versus just producing it and then evolving it over time and continuing to verify it over time. So how does this discipline look like in the world of agents purely, where now you have swarms, they're solving for sort of like competing goals like humans would in a big company. How does that that benchmark evolve to to address essentially an ecosystem problem versus a specificity problem at the level of an individual engineer or so? [00:33:12] S2: I would say that this is something that's very top of mind for us, and in fact, we're actively working on a project that is super relevant to this. The short answer is the unexplored space is agents as coworkers. Right? And agents actually talking to each other. Right? We have these benchmarks that evaluate how well can an agent do a task when it's given a task. But do we have benchmarks that evaluate how well agents can work with one another? The answer is no, and it's very top of mind from recording. [00:33:37] S1: And similarly, like, taking taking the the same concept a little bit further, let's say external environmental factors change, that creates a level of uncertainty. Humans react to it in a certain way. Nancy may react to it differently, you may react to it differently, I may react to it differently. Agents are more deterministic in that sense, like, you know, there are instructions associated with it.
00:33:37 · Dev Tagare, Adarsh Hiremath
[00:33:37] S1: And similarly, like, taking taking the the same concept a little bit further, let's say external environmental factors change, that creates a level of uncertainty. Humans react to it in a certain way. Nancy may react to it differently, you may react to it differently, I may react to it differently. Agents are more deterministic in that sense, like, you know, there are instructions associated with it. [00:34:02] S2: They have less feelings. [00:34:03] S1: Less feelings, no feelings, you could say that. Right? More predictable. More predictable. For sure. More predictable for sure. So when uncertainty hits you or, like, environmental factors change, how are you thinking about evolving this ecosystem to account for it, and how would you think about measuring how the agent did versus how the human did? [00:34:23] S2: I mean, I know I sound like a broken record. The answer is just the eval. Right? Like, you... In the same way that you can scaffold the task trajectory and output, you can do the same thing with a human. Right? You can get as granular as literally recording what they're doing. Right? And the audio and the video and the screenshots and all the tools. Or you can use documents or you could do AI moderated interviews, but there are ways to create a similar infrastructure for humans and then directly compare with with agents via an eval.
00:34:53 · Dev Tagare, Nancy Wang
[00:34:53] S1: Makes sense. And and, like, the the feedback loop there at that point is a little bit latent, as in, like, delayed, if you will. You think something like a policy evaluator on... Let's say you have a stated skill for what my company's policies are or my approach is. Let's say you codify Adarsh into a skill, and there's your thinking, there's your principles, there's your operational style, and then that keeps running in the background and sort of, like, grading what... What's happening, if you will. Recently, there's news that, you know, Meta has done something similar with Zuck's personality or Zuck's skills. You think that could be, like, an interesting way, like, the company takes on its own skill, the enterprise takes on its own scale, and then that becomes like a continuous feedback loop, agnostic of evals. Like, for sure, that that loop will also have evals. Yeah. But on an ongoing basis, am I... Is is an employee doing better or is a agent doing better or not, is driven effectively by guardrails and principles and policies? [00:35:57] S3: I have a vested interest, actually, in that question, actually my CISO and I have both created let's call it Nancy and Jacob skills, which we've shared with the rest of the company in terms of, well, what would our CISO think about this problem? Or what would Nancy say about this problem? [00:36:12] S1: Well, have my own swarm of agents that I run personally.
00:36:12 · Dev Tagare, Adarsh Hiremath
[00:36:12] S1: Well, have my own swarm of agents that I run personally. [00:36:15] S2: Yeah. I mean, the answer is that, I I think this is very much a possible thing. And as Nancy mentioned, you know, one password is already doing this. I think more broadly, the thing that's interesting to me is that agents present a very unique opportunity to democratize and diffuse knowledge across the company. Right? Where without these agents, without the CEO agent, right, people operate in the silo of their specific organization or team or KPI that they're trying to hill climb. And that can cause problems, misaligned incentives, not necessarily out of malice, but just out of a lack of context. But in a world where, let's say, you have the Nancy agent and the engineering team might have a question about Nancy's overall strategy and 1Password's overall strategy. You could just ask the agent and has all of the access to the documents that Nancy has, and it becomes a very, very easy way to democratize and diffuse that context in your organization in a way that I think is much, much harder to do without agents. I feel very strongly that what we're seeing with the Zuck agent, the Nancy agent, a bunch of a bunch of other agents will will will will become the norm in these companies.
00:36:15 · Adarsh Hiremath, Dev Tagare, Nancy Wang
[00:36:15] S2: Yeah. I mean, the answer is that, I I think this is very much a possible thing. And as Nancy mentioned, you know, one password is already doing this. I think more broadly, the thing that's interesting to me is that agents present a very unique opportunity to democratize and diffuse knowledge across the company. Right? Where without these agents, without the CEO agent, right, people operate in the silo of their specific organization or team or KPI that they're trying to hill climb. And that can cause problems, misaligned incentives, not necessarily out of malice, but just out of a lack of context. But in a world where, let's say, you have the Nancy agent and the engineering team might have a question about Nancy's overall strategy and 1Password's overall strategy. You could just ask the agent and has all of the access to the documents that Nancy has, and it becomes a very, very easy way to democratize and diffuse that context in your organization in a way that I think is much, much harder to do without agents. I feel very strongly that what we're seeing with the Zuck agent, the Nancy agent, a bunch of a bunch of other agents will will will will become the norm in these companies. [00:37:18] S1: Related technique I'm trying out was like, hey, what's top of mind for you kind of thing. Yep. So on a weekly basis, I've been marking, you know, meetings or like summaries of meetings, stuff like that as like top of mind categories. So when someone triggers like, hey, what's top of mind for dev? They actually get that response on a weekly basis. [00:37:40] S3: So
00:37:40 · Nancy Wang, Dev Tagare, Adarsh Hiremath
[00:37:40] S3: So [00:37:41] S1: that way, like, no one really needs to ping me for anything. They can just go to, like, your chat space and, you know, add me and that shows up. Yeah. Love that. So that's like an interesting, you cheap way to to communicate information or Yeah. Submit [00:37:57] S3: It, like, really scales your time. [00:37:59] S1: Yeah. Yeah. It definitely does. Definitely does. Yes. [00:38:01] S2: Yeah. I I mean, in some cases, at least for me, agents can be, like, better at at structuring my thoughts than than I can be. Right? Because it's able to actually process a bunch of information that I'm not able to process. Mhmm. Or even if the conclusions are the same, there might be something novel that we can also learn from as well. So I I find these agents to be great Yeah. Assistants and and companions. And and, like, what's fun is companions, coworkers. [00:38:24] S1: Coworkers. Yeah. They could be companions at some point. [00:38:28] S3: Yeah. Wasn't that one of the first wave of AI companies? Yeah. Companions.