Zero-Shot Learning
00:28:20 · Dev Tagare, Nancy Wang, Adarsh Hiremath
[00:28:20] S1: Perfect. And like you throw an HITL somewhere in the loop with and without production. Right. That way, you have some control over what could happen, what should happen, so on and so forth. [00:28:31] S3: That's right. And as a quick aside, don't want to take us down, like, a huge rabbit hole. But, you know, I was on a call with a large multinational bank this morning, where they're saying that for more of their sensitive workloads, they're actually running it not just in sandboxes, but in remote dev environments. Are you seeing that request from some of your more compliance minded customers? [00:28:50] S2: Yes. [00:28:50] S3: Yes. Okay. And they're all kind of workloads. 100%. [00:28:53] S2: Yeah. Mean, I the typical ones are are are workflows in highly regulated industries. Right? Like, financial services related use cases being a, being a big one. The answer is yes. Right? Like, people... Like, the the general idea of you do not wanna run the agent in production for the first time. You wanna run it on sandbox environments, locally, remote just to every... The whole thing under the sun, people are are experimenting with that. [00:29:17] S1: You know, sandboxing and all of those is gonna be a big business very soon. [00:29:21] S3: I see actually the worlds of sandboxing and remote dev environments, Coder, for example, and others, converging over time.
00:29:21 · Nancy Wang, Adarsh Hiremath
[00:29:21] S3: I see actually the worlds of sandboxing and remote dev environments, Coder, for example, and others, converging over time. [00:29:27] S2: Totally. Right? And and we, course, do a lot of that work with the labs, right, which is, these RL environments, right? Instead of if you don't have access to the real production tools and context, or you don't wanna give agents that access when you're rolling something out, it can be helpful to create a mock up. Right? Which is select the agent, select the set of tasks you wanna test on, create mock ups of everything, mock ups of all the tools that that the agent would need to have access to, use those mock outs to... Mock ups to roll out a trajectory, eval the trajectory, and then eval the output. Is is what people are doing in in context like these. Totally. [00:30:04] S3: So speaking of outputs, I wanna maybe take us to actually the recent benchmark that y'all published, Apex Suite. Yeah. Yeah. Super cool. And just for listeners here, Apex suite tries to measure, you know, whether frontier models can do economically valuable work over time. So I'm curious, you know, what led you all to create this benchmark, and what was missing from existing benchmarks?
00:30:26 · Adarsh Hiremath
[00:30:26] S2: The thing that inspired the whole Apex series of benchmarks is that we felt that there are benchmarks that are really good at maybe evaluating how well the model can do an academic task. Mhmm. Can it can it do IMO math? Yeah. Maybe modeling how well, you know, these models can do tasks that are kind of like an approximation of the economy, right, like GDP val. But there's no benchmark that can really evaluate how well can the model do enterprise work output, right? Or or knowledge work, or or just like real work. That's when we started with Apex, the first version, which was largely for professional services. We made that benchmark agentic, and then we also rolled out Apex Suite. In the software engineering context, the biggest gap that I see is that a lot of these benchmarks are isolated to small patch tests. Right? Okay. This whole repository is messed up. Just like patch these kind of functions. Right? So it focuses on these patches as opposed to the whole system. And secondly, as you see the failures in coding models, just based on these benchmarks. Right? The failure of the benchmarks to create this distribution of real software engineering work is also a failure that we see in the models in production. So APEC SWE is our attempt at addressing that. And there are two types of tasks in APEC SWE. The first one is observability, right? Which is in a real enterprise context, if system fails, right, you would need the agent to triage across all these different things, go look at the logs and say, hey, this is not right, and then raise that to the user in whatever area that they're doing work. And the second is integration, right, which is just not a patch test or repository. It's a set of systems and how they integrate with one another. The last thing about Apex is, as we know, just correctness, whether the code compiles or not, is, like, not the the only thing. Yeah. It's also these other things that are super relevant. Right? Is this a piece of code that you would be excited about, being in your code base six months from now? Does it follow style requirements? Does it follow all these other things? So the way that we actually come up with the benchmark scores in Apex Suite is a rubric that is not just correctness based, but it also has other criteria, to evaluate how well the models are actually performing in a real context.
00:32:33 · Dev Tagare, Adarsh Hiremath
[00:32:33] S1: So one of the the themes that comes out of the paper is, like, strong performance seems to come from, you know, epitomistic discipline, knowing that you've verified something versus just producing it and then evolving it over time and continuing to verify it over time. So how does this discipline look like in the world of agents purely, where now you have swarms, they're solving for sort of like competing goals like humans would in a big company. How does that that benchmark evolve to to address essentially an ecosystem problem versus a specificity problem at the level of an individual engineer or so? [00:33:12] S2: I would say that this is something that's very top of mind for us, and in fact, we're actively working on a project that is super relevant to this. The short answer is the unexplored space is agents as coworkers. Right? And agents actually talking to each other. Right? We have these benchmarks that evaluate how well can an agent do a task when it's given a task. But do we have benchmarks that evaluate how well agents can work with one another? The answer is no, and it's very top of mind from recording. [00:33:37] S1: And similarly, like, taking taking the the same concept a little bit further, let's say external environmental factors change, that creates a level of uncertainty. Humans react to it in a certain way. Nancy may react to it differently, you may react to it differently, I may react to it differently. Agents are more deterministic in that sense, like, you know, there are instructions associated with it.
00:33:37 · Dev Tagare, Adarsh Hiremath
[00:33:37] S1: And similarly, like, taking taking the the same concept a little bit further, let's say external environmental factors change, that creates a level of uncertainty. Humans react to it in a certain way. Nancy may react to it differently, you may react to it differently, I may react to it differently. Agents are more deterministic in that sense, like, you know, there are instructions associated with it. [00:34:02] S2: They have less feelings. [00:34:03] S1: Less feelings, no feelings, you could say that. Right? More predictable. More predictable. For sure. More predictable for sure. So when uncertainty hits you or, like, environmental factors change, how are you thinking about evolving this ecosystem to account for it, and how would you think about measuring how the agent did versus how the human did? [00:34:23] S2: I mean, I know I sound like a broken record. The answer is just the eval. Right? Like, you... In the same way that you can scaffold the task trajectory and output, you can do the same thing with a human. Right? You can get as granular as literally recording what they're doing. Right? And the audio and the video and the screenshots and all the tools. Or you can use documents or you could do AI moderated interviews, but there are ways to create a similar infrastructure for humans and then directly compare with with agents via an eval.
00:34:53 · Dev Tagare, Nancy Wang
[00:34:53] S1: Makes sense. And and, like, the the feedback loop there at that point is a little bit latent, as in, like, delayed, if you will. You think something like a policy evaluator on... Let's say you have a stated skill for what my company's policies are or my approach is. Let's say you codify Adarsh into a skill, and there's your thinking, there's your principles, there's your operational style, and then that keeps running in the background and sort of, like, grading what... What's happening, if you will. Recently, there's news that, you know, Meta has done something similar with Zuck's personality or Zuck's skills. You think that could be, like, an interesting way, like, the company takes on its own skill, the enterprise takes on its own scale, and then that becomes like a continuous feedback loop, agnostic of evals. Like, for sure, that that loop will also have evals. Yeah. But on an ongoing basis, am I... Is is an employee doing better or is a agent doing better or not, is driven effectively by guardrails and principles and policies? [00:35:57] S3: I have a vested interest, actually, in that question, actually my CISO and I have both created let's call it Nancy and Jacob skills, which we've shared with the rest of the company in terms of, well, what would our CISO think about this problem? Or what would Nancy say about this problem? [00:36:12] S1: Well, have my own swarm of agents that I run personally.