Zero-Shot Learning
00:00:02 · Nancy Wang
[00:00:02] S0: Hey, everybody. Welcome to Zero Shot Learning, a podcast about the reality of how you build with AI from the people actually doing the work. I'm Nancy Wang, and I'm the chief technology officer at 1Password. I cohost this podcast show with Dave Tagare, who is a senior director and head of engineering for Gemini Enterprise and Business at Google. You'll see Dave in the interview, but I'll share now that he's joining us in a personal capacity, and his views are entirely his own. In this episode, we sat down with Adarsh Hiramath, who's a co founder and now co CEO of Merkor, an AI powered hiring platform. While he was in college, Adarsh recognized a massive bottleneck in the AI revolution. Frontier AI models can't scale without expert human data and sourcing that feedback across specialized domains is pretty difficult. Merkur is building the infrastructure to solve that exact problem. Under Adarsh's technical leadership, Merkur has built an automated talent assessment engine that completely bypasses resume bias to evaluate human capability through AI driven indexing and interviews. In this interview, we talk about how his platform acts as a bridge for Silicon Valley's largest AI labs, where it can place thousands of specialized experts like software engineers, filmmakers, economists, really anyone who can evaluate and train the next generation of LLMs and autonomous agents. If you've been thinking about where the human judgment behind model training actually comes from and what it takes to build a reliable supply chain for this, This conversation will get us into the details. Lock in. This is Zero Shot Learning.
00:01:54 · Dev Tagare, Adarsh Hiremath, Nancy Wang, Nancy Wang
[00:01:54] S1: Welcome, Adarsh. Lovely to have you on the pod. [00:01:57] S2: Thank you for having me. [00:01:58] S0: Quick intro on Adarsh. [00:02:00] S3: We're gonna, you know, talk about all the cool things that you've done as the co founder and now co CEO of Merkor. A couple of questions. We talk a lot about how Merkor has been helping enterprises and businesses capture the way they do work. Well, what does good work actually look like inside an organization? And then secondly is, how can you actually evaluate whether AI systems are performing real work, potentially, let's say, in the world of software engineering, versus just, let's say, narrowly prompting or helping them ace coding tests? And so, as we dig into the technical side of that loop, let's maybe talk more about how you at Merkor build reliable human workflows using agents.
00:02:39 · Adarsh Hiremath
[00:02:39] S2: So for the past couple of years, we've spent a bunch of time supporting the labs with training, deploying, and evaluating agents. Training meaning making them better, deploying, making them practical, and evaluating meaning the evals that, help us understand whether or not the agent is is behaving as as how we intended. So very, very quickly, we realized that a bunch of enterprises started reaching out to us, asking for help with the exact same thing. Right? We wanna train agents. We wanna deploy agents. We wanna evaluate them. And Mercur became a preferred vendor for that. So we support everything from the hyperscalers who wanna build evals for every single one of their departments in the company, to companies that have a specific use case, and a specific agent that they wanna build. The two challenges that we've observed come from telling the agent what to do, which is thing one. And then thing two is evaluating whether or not it's doing it well. And in an enterprise context, telling the agent what to do has its own set of complexities. Right? Because oftentimes, the most valuable big rock problems to solve Mhmm. The context to solve those problems is kind of like this tribal knowledge that's deep within an enterprise. And even if that knowledge exists, it's not in a digital space that's actionable or useful for an agent. So actually discovering that content, pushing... Pulling it out, putting it in the specification of an agent is a really, really challenging problem. Then the second thing is once that agent is actually live and in production, how do you understand whether or not the agent is performing well? This And is where Merkur's bread and butter comes in, which is evals. Right? You need to understand what does the human workflow look like? What does success or good look like at the agent attempting that workflow? And when it fails, how do you have this continuous learning loop to make sure that that mistake never happens again? So we have productized that whole thing and are now working with enterprises on their AI deployments to make sure they're as reliable as possible.
00:04:18 · Nancy Wang
[00:04:18] S3: So given that the problem space, right, of many businesses is so vast... So for example, take 1Password. Right? For us, we're looking to build, for example, dev specific workflows. Another company might want to build customer support, or maybe even recruiting specific workflows. How do you even know what good looks like in that case?
00:04:35 · Adarsh Hiremath
[00:04:35] S2: Yeah, yeah. Even just taking the example of 1Password, right? The key thing is it's not just an agent that does dev workflows. It's an agent that does dev workflows workflows within one password. And for that agent to do a good job, it needs to know what one password considers is good for a specific role or a set of developer tasks. Right? And that's like the the the magic thing here. Right? Even for simple workflows, just the general intelligence of the model is not sufficient. Right? Let's just say reviewing a resume. You could take the smartest model and then have it review your resume. If it's not calibrated with your recruiting team and what that team's looking for in the talent profile of a resume, it's just gonna fall flat on its face. So the question of how do we understand what good looks like requires extracting the knowledge from an enterprise. It can be everything from connecting to SaaS tools, uploading documents, AI moderated interviews of people actually doing the job. And the second thing is an eval that helps us understand the failure modes. Specifically, the eval would be the task the model is trying to do, the trajectory or reasoning steps that that agent is taking, and then the actual outcome graded against a set of subjective or objective criteria that we call a rubric.
00:05:39 · Dev Tagare
[00:05:39] S1: Makes sense. Yeah. So so just double clicking into this a little bit, Azarsh. You know, there's always this thing of, like, expert judgment in any of the enterprise context. Right? Like, usually enterprises are deep in a specific domain area. And when you go a transition from quantitative data to somewhat expert opinion or expert judgment, more qualitative, more intuition driven, there is a lot of spectrum for signal versus noise in this journey. One, how do you solve for it? Two, when you really dig into this, are there any, like, failure patterns or modes that have surprised you more than than others?