Entrepreneurial Thought Leaders (ETL)
00:12:29 · Emily Ma, Brendan Foody, S5
[00:12:29] S1: Okay. How did you guys divide and conquer? You did go to market. Adarsh was very close to India. Sounds like Surya took care of sort of internal functions. How did you decide on that, how to divide and conquer based on, you know, all the things that needed to get done? [00:12:42] S2: Well, it's interesting. I actually coded our initial product. Oh. And I... Or I would code the front ends and then work with some Indian engineers on the back ends. It's just fun to, like, spin up the coding tools today. And then, eventually, once I started getting contracts, we figured a cleaner structure would be that Adarsh would be CTO at the time to manage engineering since he was studying CS and had spent the most time coding Yeah. Out of all of us. [00:13:14] S5: Mhmm. [00:13:15] S2: Surya was... Is very well connected to the IITs in India. Ah. And so he set up a lot of our initial recruiting functions. And I don't know. It just felt like a natural way that things panned out. [00:13:30] S1: And is it still that way? How has it evolved since?
00:13:30 · Emily Ma, Brendan Foody
[00:13:30] S1: And is it still that way? How has it evolved since? [00:13:33] S2: Not really. It's, it's taken many different forms because I feel like it's always about working on whatever the most important problem is at any given time so it's more dynamic. But the rough split right now is that I focus on the AI labs and really our frontier customers that are, you know, building the best agents in the world that everyone uses every day. And then Adarsh is focused more so on enterprises. Mhmm. How do we take all of the processes, know how, technology, etcetera, that we've built for the AI labs over the last two years and apply that to the rest of the Fortune 500. [00:14:11] S1: Got it. Okay. You know, we've kind of putz around a little bit about exactly what Merkor does, and not everybody in this room is a computer scientist or working on AI. We have an incredible, like, breadth and diversity of students here. Could you maybe start from, like, square one for somebody who is a layman, a layperson, a student who isn't familiar with, you know, the AI labs, like, building models? Like, could you tell us a little bit about, you know, what Merkor does? And then I wanna then tie that to benchmarking. Right? So defining benchmarking as the most basic definition.
00:14:44 · Brendan Foody
[00:14:44] S2: Totally. Yeah. So actually, I'll also give a little bit of history alongside this. The first stages of data creation were behavior cloning data where the two most common types are supervised fine tuning data where you would have inputs and outputs for people that have played around with fine tuning APIs, and then RLHF data where you would have the preference that you sometimes see in ChatGPT if the model generates a couple responses and you choose which you prefer. What eventually happened as we move towards agentic data is that people started to create environments where they instead have the human structure the success criteria, similar to how a professor might create a rubric for your exam in a class of, you know, plus five points if the model does this, plus 10 points if they do this, minus five points if it makes this mistake, and that could be in the form of rubric criteria or unit tests. And then you're able to use that data to either measure how well the model is performing to see which model should we be using, whether it's the newer model or the older model. And as researchers experiment, that would be a benchmark to measure success, evaluate success, and you see the benchmark scores whenever labs are announcing their new models. The other way that you can use it is as a reward function for reinforcement learning, where you could have the model attempt the problem a 100 times or 50 times. You could score all of those attempts, and the model can very quickly learn to help climb that. And so that's, like, the more granular technical explanation, but what we do is we build out all of these rubrics, unit tests, and all the context and tools that models use to cover pretty much the distribution of anything you could do in ChatGPT, Cloud Code, Cloud Cowork, or Gemini.
00:16:34 · Emily Ma, Brendan Foody
[00:16:34] S1: Amazing. Thank you. [00:16:36] S2: Yeah. [00:16:36] S1: Okay. So let's talk about Apex. I think that was a huge win, and it's not just one Apex. I know you started with investment banking and management consulting and law and whatnot, but that's branched out to include AgenTic and include SWISE and whatnot. Why is that important? Where is this all going?
00:16:36 · Emily Ma, Brendan Foody
[00:16:36] S1: Okay. So let's talk about Apex. I think that was a huge win, and it's not just one Apex. I know you started with investment banking and management consulting and law and whatnot, but that's branched out to include AgenTic and include SWISE and whatnot. Why is that important? Where is this all going? [00:16:54] S2: Yeah. So it actually ties back to... I remember my first conversation with Sean, we talked about IMOmedalists. And in the early days of models, everyone was talking about very academic benchmarks to set the goals for models of how could we get IMO medalist to measure Olympiad math? How could we get PhDs in GPQA? All the way to how could we get, you know, IOI medalist for, various coding evals? And then there was this big transition in the market that we saw early on moving away from academic capabilities towards economically valuable capabilities of how do we measure the things that professionals do in their real jobs, How do we create the distribution of the prompts, the context, the rubrics or success criteria corresponding to that? And measure that distribution across investment banking, law, medicine, software engineering, and all sorts of the most economically valuable domains. And so we built Apex with that purpose, and it's quickly gotten adopted by a lot of the top labs as, becoming this industry standard for how labs determine whether their models are doing well at the capabilities that enterprises care about and how enterprises make buying decisions for what model they should be using. Yeah.
00:16:54 · Brendan Foody, Emily Ma
[00:16:54] S2: Yeah. So it actually ties back to... I remember my first conversation with Sean, we talked about IMOmedalists. And in the early days of models, everyone was talking about very academic benchmarks to set the goals for models of how could we get IMO medalist to measure Olympiad math? How could we get PhDs in GPQA? All the way to how could we get, you know, IOI medalist for, various coding evals? And then there was this big transition in the market that we saw early on moving away from academic capabilities towards economically valuable capabilities of how do we measure the things that professionals do in their real jobs, How do we create the distribution of the prompts, the context, the rubrics or success criteria corresponding to that? And measure that distribution across investment banking, law, medicine, software engineering, and all sorts of the most economically valuable domains. And so we built Apex with that purpose, and it's quickly gotten adopted by a lot of the top labs as, becoming this industry standard for how labs determine whether their models are doing well at the capabilities that enterprises care about and how enterprises make buying decisions for what model they should be using. Yeah. [00:18:11] S1: How do you find, like, how do you find the authors? How do you find the right people? [00:18:17] S2: The experts that help to create these datasets. [00:18:20] S1: Yep.