Lenny's Podcast: Product | Career | Growth
00:06:11 · Brendan Foody, Lenny Rachitsky
[00:06:11] S0: If the model is the product, then the eval is the product requirement document. And the way that researchers' day to day looks is that they'll run dozens of experiments where they'll make small improvements on an eval set. And reinforcement learning is becoming so effective that once they have an eval, they can help climb it, right? If you look at just how fast people were able to saturate Olympiad math once they focused on it, how fast we're even saturating SWE bench once we focus on it. And so in many ways, the barrier to applying agents, the entire economy to automate every workflow is how do we measure success? How do we eval it and write the PRDs for everything that we want agents to do, which Merkor is obviously a huge part of doing. [00:06:56] S1: So people hearing this are like, okay, okay, shit, I gotta really pay attention to this eval stuff. Any advice about learning how to do this well? What companies that are doing this well are doing differently? Like help people get better at this thing.
00:06:56 · Lenny Rachitsky, Brendan Foody
[00:06:56] S1: So people hearing this are like, okay, okay, shit, I gotta really pay attention to this eval stuff. Any advice about learning how to do this well? What companies that are doing this well are doing differently? Like help people get better at this thing. [00:07:10] S0: Yeah, I think that for enterprises especially, the core way to think about it is how can they build a test or a systematic way to measure how well AI automates their core value chain? So if it's an architecture firm that's producing these architecture diagrams of what they provide to their end customer, how can they effectively measure that? And each company has its own value chain, or maybe a handful of them if it's a multi product company. And just thinking about how they measure that is the prerequisite to really effectively applying AI throughout their entire business. [00:07:52] S1: I saw you talking about this on the No Priors podcast with Sarah and Elad, and I don't know if it was after this or before this, but Sarah tweeted, Evals equals your new marketing. What do you what does that mean? What do you think she's saying there?
00:07:52 · Lenny Rachitsky, Brendan Foody
[00:07:52] S1: I saw you talking about this on the No Priors podcast with Sarah and Elad, and I don't know if it was after this or before this, but Sarah tweeted, Evals equals your new marketing. What do you what does that mean? What do you think she's saying there? [00:08:03] S0: Yeah. Well, ties to what I said earlier about how if the model is the product, evals are the PRD, but also subsequently the sales collateral, right? Because evals are what you give to researchers to show them what they should be building and going on, but they're also the way that you demonstrate the efficacy of capabilities. And historically, everyone's been pointing to these academic evals of PhD level reasoning with GPQA, Humanities Last Exam, or Olympiad Math, but now it's moving towards the capabilities that people practically care about, of how do we get models to automate the way that we build a software platform or automate the way that we do an investment banking analysis. I think labs will increasingly use labs as well as application layer companies will increasingly use EFLs to demonstrate the capabilities of their models and their products.
00:08:58 · Lenny Rachitsky
[00:08:58] S1: Okay. So let's kind of build on this and zoom out a little bit and talk about the landscape of the market that you're in. And I was just reflecting on this as I was preparing for this conversation. If you think about the companies growing faster than any company's ever grown in history, there's essentially three buckets. There's the foundational model companies, there's Vibe coding apps, Cursor and Lovable, and Bolt and Replit and all these v0, and then there's data labeling data companies like you. So I've had the CEO of Handshake on the podcast, I have the CEO of Scale coming on, there's also Surge, there's you guys. Help us just understand the landscape of what this is all about, because I think people don't really know what the hell is going on and see all these companies growing like crazy.
00:09:37 · Brendan Foody
[00:09:37] S0: Yeah, I'll give a little bit of the origin story and that and how it sort of frames the landscape. Because when we started the company, I met my co founders in when we were 14 years old. We started the company together when we were 19 initially hire... In January 2023, initially hiring people internationally, matching them with our friends and automating all the processes of how we did that. So similar to how a human would review a resume, conduct an interview and decided to hire, we automated all of those processes with LMs, bootstrapped the company to a million dollar revenue run rate before we dropped out of college. And then, a handful of other things happened, but we met OpenAI and we saw that there was this enormous transition in the human data market, where it was moving away from this crowdsourcing problem of how do you find low and medium skilled people that can write barely grammatically correct sentences for early versions of LLMs, and moving towards this sourcing and vetting problem. How do we source and assess the best professionals, the experienced THANGS software engineers, the investment bankers and doctors and lawyers that can actually help to evaluate and interpret all of the capabilities that people want models to have. So from there, we started working with all of the top AI labs. We grew from 100 to $400,000,000 in revenue run rate in, sixteen months, and it's been an extraordinary journey, and super exciting.
00:11:07 · Lenny Rachitsky, Brendan Foody
[00:11:07] S1: Okay. First of all, that is out of control. I don't know if people have under... I think this is the first time you're sharing that number. Know recording this. You'll have announced it by now, but 1 to $400,000,000 in revenue in sixteen months. [00:11:21] S0: Exactly. So fastest ascent in history, which is an exciting statistic we're very proud of. [00:11:28] S1: Okay, so something big is happening here. Why is this so valuable? What is going on here? So it's just to try to summarize what you guys do simply is you help hire people for labs to train... Help them train their models, and you help them find not just generalist labor, but experts helping them with very specific gaps in the model's knowledge. [00:11:52] S0: Yeah, precisely. And so it really ties to your first question around the era of evals that's framing all of this, which is that the lab's primary bottleneck to being able to improve models is how they can effectively have some way of measuring what success looks like for the model, both to use it as the eval for the tests that they're measuring their progress against, as well as the verifiers in an RL environment to then reward the model, improve capabilities, etcetera. And they need this across every domain for every capability that models don't know how to use. And the wealthiest companies in the world are willing to spend whatever it takes to improve model capabilities where Merkur is sitting at the forefront, and sort of the primary bottleneck.