Lenny's Podcast: Product | Career | Growth
00:07:52 · Lenny Rachitsky, Brendan Foody
[00:07:52] S1: I saw you talking about this on the No Priors podcast with Sarah and Elad, and I don't know if it was after this or before this, but Sarah tweeted, Evals equals your new marketing. What do you what does that mean? What do you think she's saying there? [00:08:03] S0: Yeah. Well, ties to what I said earlier about how if the model is the product, evals are the PRD, but also subsequently the sales collateral, right? Because evals are what you give to researchers to show them what they should be building and going on, but they're also the way that you demonstrate the efficacy of capabilities. And historically, everyone's been pointing to these academic evals of PhD level reasoning with GPQA, Humanities Last Exam, or Olympiad Math, but now it's moving towards the capabilities that people practically care about, of how do we get models to automate the way that we build a software platform or automate the way that we do an investment banking analysis. I think labs will increasingly use labs as well as application layer companies will increasingly use EFLs to demonstrate the capabilities of their models and their products.
00:08:58 · Lenny Rachitsky
[00:08:58] S1: Okay. So let's kind of build on this and zoom out a little bit and talk about the landscape of the market that you're in. And I was just reflecting on this as I was preparing for this conversation. If you think about the companies growing faster than any company's ever grown in history, there's essentially three buckets. There's the foundational model companies, there's Vibe coding apps, Cursor and Lovable, and Bolt and Replit and all these v0, and then there's data labeling data companies like you. So I've had the CEO of Handshake on the podcast, I have the CEO of Scale coming on, there's also Surge, there's you guys. Help us just understand the landscape of what this is all about, because I think people don't really know what the hell is going on and see all these companies growing like crazy.
00:09:37 · Brendan Foody
[00:09:37] S0: Yeah, I'll give a little bit of the origin story and that and how it sort of frames the landscape. Because when we started the company, I met my co founders in when we were 14 years old. We started the company together when we were 19 initially hire... In January 2023, initially hiring people internationally, matching them with our friends and automating all the processes of how we did that. So similar to how a human would review a resume, conduct an interview and decided to hire, we automated all of those processes with LMs, bootstrapped the company to a million dollar revenue run rate before we dropped out of college. And then, a handful of other things happened, but we met OpenAI and we saw that there was this enormous transition in the human data market, where it was moving away from this crowdsourcing problem of how do you find low and medium skilled people that can write barely grammatically correct sentences for early versions of LLMs, and moving towards this sourcing and vetting problem. How do we source and assess the best professionals, the experienced THANGS software engineers, the investment bankers and doctors and lawyers that can actually help to evaluate and interpret all of the capabilities that people want models to have. So from there, we started working with all of the top AI labs. We grew from 100 to $400,000,000 in revenue run rate in, sixteen months, and it's been an extraordinary journey, and super exciting.
00:11:07 · Lenny Rachitsky, Brendan Foody
[00:11:07] S1: Okay. First of all, that is out of control. I don't know if people have under... I think this is the first time you're sharing that number. Know recording this. You'll have announced it by now, but 1 to $400,000,000 in revenue in sixteen months. [00:11:21] S0: Exactly. So fastest ascent in history, which is an exciting statistic we're very proud of. [00:11:28] S1: Okay, so something big is happening here. Why is this so valuable? What is going on here? So it's just to try to summarize what you guys do simply is you help hire people for labs to train... Help them train their models, and you help them find not just generalist labor, but experts helping them with very specific gaps in the model's knowledge. [00:11:52] S0: Yeah, precisely. And so it really ties to your first question around the era of evals that's framing all of this, which is that the lab's primary bottleneck to being able to improve models is how they can effectively have some way of measuring what success looks like for the model, both to use it as the eval for the tests that they're measuring their progress against, as well as the verifiers in an RL environment to then reward the model, improve capabilities, etcetera. And they need this across every domain for every capability that models don't know how to use. And the wealthiest companies in the world are willing to spend whatever it takes to improve model capabilities where Merkur is sitting at the forefront, and sort of the primary bottleneck.
00:11:52 · Brendan Foody, Lenny Rachitsky
[00:11:52] S0: Yeah, precisely. And so it really ties to your first question around the era of evals that's framing all of this, which is that the lab's primary bottleneck to being able to improve models is how they can effectively have some way of measuring what success looks like for the model, both to use it as the eval for the tests that they're measuring their progress against, as well as the verifiers in an RL environment to then reward the model, improve capabilities, etcetera. And they need this across every domain for every capability that models don't know how to use. And the wealthiest companies in the world are willing to spend whatever it takes to improve model capabilities where Merkur is sitting at the forefront, and sort of the primary bottleneck. [00:12:43] S1: Okay. What are these people actually doing? So what's an example of a kind of person that is sought after, and then what are they doing, like sitting there at the computer?
00:12:43 · Lenny Rachitsky, Brendan Foody
[00:12:43] S1: Okay. What are these people actually doing? So what's an example of a kind of person that is sought after, and then what are they doing, like sitting there at the computer? [00:12:51] S0: Effectively, the market is bound by the amount of things where humans can do something that models can't. So I'll make that very concrete. Say you have a model that you want to write, like, a red line for a contract in the way that a lawyer would, and it makes a handful of mistakes, misses a bunch of key points in doing so. What you could do is have a lawyer create a rubric, similar to how a professor might create a rubric to create a deliverable for what are the things we want the model to be able to do, so it can effectively score that, right? Like, you know, plus however much of it identifies this, or, you know, XYZ key point. And that's really the foundation to measuring what does progress look like for models? Is this model achieving the capabilities that these professionals want? As well as how do we use this as training data to reward and to reinforce a lot of the capabilities that people want models to achieve. [00:13:50] S1: Okay. So they're essentially writing evals just to connect it back to original conversation.