Lenny's Podcast: Product | Career | Growth
00:08:58 · Lenny Rachitsky
[00:08:58] S1: Okay. So let's kind of build on this and zoom out a little bit and talk about the landscape of the market that you're in. And I was just reflecting on this as I was preparing for this conversation. If you think about the companies growing faster than any company's ever grown in history, there's essentially three buckets. There's the foundational model companies, there's Vibe coding apps, Cursor and Lovable, and Bolt and Replit and all these v0, and then there's data labeling data companies like you. So I've had the CEO of Handshake on the podcast, I have the CEO of Scale coming on, there's also Surge, there's you guys. Help us just understand the landscape of what this is all about, because I think people don't really know what the hell is going on and see all these companies growing like crazy.
00:09:37 · Brendan Foody
[00:09:37] S0: Yeah, I'll give a little bit of the origin story and that and how it sort of frames the landscape. Because when we started the company, I met my co founders in when we were 14 years old. We started the company together when we were 19 initially hire... In January 2023, initially hiring people internationally, matching them with our friends and automating all the processes of how we did that. So similar to how a human would review a resume, conduct an interview and decided to hire, we automated all of those processes with LMs, bootstrapped the company to a million dollar revenue run rate before we dropped out of college. And then, a handful of other things happened, but we met OpenAI and we saw that there was this enormous transition in the human data market, where it was moving away from this crowdsourcing problem of how do you find low and medium skilled people that can write barely grammatically correct sentences for early versions of LLMs, and moving towards this sourcing and vetting problem. How do we source and assess the best professionals, the experienced THANGS software engineers, the investment bankers and doctors and lawyers that can actually help to evaluate and interpret all of the capabilities that people want models to have. So from there, we started working with all of the top AI labs. We grew from 100 to $400,000,000 in revenue run rate in, sixteen months, and it's been an extraordinary journey, and super exciting.
00:11:07 · Lenny Rachitsky, Brendan Foody
[00:11:07] S1: Okay. First of all, that is out of control. I don't know if people have under... I think this is the first time you're sharing that number. Know recording this. You'll have announced it by now, but 1 to $400,000,000 in revenue in sixteen months. [00:11:21] S0: Exactly. So fastest ascent in history, which is an exciting statistic we're very proud of. [00:11:28] S1: Okay, so something big is happening here. Why is this so valuable? What is going on here? So it's just to try to summarize what you guys do simply is you help hire people for labs to train... Help them train their models, and you help them find not just generalist labor, but experts helping them with very specific gaps in the model's knowledge. [00:11:52] S0: Yeah, precisely. And so it really ties to your first question around the era of evals that's framing all of this, which is that the lab's primary bottleneck to being able to improve models is how they can effectively have some way of measuring what success looks like for the model, both to use it as the eval for the tests that they're measuring their progress against, as well as the verifiers in an RL environment to then reward the model, improve capabilities, etcetera. And they need this across every domain for every capability that models don't know how to use. And the wealthiest companies in the world are willing to spend whatever it takes to improve model capabilities where Merkur is sitting at the forefront, and sort of the primary bottleneck.
00:11:52 · Brendan Foody, Lenny Rachitsky
[00:11:52] S0: Yeah, precisely. And so it really ties to your first question around the era of evals that's framing all of this, which is that the lab's primary bottleneck to being able to improve models is how they can effectively have some way of measuring what success looks like for the model, both to use it as the eval for the tests that they're measuring their progress against, as well as the verifiers in an RL environment to then reward the model, improve capabilities, etcetera. And they need this across every domain for every capability that models don't know how to use. And the wealthiest companies in the world are willing to spend whatever it takes to improve model capabilities where Merkur is sitting at the forefront, and sort of the primary bottleneck. [00:12:43] S1: Okay. What are these people actually doing? So what's an example of a kind of person that is sought after, and then what are they doing, like sitting there at the computer?
00:12:43 · Lenny Rachitsky, Brendan Foody
[00:12:43] S1: Okay. What are these people actually doing? So what's an example of a kind of person that is sought after, and then what are they doing, like sitting there at the computer? [00:12:51] S0: Effectively, the market is bound by the amount of things where humans can do something that models can't. So I'll make that very concrete. Say you have a model that you want to write, like, a red line for a contract in the way that a lawyer would, and it makes a handful of mistakes, misses a bunch of key points in doing so. What you could do is have a lawyer create a rubric, similar to how a professor might create a rubric to create a deliverable for what are the things we want the model to be able to do, so it can effectively score that, right? Like, you know, plus however much of it identifies this, or, you know, XYZ key point. And that's really the foundation to measuring what does progress look like for models? Is this model achieving the capabilities that these professionals want? As well as how do we use this as training data to reward and to reinforce a lot of the capabilities that people want models to achieve. [00:13:50] S1: Okay. So they're essentially writing evals just to connect it back to original conversation.
00:13:50 · Lenny Rachitsky, Brendan Foody
[00:13:50] S1: Okay. So they're essentially writing evals just to connect it back to original conversation. [00:13:55] S0: Exactly. Well, that's an interesting thing is everyone talks about RL environment. I feel like the two hot button things are RL environments and evals. But one thing Andre Carvathi has tweeted out about a bunch is there's not actually a nuance in the data type. It's more just a different semantic way of describing what it's being used for. But ultimately it's just some stasis point for how do you measure what good looks like? And you can use that either as the benchmark to the sales collateral, as Sarah was saying, to say, here is why our model's the best model in the world and here's the capabilities that we've been working towards. Or you can use it on the post training side to reward certain model trajectories and achieve those capabilities. [00:14:39] S1: Okay. So say this lawyer, so this person is writing, here's what a great red line contract looks like, and here's the rubric of what excellent is. And then are they also providing data, like actual examples of Redline documents as a part of that?