Lenny's Podcast: Product | Career | Growth
00:32:27 · Lenny Rachitsky, Brendan Foody
[00:32:27] S1: It's interesting. There's always been these companies, AlphaSight and, GLG, that like sort of did this before AI, or is like paid to connect to an expert and ask them questions about stuff. And essentially, okay, it turns out this is really useful for models. We don't need the person in the middle. [00:32:42] S0: Exactly. Yeah. Well, the one core difference is that alpha sites would generally be a one off call versus a lot of our work is really hiring people for projects, right? Of how do they work on something for a longer period of time? And so that's, I think, one of the reasons that some of the traditional expert networks have struggled to get into this. And also, how do you retain those people and think about all the incentives where it actually looks more similar in some ways to one of the traditional labor marketplaces of an Uber or DoorDash, just with much higher skilled talent that's treated exceptionally well. [00:33:22] S1: It's such a good opportunity for me to learn so much about this. I'm just going to ask questions. Yeah, sure. So interesting to me. How much of the experts are focused on specific concrete knowledge versus personality and like softer skills? How much of it is like, here's how you do an exam, here's how you do an x-ray?
00:33:22 · Lenny Rachitsky, Brendan Foody
[00:33:22] S1: It's such a good opportunity for me to learn so much about this. I'm just going to ask questions. Yeah, sure. So interesting to me. How much of the experts are focused on specific concrete knowledge versus personality and like softer skills? How much of it is like, here's how you do an exam, here's how you do an x-ray? [00:33:40] S0: It depends on the lab. It's a lot of both. I think that previously it might've been more softer skills, but now a lot of the labs are focused on their business models of what are the economically valuable capabilities that drive revenue and leaning a lot into these professional domains. But I think the creative side is also still really important to everyone. And so we're seeing a meaningful amount of both. Like we hired all the people from the Harvard Lampoon a couple months ago, their comedy club to help with making models funnier. And so do all sorts of stuff like that, hiring Emmy award winning screenwriters and everything across the board on creative capabilities that you'd look for. [00:34:23] S1: That is amazing. What a cool story. Yeah. I'm excited for this to kick in. How fast do these things turn around? Like, say you hired this team, like how fast are we going to see the impact potentially? Is it like months? Is it years? [00:34:34] S0: Well, it depends because some models, or some labs will release iteratively where they'll just improve the model behind the scenes, sort of every [00:34:42] S1: Without announcing a new model.
00:34:42 · Lenny Rachitsky, Brendan Foody
[00:34:42] S1: Without announcing a new model. [00:34:44] S0: Exactly. As sort of every couple of weeks versus others do these big releases. And so it depends a lot. We're behind all of them, but the, I mean, we move really fast. It would be a customer gives us a request of we need these, you know, award winning screenwriters. And within twenty four hours, we'll turn around, the experts. And there's also this really interesting dynamic where in a set of a 100 people that we hire, oftentimes the top 10% of people will drive majority of the model improvement. It's sort of like a company, right? If you have a 100% company, oftentimes the top 10% of the company, will drive majority of the impact. And what that means is that when we're able to build proprietary advantages in identifying who are those top 10 of people, both insofar as how do we have them on our platform, but also identify and match them effectively, it creates so much value for customers that it's difficult to compete against. And so it really does tie back to the founding thesis of the company, which is how do we find these extraordinary people and identify them so that we can reliably deliver these top 10% or top 10x experiences for our customers. [00:35:56] S1: So on that, so is the idea you hire Jane, she's incredible at coding, and she now works for Anthropic, and that's her full time job doing this, or is this like a part time thing? Is this a project thing mostly?
00:35:56 · Lenny Rachitsky, Brendan Foody
[00:35:56] S1: So on that, so is the idea you hire Jane, she's incredible at coding, and she now works for Anthropic, and that's her full time job doing this, or is this like a part time thing? Is this a project thing mostly? [00:36:09] S0: It would sometimes be part time, sometimes it would be full time. I would say most often it's part time, where it's like, you know, someone might work at fan a company where they're underemployed, maybe one of the ones that's moving slower, where they have an extra twenty hours a week, and then they're able to do this on the side, or, you know, whatever the equivalent is sort of across a bunch of different industries. But we also do a lot of, you know, forty hour a week roles as well. [00:36:39] S1: And these, how much are they making? Is it like, like meaningful enough for Fannie engineers to spend time on this?
00:36:39 · Lenny Rachitsky, Brendan Foody
[00:36:39] S1: And these, how much are they making? Is it like, like meaningful enough for Fannie engineers to spend time on this? [00:36:45] S0: Yeah, very meaningful. I mean, so our median pay rate in the marketplace 95 an hour, but it can flex up well up into like $500 an hour, based on the depth of someone's expertise. And one thing that highlights this difference relative to a lot of the crowdsourcing companies is if you look at the economics of the crowdsourcing companies, oftentimes they would pay like $30 an hour to Talon as sort of the average. And so think about the, you know, people that you can hire, the undergrads for $30 an hour versus the you know, Goldman bankers, the McKinsey analysts, the, FANG software engineers. And ultimately it comes down to what are the capabilities that labs want their models to have? And it much more falls in the latter bucket than the former one. [00:37:33] S1: I know there s only so much you can talk about with this stuff, but so Anthropic Claude has been so good at coding, so much better historically than other models. I also use it for writing, giving feedback on writing. What is it that allowed them to get so good at this and continue to be so good at this? [00:37:51] S0: Well, can't go too much into detail about customer work, but I think that it's this trend of reinforcement learning and being very thoughtful about defining the right rewards that we're really seeing across the board, and how we can mitigate reward hacking, set up the right rewards. That's super impactful. [00:38:14] S1: Evals, [00:38:14] S0: again,
00:38:14 · Brendan Foody, Lenny Rachitsky
[00:38:14] S0: again, [00:38:15] S1: Back evals is [00:38:15] S0: all you to evals. One of my favorite quotes from customers is that models are only as good as their evals, which has always held true. [00:38:24] S1: I think Greg Brockman tweeted this once, evals are all you need. [00:38:28] S0: Yeah, no, truly. [00:38:29] S1: Let's talk about Marcour a little bit more. One of the maybe... Not even maybe. I believe, the data tells us it's the fastest growing company in history. [00:38:39] S0: Yeah. [00:38:40] S1: I wanna understand what you did to make this happen. So let me just ask, what do think are some of the core tenets of how you built Morcord that most contributed to being this successful?