Lenny's Podcast: Product | Career | Growth
00:27:06 · Lenny Rachitsky, Brendan Foody
[00:27:06] S1: What I've seen in how hiring has changed, I'm doing research on this with a partner Gnome, it s so much easier to apply for companies that everyone is just applying out to hundreds of companies. AI is just making it easy to adjust their resumes and cover letters and make it feel like, oh I applied to more courses very specifically, but it was one of 100 places. And then on the flip side, hiring managers are getting flooded with applications. And so now they need AI to filter. So even if we didn't want to get to this place, we're almost being pushed into this direction of so much volume on both sides, and we need something really smart at filtering and helping us hire and select. And this is exactly what you guys have been building for a long time. [00:27:45] S0: Precisely. Yeah. And the fascinating thing, like a lot of people ask, are we, do we think about ourselves as a labor marketplace or do we think about ourselves as a data company? And I think that the reason it's an interesting question is our realization from what the labs need is that they actually need a labor marketplace. They actually need these exceptionally high caliber people. And of course, we'll layer on some project management and some software platform associated with it. But the really core thing that they want is how do they find these extraordinary professionals across all of these different domains that can measure model capabilities and work to build that future work together.
00:28:28 · Lenny Rachitsky
[00:28:28] S1: This episode is brought to you by Interpret. Interpret is a customer intelligence platform used by leading CXN product orgs like Canva, Notion, Perplexity, Strava, Hinge, and Linear to leverage the voice of the customer and build best in class products. Interpret unifies all customer conversations in real time from Gong recordings to Zendesk tickets to Twitter threads and makes it available for your team for analysis and for action. What makes Interpret unique is its ability to build and update a customer specific knowledge graph that provides the most granular and accurate categorization of all customer feedback and connects that customer feedback to critical metrics like revenue and CSAT. If modernizing your voice of customer program to a generational upgrade is a 2025 priority, like customer centric industry leaders like Canva, Notion, Perplexity, and Linear, reach out to the team at interpret dot com slash Lenny. That's e n t e r p r e t dot com slash Lenny. Going back to just how this all works and what you guys do for models, I was talking to a friend who had a ankle sprain or her... His foot was hurting, and he got an X-ray, and he fed the X-ray into Chachi PT and then asked him, like, give me this specific X-ray, and he's like, okay. Sure. And then it gave him here's what you have. And he was talking to me, is out there on the internet to train this model to know this stuff? I was like no, it's actually somebody sitting there helping the model understand this once they recognize it doesn't fully understand this. Like humans are actually helping them learn these things.
00:30:01 · Brendan Foody, Lenny Rachitsky
[00:30:01] S0: Exactly. Well, so the way it works, at least what most people's understanding is there's a lot of complexity in how the models work, that pre training gets a lot of the knowledge into the model of what are all the different things that sort of see it in the world, and then post training and reinforcement learning is for all of the reasoning of what are the pieces of knowledge that are accurate, what are inaccurate, and what to prioritize, at any given time to make a decision. And so behind that, there would have been radiologists that, worked on the post training data set to create some stasis point for here's the diagnosis and rewards and penalties associated with it. And it's really the quality of those people that went into the quality of the decision and recommendation that ChatGPT ultimately made. [00:30:46] S1: So let's actually follow that thread, because that's really interesting, and I don't know how many people understand it. I sort of understand it. So the work that you do and these experts do is post training. It's not feeding data into the model that it's trained on, it's, we have this model GPT-five, now here's all the things it's missing, let's add to it. [00:31:02] S0: Exactly, yeah. It's really unlocking, allowing the model to focus on all the right tokens from pre training all the right things in model context, upgrading the effective reasoning chains, to enable the models to reason better in a more generalized way.
00:31:02 · Brendan Foody, Lenny Rachitsky
[00:31:02] S0: Exactly, yeah. It's really unlocking, allowing the model to focus on all the right tokens from pre training all the right things in model context, upgrading the effective reasoning chains, to enable the models to reason better in a more generalized way. [00:31:19] S1: What's the scale of people just working on this stuff? Is it like thousands, tens of thousands, hundreds of thousands? [00:31:24] S0: Tens of thousands at any given time, hundreds of thousands more generally. I mean, it's huge. And the most exciting thing is that it's growing really quickly. I mean, I think that to your question also about the competitive landscape, historically there were all these crowdsourcing companies that would get these super high volumes of low skilled people. I think like Scale and Surge were the primary companies that pioneered that industry. And then in this transition to higher skilled labor, what people realized is that actually you can go a lot further with just getting higher caliber people, even in smaller amounts initially, and now subsequently scaling that back up once they're able to meet the quality bar. I think that there's a bunch of companies that after our success and very rapid revenue growth that sort of started early last year have chased after that, which makes sense, right? And seeing that the market was changing very quickly, we were taking off and trying to pursue a similar thesis on the market.
00:31:24 · Brendan Foody, Lenny Rachitsky
[00:31:24] S0: Tens of thousands at any given time, hundreds of thousands more generally. I mean, it's huge. And the most exciting thing is that it's growing really quickly. I mean, I think that to your question also about the competitive landscape, historically there were all these crowdsourcing companies that would get these super high volumes of low skilled people. I think like Scale and Surge were the primary companies that pioneered that industry. And then in this transition to higher skilled labor, what people realized is that actually you can go a lot further with just getting higher caliber people, even in smaller amounts initially, and now subsequently scaling that back up once they're able to meet the quality bar. I think that there's a bunch of companies that after our success and very rapid revenue growth that sort of started early last year have chased after that, which makes sense, right? And seeing that the market was changing very quickly, we were taking off and trying to pursue a similar thesis on the market. [00:32:27] S1: It's interesting. There's always been these companies, AlphaSight and, GLG, that like sort of did this before AI, or is like paid to connect to an expert and ask them questions about stuff. And essentially, okay, it turns out this is really useful for models. We don't need the person in the middle.
00:32:27 · Lenny Rachitsky, Brendan Foody
[00:32:27] S1: It's interesting. There's always been these companies, AlphaSight and, GLG, that like sort of did this before AI, or is like paid to connect to an expert and ask them questions about stuff. And essentially, okay, it turns out this is really useful for models. We don't need the person in the middle. [00:32:42] S0: Exactly. Yeah. Well, the one core difference is that alpha sites would generally be a one off call versus a lot of our work is really hiring people for projects, right? Of how do they work on something for a longer period of time? And so that's, I think, one of the reasons that some of the traditional expert networks have struggled to get into this. And also, how do you retain those people and think about all the incentives where it actually looks more similar in some ways to one of the traditional labor marketplaces of an Uber or DoorDash, just with much higher skilled talent that's treated exceptionally well. [00:33:22] S1: It's such a good opportunity for me to learn so much about this. I'm just going to ask questions. Yeah, sure. So interesting to me. How much of the experts are focused on specific concrete knowledge versus personality and like softer skills? How much of it is like, here's how you do an exam, here's how you do an x-ray?