Lenny's Podcast: Product | Career | Growth
00:25:45 · Brendan Foody
[00:25:45] S0: Yeah, it's interesting. I remember when we started the company, as I mentioned, we were 19 and just had this gut intuition that it felt so wildly inefficient that labor markets are so disaggregated. And what I mean by that is when we would hire someone internationally, they would apply to a dozen jobs. When we, as a company in the Bay Area, were considering candidates, we would consider a fraction of a percent of candidates that were available in the market. And the reason for that is that there is this matching problem that everyone's solving manually, where they'll manually review resumes, they'll manually conduct interviews, and manually decide who to hire. But when we're able to automate that matching problem at the cost of software, it makes way for this global unified labor market that every candidate applies to and every company hires from facilitating a perfect flow of information in the economy. And I think that that future is undoubtedly what we're heading towards. But what we've realized over time is that the nature of work is also changing dramatically. And part of building that future over a ten year time horizon creating that future of work. And all of the more tactical things we do and building these incredible data sets across evals and RL environments for our customers.
00:27:06 · Lenny Rachitsky, Brendan Foody
[00:27:06] S1: What I've seen in how hiring has changed, I'm doing research on this with a partner Gnome, it s so much easier to apply for companies that everyone is just applying out to hundreds of companies. AI is just making it easy to adjust their resumes and cover letters and make it feel like, oh I applied to more courses very specifically, but it was one of 100 places. And then on the flip side, hiring managers are getting flooded with applications. And so now they need AI to filter. So even if we didn't want to get to this place, we're almost being pushed into this direction of so much volume on both sides, and we need something really smart at filtering and helping us hire and select. And this is exactly what you guys have been building for a long time. [00:27:45] S0: Precisely. Yeah. And the fascinating thing, like a lot of people ask, are we, do we think about ourselves as a labor marketplace or do we think about ourselves as a data company? And I think that the reason it's an interesting question is our realization from what the labs need is that they actually need a labor marketplace. They actually need these exceptionally high caliber people. And of course, we'll layer on some project management and some software platform associated with it. But the really core thing that they want is how do they find these extraordinary professionals across all of these different domains that can measure model capabilities and work to build that future work together.
00:28:28 · Lenny Rachitsky
[00:28:28] S1: This episode is brought to you by Interpret. Interpret is a customer intelligence platform used by leading CXN product orgs like Canva, Notion, Perplexity, Strava, Hinge, and Linear to leverage the voice of the customer and build best in class products. Interpret unifies all customer conversations in real time from Gong recordings to Zendesk tickets to Twitter threads and makes it available for your team for analysis and for action. What makes Interpret unique is its ability to build and update a customer specific knowledge graph that provides the most granular and accurate categorization of all customer feedback and connects that customer feedback to critical metrics like revenue and CSAT. If modernizing your voice of customer program to a generational upgrade is a 2025 priority, like customer centric industry leaders like Canva, Notion, Perplexity, and Linear, reach out to the team at interpret dot com slash Lenny. That's e n t e r p r e t dot com slash Lenny. Going back to just how this all works and what you guys do for models, I was talking to a friend who had a ankle sprain or her... His foot was hurting, and he got an X-ray, and he fed the X-ray into Chachi PT and then asked him, like, give me this specific X-ray, and he's like, okay. Sure. And then it gave him here's what you have. And he was talking to me, is out there on the internet to train this model to know this stuff? I was like no, it's actually somebody sitting there helping the model understand this once they recognize it doesn't fully understand this. Like humans are actually helping them learn these things.
00:30:01 · Brendan Foody, Lenny Rachitsky
[00:30:01] S0: Exactly. Well, so the way it works, at least what most people's understanding is there's a lot of complexity in how the models work, that pre training gets a lot of the knowledge into the model of what are all the different things that sort of see it in the world, and then post training and reinforcement learning is for all of the reasoning of what are the pieces of knowledge that are accurate, what are inaccurate, and what to prioritize, at any given time to make a decision. And so behind that, there would have been radiologists that, worked on the post training data set to create some stasis point for here's the diagnosis and rewards and penalties associated with it. And it's really the quality of those people that went into the quality of the decision and recommendation that ChatGPT ultimately made. [00:30:46] S1: So let's actually follow that thread, because that's really interesting, and I don't know how many people understand it. I sort of understand it. So the work that you do and these experts do is post training. It's not feeding data into the model that it's trained on, it's, we have this model GPT-five, now here's all the things it's missing, let's add to it. [00:31:02] S0: Exactly, yeah. It's really unlocking, allowing the model to focus on all the right tokens from pre training all the right things in model context, upgrading the effective reasoning chains, to enable the models to reason better in a more generalized way.
00:31:02 · Brendan Foody, Lenny Rachitsky
[00:31:02] S0: Exactly, yeah. It's really unlocking, allowing the model to focus on all the right tokens from pre training all the right things in model context, upgrading the effective reasoning chains, to enable the models to reason better in a more generalized way. [00:31:19] S1: What's the scale of people just working on this stuff? Is it like thousands, tens of thousands, hundreds of thousands? [00:31:24] S0: Tens of thousands at any given time, hundreds of thousands more generally. I mean, it's huge. And the most exciting thing is that it's growing really quickly. I mean, I think that to your question also about the competitive landscape, historically there were all these crowdsourcing companies that would get these super high volumes of low skilled people. I think like Scale and Surge were the primary companies that pioneered that industry. And then in this transition to higher skilled labor, what people realized is that actually you can go a lot further with just getting higher caliber people, even in smaller amounts initially, and now subsequently scaling that back up once they're able to meet the quality bar. I think that there's a bunch of companies that after our success and very rapid revenue growth that sort of started early last year have chased after that, which makes sense, right? And seeing that the market was changing very quickly, we were taking off and trying to pursue a similar thesis on the market.
00:31:24 · Brendan Foody, Lenny Rachitsky
[00:31:24] S0: Tens of thousands at any given time, hundreds of thousands more generally. I mean, it's huge. And the most exciting thing is that it's growing really quickly. I mean, I think that to your question also about the competitive landscape, historically there were all these crowdsourcing companies that would get these super high volumes of low skilled people. I think like Scale and Surge were the primary companies that pioneered that industry. And then in this transition to higher skilled labor, what people realized is that actually you can go a lot further with just getting higher caliber people, even in smaller amounts initially, and now subsequently scaling that back up once they're able to meet the quality bar. I think that there's a bunch of companies that after our success and very rapid revenue growth that sort of started early last year have chased after that, which makes sense, right? And seeing that the market was changing very quickly, we were taking off and trying to pursue a similar thesis on the market. [00:32:27] S1: It's interesting. There's always been these companies, AlphaSight and, GLG, that like sort of did this before AI, or is like paid to connect to an expert and ask them questions about stuff. And essentially, okay, it turns out this is really useful for models. We don't need the person in the middle.