Invest Like the Best with Patrick O'Shaughnessy
· Gokul Rajaram
So they sit with the engineers and the researchers and write code, do prototypes, do anything and everything it needs in a hands-on way. The first thing we are seeing now happen is that PMs are starting to check in code with either Codex or Claude Code, into the actual production repository. Right now, engineers have to review the code, but you're going to soon see that Claude Code, Codex, and other tools actually review the code itself before engineers commit. So all the companies are struggling with how to evaluate these people. Earlier, there was nothing called a prototyping interview. Now, there's an explicit interview, in the interview loop called prototyping. It literally forces product managers to be hands-on. Second, the product manager and designer role are merging increasingly. The designer role is an interesting role in particular. A lot of companies are going through headcount allocation this year, and I'm hearing from many teams that when given the choice between an extra designer and extra engineer, they're saying, "You know what, the design systems are already laid out. Now that we have the design systems already laid out, we can use AI to do work around these design systems. So we need maybe a small number of designers at the company level to manage the design systems and the design language, but AI can leverage the design language to do designs. So please give us an extra engineer." The number of designers and product managers relative to the number of engineers, when I was growing up in product, it used to be 1:3 or 1:10. It's going to 1:20 now. And then I think the other very, very important thing that's happened, which is fundamentally different is – when I was growing up, products were deterministic. Where there was a workflow, you knew if a user did X, Y happened. Today, you could do X, Y happens. But if you do slight variation of X, something completely different happens, non-deterministic software. What that means is, you have to be on the other side, an evaluation, or what is called evals in AI. And someone has to evaluate whether or not what the software is producing is reasonable or not across various use cases. Obviously, they can be human evals, AI evals, et cetera. But who owns the evals? It's the PMs. It's the PMs and the researchers.
· Gokul Rajaram, Patrick O'Shaughnessy
So the PM's job is to be very clear at a high level about what the user needs are, and then have a very clear sense of whether this product is good to ship or not by evaluating it. Many times you've got to write AI yourself to evaluate the results of AI, because humans can't. So PMs are good at coming up with evaluation techniques. It's the non-determinism of software, the speed of which things are going, and overall, the notion that the capability frontier is being pushed out every two months makes it an incredibly challenging, yet incredibly exciting time to be in product development. [00:00:00] S0: If you think about, my friend Zach has this great way of thinking about AI, which is, we had the industrial revolution for goods, and that basically this kicks off an industrial revolution for services. This is an interesting opportunity to ask about what your philosophy of product is. You're such a product-centric person and builder. That's what you've done. That's what you've invested in. As we face down this industrial revolution for services, what is your broadest possible philosophy of product as we enter this era?
· Gokul Rajaram
[00:00:00] S1: Very simple. A product person, or product manager if you call them, their job is to balance customer needs and business needs. The product manager, there has to be somebody at the company who's the keeper of the why. Why are we building it? What customer need are we solving? Why is this a pain point? How intense is it and how deep is it? And second, how does it add value to the company? If you build this thing, solving this customer need, how does the value add to the company? And I think balancing those two is a very delicate act. You can build something amazing that adds a tremendous amount of value to the customer, but doesn't build any value to the business. And you can do something that is awesome for the business by raising prices, but it is value detracting for the customer. So balancing customer needs and business needs at the highest level is what I think of the product. And what it comes down to, in my opinion, over the last 10 or 15 years, I've really gone down to this notion of outcomes. Outcomes, I think, are what define the best product people, and outcomes have to be defined in the form of customer behavior. Because customer behaviors are leading indicators for every business outcome. If you think about it, the simplest thing that a product does is to make somebody go from not a customer state to becoming a customer state. And from becoming a customer state to becoming a loyal customer. And then maybe to become a loyal customer to become a paying customer. Or if you do a poor job, you can go from becoming a loyal customer to becoming a churn customer. So these are all behaviors. Everything you do or build should be attuned to the goal of, what customer state change does it lead to? What customer behavior change does it lead to? So, I tell every CEO I meet that is trying to hire their first PM or doing their first product review; you need to ask why. The only question you need to ask is why. Why are you launching this feature? And you should not let any feature go out if there's not a clear hypothesis behind this feature. And the hypothesis has to be articulated in the form of a customer behavior change. We believe that by launching this thing, the customers will go from doing X to doing Y, or from spending X minutes a month doing this to Y minutes a month doing this.
· Gokul Rajaram, Patrick O'Shaughnessy
You have to have a hypothesis, which is grounded in some data or something you know about the customer, some secret about the customer. [00:00:00] S0: You mentioned at the start, the difference between the video transcriptions tool six months ago versus more recently and how quickly that changed. It's just such a hard future to reason about given the pace of change. So how do you reason about it? Is there anything that can be truly future-proof?
· Gokul Rajaram
[00:00:00] S1: Yes. The one thing I think that's going to be truly future-proof is judgment. Why? Because what is the biggest challenge you have when you have a thousand AI engineers writing code? You have the big challenge of AI slop. Every product director I’ve talked to is extremely worried that because you have these engineers running rampant, they're just going to produce lots of code. Which of this code is even valuable? In an era when you can do everything, the question is which of these things matter and you should truly do. On the product side is judgment around what needs to be built and evaluating the output. On the engineer side is evaluating the code, because if you don't understand what the code says, I think you can have AI engineers writing beautiful code that could be wrong, that could have bugs in it, that could be vulnerable. Someone needs to review it and make sure – you have to have human review at some point, especially the critical code that is in the core of your system. And similarly in design, you have to have judgment around, does it make sense in the broader design system? So I think this judgment is the number one thing that humans are going to bring. Era of infinite productivity, the question is what are the things we're productive on, and are we building the right things?
· Patrick O'Shaughnessy
[00:00:00] S0: As you evaluate companies today, build things yourself and just think about the trajectory of these tools. Maybe walk through how someone should think about building an AI application. There's so many people excited about… it feels like a gold rush with this new technology. So many things that we can do that we couldn't do before or things that specific people couldn't do because they weren't technical that they can now do. How should people think about building an application using AI starting today?