Uncapped with Jack Altman
· Bret Taylor, Jack Altman
[00:00:00] S1: It’s a Fortune 20 healthcare company. I was on stage with Sachin, who runs their AI practice, at the HLTH conference, and he was talking about this. Part of that is, how can you show up? We’re really great at AI, Cigna’s really great at healthcare. How do you bring those two together to move extremely fast? So for a lot of our clients, the reason they bring us on is, can you help us move quickly? That requires knowledge of AI, knowledge of business. I think we show up with a greater sense of maturity there. [00:00:00] S0: You mentioned that the pricing scheme was one of the difficult things in the past era. We don’t have to belabor it but obviously, going from just buying a license to a cloud subscription… and now usage-based is the future. What are you feeling is important as you’ve created, and probably continue to iterate, on pricing? What are the important levers for agent companies? [00:00:00] S1: We do something specific at Sierra that I’m an evangelist for, which is outcomes-based pricing. It turns out in our industry, the outcome is usually well-defined. In a service context, the outcome is: could the agent solve the problem? In a sales context—we do a lot of sales agents as well—could it make the sale? Your companies paid your salespeople commissions, right? [00:00:00] S0: Yeah.
· Jack Altman, Bret Taylor
[00:00:00] S0: Yeah. [00:00:00] S1: If you can measure the outcome, you want to incentivize the outcome. The interesting thing about agents is they’re autonomous, or can be autonomous. So if the outcome is measurable and trackable, what an interesting opportunity to actually charge for that. Look at the history of software. Let’s take advertising. We went from impression-based ads to cost-per-click ads to now, for mobile ads, you can do pay-per-install. At least that’s my understanding. Then you had enterprise software where you went from on-premises licenses to subscription-based software. Could outcome-based software be the next? What’s so neat about that is, for a company, what an interesting and accountable business model. There are some challenges to it because you obviously put some revenue at risk. But I don’t think most advertising tech people would say CPC ads put revenue at risk. It’s the opposite. The closer you get to the outcome, the more valuable it is for the companies. They’re actually willing to invest in it. My view is that, to the degree agents have a measurable outcome, outcome-based pricing feels like the secular business model for agents. I think it’s both disruptive and a huge step forward. [00:00:00] S0: Why is it better than token-based? If those are the two reasonable options now, why is an outcome better than token-based, even over the long term?
· Bret Taylor
[00:00:00] S1: Let’s say you had an AI agent to generate leads for your sales team. What do you care about? You care about the number and quality of the leads. You really don’t care how many tokens the model uses. In fact, it’s not obvious to me that there’s a correlation between used tokens and leads generated. In the same way, there’s no correlation in a SaaS product between their cost to serve and the quality of the product. You could have a really good engineer write it or a really bad engineer write it. What you really care about is the quality of the product. The reason why I don’t think token-based makes sense is that it’s charging for an input that is uncorrelated with the output that your clients actually care about. I’m a huge believer in applied AI, but I actually define applied AI as: can you describe your value proposition without mentioning models? Because if you think about, “Hey, we can answer the phone and solve 80% of phone calls without human intervention, with a CSAT score of 4.8 out of five”, you don’t mention models. Models are an input to that, not an output. If you have to mention token utilization, it’s probably a tool. It’s not an application of AI. It’s just a tool around AI. The closer you get to a business outcome, you should charge for the business outcome, which is uncorrelated with tokens. I also think it’s almost a measure of whether you’re actually an applied AI company, if you don’t have to talk about tokens.
· Jack Altman
[00:00:00] S0: Do you think there will be markets where things get so competitive that people have to price based on cost rather than value? Or maybe the other format would be where you can’t describe the outcome cleanly. For example with coding, which we both probably think is super important, it’s a little harder to say what the outcome is there versus usage. What are the conditions where tokens do make sense?
· Bret Taylor
[00:00:00] S1: There’s this old Apple site where they had Apple folklore. There was this one boss at Apple that made people fill out a form saying, “How many lines of code did you write?” This engineer infamously wrote a negative number because he had just refactored a bunch of stuff. It’s the good historical analogy for why tokens don’t matter. It was his way of saying, “Fuck the man. Lines of code have nothing to do with my value”, and he was doing it to piss off a middle manager to make that point. It’s interesting because in the world of software engineering, the customers of coding agents right now are software engineers who intimately understand these models. So there’s a bit of a customer-product-market fit. So it’s a nuanced point. But I’ll say where I see it might happen. Right now, if you’re evaluating a coding agent, you’re probably comparing it to the cost of a software engineer. If you fast forward five years, you’ll probably be comparing it to the cost of other coding agents. So I think the second-order effect as AI becomes prevalent is that the reference point for its value will change. The thing I would say is that’s true where you’re thinking about a cost center. If you’re thinking about top-line revenue growth, that doesn’t necessarily apply. In my example of an AI agent generating leads for your sales team, depending on what you’re selling, a lead is a lead is a lead. You’ll probably value quantity and quality of leads, and there’s a math equation. That probably will remain independent of token costs, is my guess. A large part of AI is productivity and reducing costs, and there’s a big part of it. But the other side of it is outcomes. Could you imagine a world in four or five years where there’s one coding agent that can actually produce something of greater value for your company? Will you value that? Or will you just look at the token cost? I think you’ll probably start looking for value. Will they all be the same? I don’t know. I was just reflecting on the past year. There have been all these articles about whether AI progress has slowed down. In our world of software engineering, it’s been the opposite. Every new model comes out and you’re like, “Oh my gosh, it can write increasingly complex software.” My theory of that is that it depends on what you’re testing.
· Bret Taylor, Jack Altman
If you’re using ChatGPT for trip planning, you probably haven’t seen a material change over the past year and a half because you reached sufficient intelligence for trip planning a long time ago. If you’re using AI to write Rust code, Codex is mind-blowing right now. So one of the interesting things when I think about second and third-order effects and the progress of AI is where you pass the horizon where every model is sufficient in that task. Then there’ll be some things where the frontier continues to move. It’s hard to imagine, but we’re in a crazy time. [00:00:00] S0: Where are we at with support agents right now? Are there still edge cases, last-mile things that AI can’t do still?