Uncapped with Jack Altman
· Jack Altman, Bret Taylor
[00:00:00] S0: Yeah. [00:00:00] S1: If you can measure the outcome, you want to incentivize the outcome. The interesting thing about agents is they’re autonomous, or can be autonomous. So if the outcome is measurable and trackable, what an interesting opportunity to actually charge for that. Look at the history of software. Let’s take advertising. We went from impression-based ads to cost-per-click ads to now, for mobile ads, you can do pay-per-install. At least that’s my understanding. Then you had enterprise software where you went from on-premises licenses to subscription-based software. Could outcome-based software be the next? What’s so neat about that is, for a company, what an interesting and accountable business model. There are some challenges to it because you obviously put some revenue at risk. But I don’t think most advertising tech people would say CPC ads put revenue at risk. It’s the opposite. The closer you get to the outcome, the more valuable it is for the companies. They’re actually willing to invest in it. My view is that, to the degree agents have a measurable outcome, outcome-based pricing feels like the secular business model for agents. I think it’s both disruptive and a huge step forward. [00:00:00] S0: Why is it better than token-based? If those are the two reasonable options now, why is an outcome better than token-based, even over the long term?
· Bret Taylor
[00:00:00] S1: Let’s say you had an AI agent to generate leads for your sales team. What do you care about? You care about the number and quality of the leads. You really don’t care how many tokens the model uses. In fact, it’s not obvious to me that there’s a correlation between used tokens and leads generated. In the same way, there’s no correlation in a SaaS product between their cost to serve and the quality of the product. You could have a really good engineer write it or a really bad engineer write it. What you really care about is the quality of the product. The reason why I don’t think token-based makes sense is that it’s charging for an input that is uncorrelated with the output that your clients actually care about. I’m a huge believer in applied AI, but I actually define applied AI as: can you describe your value proposition without mentioning models? Because if you think about, “Hey, we can answer the phone and solve 80% of phone calls without human intervention, with a CSAT score of 4.8 out of five”, you don’t mention models. Models are an input to that, not an output. If you have to mention token utilization, it’s probably a tool. It’s not an application of AI. It’s just a tool around AI. The closer you get to a business outcome, you should charge for the business outcome, which is uncorrelated with tokens. I also think it’s almost a measure of whether you’re actually an applied AI company, if you don’t have to talk about tokens.
· Jack Altman
[00:00:00] S0: Do you think there will be markets where things get so competitive that people have to price based on cost rather than value? Or maybe the other format would be where you can’t describe the outcome cleanly. For example with coding, which we both probably think is super important, it’s a little harder to say what the outcome is there versus usage. What are the conditions where tokens do make sense?
· Bret Taylor
[00:00:00] S1: There’s this old Apple site where they had Apple folklore. There was this one boss at Apple that made people fill out a form saying, “How many lines of code did you write?” This engineer infamously wrote a negative number because he had just refactored a bunch of stuff. It’s the good historical analogy for why tokens don’t matter. It was his way of saying, “Fuck the man. Lines of code have nothing to do with my value”, and he was doing it to piss off a middle manager to make that point. It’s interesting because in the world of software engineering, the customers of coding agents right now are software engineers who intimately understand these models. So there’s a bit of a customer-product-market fit. So it’s a nuanced point. But I’ll say where I see it might happen. Right now, if you’re evaluating a coding agent, you’re probably comparing it to the cost of a software engineer. If you fast forward five years, you’ll probably be comparing it to the cost of other coding agents. So I think the second-order effect as AI becomes prevalent is that the reference point for its value will change. The thing I would say is that’s true where you’re thinking about a cost center. If you’re thinking about top-line revenue growth, that doesn’t necessarily apply. In my example of an AI agent generating leads for your sales team, depending on what you’re selling, a lead is a lead is a lead. You’ll probably value quantity and quality of leads, and there’s a math equation. That probably will remain independent of token costs, is my guess. A large part of AI is productivity and reducing costs, and there’s a big part of it. But the other side of it is outcomes. Could you imagine a world in four or five years where there’s one coding agent that can actually produce something of greater value for your company? Will you value that? Or will you just look at the token cost? I think you’ll probably start looking for value. Will they all be the same? I don’t know. I was just reflecting on the past year. There have been all these articles about whether AI progress has slowed down. In our world of software engineering, it’s been the opposite. Every new model comes out and you’re like, “Oh my gosh, it can write increasingly complex software.” My theory of that is that it depends on what you’re testing.
· Bret Taylor, Jack Altman
If you’re using ChatGPT for trip planning, you probably haven’t seen a material change over the past year and a half because you reached sufficient intelligence for trip planning a long time ago. If you’re using AI to write Rust code, Codex is mind-blowing right now. So one of the interesting things when I think about second and third-order effects and the progress of AI is where you pass the horizon where every model is sufficient in that task. Then there’ll be some things where the frontier continues to move. It’s hard to imagine, but we’re in a crazy time. [00:00:00] S0: Where are we at with support agents right now? Are there still edge cases, last-mile things that AI can’t do still?
· Bret Taylor
[00:00:00] S1: I imagine a lot of the technical problems, as opposed to product problems, will become easier. But there’s still a lot of them. We at Sierra support most spoken languages in the world. If you want to support Cantonese and Tagalog, most of the good voice models don’t come from the traditional Western model companies. One of our clients is Safelite AutoGlass, roadside assistance. It turns out that car noise, background noise, kids talking in the background, are actually all fairly hard problems to solve. Even in some of the advanced voice mode stuff, if you are in a noisy environment, it constantly thinks it’s being interrupted. So you end up having to build proprietary voice activity detection, multiple speaker detection, all these other things. We develop all this technology because we need to be the best now and I think we are the best now. You’re like, “Okay, that’s probably going to be a commodity two years from now, one year from now.” Who knows? But you have to do it because you need to be the best at every stage of your company’s existence. The way we think about the world is we have a product called Agent Studio, Agent OS. In three years you’ll judge us by our product. Right now, our clients don’t really put it this way, but we’re judged by the technology. If you go back to 1996, I remember when Netscape had a web server and Apache was new. No one cares how you serve webpages now. It’s a commodity. But at the time, that was what you sold. Now you have increasingly higher-order website building like Shopify. I just think the AI agent market’s going to take that progression. We’re going from a tech-centric sales cycle to a product-centric sales cycle.