Uncapped with Jack Altman
· Bret Taylor
[00:00:00] S1: Let’s say you had an AI agent to generate leads for your sales team. What do you care about? You care about the number and quality of the leads. You really don’t care how many tokens the model uses. In fact, it’s not obvious to me that there’s a correlation between used tokens and leads generated. In the same way, there’s no correlation in a SaaS product between their cost to serve and the quality of the product. You could have a really good engineer write it or a really bad engineer write it. What you really care about is the quality of the product. The reason why I don’t think token-based makes sense is that it’s charging for an input that is uncorrelated with the output that your clients actually care about. I’m a huge believer in applied AI, but I actually define applied AI as: can you describe your value proposition without mentioning models? Because if you think about, “Hey, we can answer the phone and solve 80% of phone calls without human intervention, with a CSAT score of 4.8 out of five”, you don’t mention models. Models are an input to that, not an output. If you have to mention token utilization, it’s probably a tool. It’s not an application of AI. It’s just a tool around AI. The closer you get to a business outcome, you should charge for the business outcome, which is uncorrelated with tokens. I also think it’s almost a measure of whether you’re actually an applied AI company, if you don’t have to talk about tokens.
· Jack Altman
[00:00:00] S0: Do you think there will be markets where things get so competitive that people have to price based on cost rather than value? Or maybe the other format would be where you can’t describe the outcome cleanly. For example with coding, which we both probably think is super important, it’s a little harder to say what the outcome is there versus usage. What are the conditions where tokens do make sense?
· Bret Taylor
[00:00:00] S1: There’s this old Apple site where they had Apple folklore. There was this one boss at Apple that made people fill out a form saying, “How many lines of code did you write?” This engineer infamously wrote a negative number because he had just refactored a bunch of stuff. It’s the good historical analogy for why tokens don’t matter. It was his way of saying, “Fuck the man. Lines of code have nothing to do with my value”, and he was doing it to piss off a middle manager to make that point. It’s interesting because in the world of software engineering, the customers of coding agents right now are software engineers who intimately understand these models. So there’s a bit of a customer-product-market fit. So it’s a nuanced point. But I’ll say where I see it might happen. Right now, if you’re evaluating a coding agent, you’re probably comparing it to the cost of a software engineer. If you fast forward five years, you’ll probably be comparing it to the cost of other coding agents. So I think the second-order effect as AI becomes prevalent is that the reference point for its value will change. The thing I would say is that’s true where you’re thinking about a cost center. If you’re thinking about top-line revenue growth, that doesn’t necessarily apply. In my example of an AI agent generating leads for your sales team, depending on what you’re selling, a lead is a lead is a lead. You’ll probably value quantity and quality of leads, and there’s a math equation. That probably will remain independent of token costs, is my guess. A large part of AI is productivity and reducing costs, and there’s a big part of it. But the other side of it is outcomes. Could you imagine a world in four or five years where there’s one coding agent that can actually produce something of greater value for your company? Will you value that? Or will you just look at the token cost? I think you’ll probably start looking for value. Will they all be the same? I don’t know. I was just reflecting on the past year. There have been all these articles about whether AI progress has slowed down. In our world of software engineering, it’s been the opposite. Every new model comes out and you’re like, “Oh my gosh, it can write increasingly complex software.” My theory of that is that it depends on what you’re testing.
· Bret Taylor, Jack Altman
If you’re using ChatGPT for trip planning, you probably haven’t seen a material change over the past year and a half because you reached sufficient intelligence for trip planning a long time ago. If you’re using AI to write Rust code, Codex is mind-blowing right now. So one of the interesting things when I think about second and third-order effects and the progress of AI is where you pass the horizon where every model is sufficient in that task. Then there’ll be some things where the frontier continues to move. It’s hard to imagine, but we’re in a crazy time. [00:00:00] S0: Where are we at with support agents right now? Are there still edge cases, last-mile things that AI can’t do still?
· Bret Taylor
[00:00:00] S1: I imagine a lot of the technical problems, as opposed to product problems, will become easier. But there’s still a lot of them. We at Sierra support most spoken languages in the world. If you want to support Cantonese and Tagalog, most of the good voice models don’t come from the traditional Western model companies. One of our clients is Safelite AutoGlass, roadside assistance. It turns out that car noise, background noise, kids talking in the background, are actually all fairly hard problems to solve. Even in some of the advanced voice mode stuff, if you are in a noisy environment, it constantly thinks it’s being interrupted. So you end up having to build proprietary voice activity detection, multiple speaker detection, all these other things. We develop all this technology because we need to be the best now and I think we are the best now. You’re like, “Okay, that’s probably going to be a commodity two years from now, one year from now.” Who knows? But you have to do it because you need to be the best at every stage of your company’s existence. The way we think about the world is we have a product called Agent Studio, Agent OS. In three years you’ll judge us by our product. Right now, our clients don’t really put it this way, but we’re judged by the technology. If you go back to 1996, I remember when Netscape had a web server and Apache was new. No one cares how you serve webpages now. It’s a commodity. But at the time, that was what you sold. Now you have increasingly higher-order website building like Shopify. I just think the AI agent market’s going to take that progression. We’re going from a tech-centric sales cycle to a product-centric sales cycle.
· Jack Altman, Bret Taylor
[00:00:00] S0: It’s interesting that you’re obviously having to be the best at something that you know is going to get commoditized. I don’t know if you ever had to experience something like that. For that to be true, you just have to be in the middle of an insane rate of change. But that means you have teams who are putting a lot of their life force for two years into something that everybody knows is just for two years. But it still matters nonetheless. [00:00:00] S1: It’s crazy. If you look at traditional enterprise software—consumer’s a little different—you think about building up this asset, your intellectual property. There’s a fancy name for it. “Look at this platform that we’re building. It took so many years to build it. It’s got all these features. Now you’re like, “I’m building this and I’m a hundred percent certain we’ll throw it away in the next 48 months.” [00:00:00] S0: It’s a sand castle.