· David Senra
[00:00:00] S0: It's more than that. It's like, "Zuck cannot win AI." It's said over and over again for a long period of time. And again, going back to Jeremy Stern's profile on him, he then, off the record, starts talking to his competitors, including one former researcher that is now at a competitor that used to work with Zuck and is now at another lab. And he goes, "I don't want to have to compete with him because Mark is like the fucking..." I think the line is, "Mark is the fucking terminator." And it was like, he just will not stop. And then he talks about one of his superpowers is that he's never happy with the state of the product, so he always wants constant improvement, but he's not an asshole. Like, he's not a dict- he's definitely a dictator, but he's not rude and will destroy the chemistry of a team. He actually talks about in the piece that team cohesion is excessively important to him.
· Alexandr Wang, David Senra
[00:00:00] S1: I think that's all right. I think one of the magical things about Muse was that we really took quite a bit of time to sweat all the details before ultimately coming out with the product. Because I think that personal agents was one of these product areas that kind of rested on a knife's edge, which is if it can reliably deliver these magical experiences, it has the potential to be one of the greatest consumer products ever. But you're dealing with models, you're dealing with something non-deterministic, you're dealing with agents. They can break, they can be unreliable. Like, if people have these unreliable experiences, they think it's trash. It was one of these things where getting to the product to a point of quality and value such that it resonated with so many of the people who downloaded and used it. [00:00:00] S0: I want to talk more about this process. I'm curious how much resources, how big the team, how many people are working on it. Do you guys have any data on like, "Okay, I'm using Muse, nine great experiences, two inconsistent or shitty experiences, and then I churn out?" Like, do you have any information on this?
· Alexandr Wang, David Senra
[00:00:00] S1: You can see this from a lot of the other agent products at the start of the year. Like, almost everyone churned out of those, and it's because they were unreliable. We did a lot of user testing of Muse. Like, we did many, many rounds of new people trying to download it and use it from scratch, and just seeing what that experience was like. A huge part of the premise of Muse, and I think what's really landed, is that it is a different thing from what most people have experienced with consumer AI. Like, it is not a chatbot, it is an agent, and it can do agent things for you. And I think to a developer, this is kind of like old news because they've had to download agents and have been using them for a while. But for most people, just even that graduation process, it needs to be explained and needs to be onboarded on a very thoughtful way. And then for sure, we definitely have seen when... Because we've done tests with different versions of the model and various model versions and continuously do those A/B tests. The product experience is very, very sensitive to model quality. It's very sensitive to how reliably the models can undergo these tasks, how reliably they communicate with the user about them, how reliably they check back with the user if there's things that they require guidance on. [00:00:00] S0: But how'd you know it was ready to be released?
· David Senra, Alexandr Wang
[00:00:00] S0: But how'd you know it was ready to be released? [00:00:00] S1: Part of it was pretty numerical, which is like we had set... Go back to that spreadsheet of 100 plus things. We had set for each of those rows what threshold is launch blocking and above what threshold is good. We had many checkpoints that were green on 80 of the rows, but red on 20 of the rows. We knew that wasn't good enough. And we had our first model checkpoint, which was green across the board, which was a specially trained version of Muse Spark 1.3. So there was the very numerical part of it, and then there was just the experiential part, which is we tried the model. We could tell that it was a lot better than anything we had iterated with before. [00:00:00] S0: So going back to that seven-month period you were talking about, how many people were working on the product? [00:00:00] S1: Between the model work and the product work, like under 200 people total, which I guess sounds like a lot, but in big company land is- [00:00:00] S0: It's actually a lot smaller.
· David Senra, Alexandr Wang
[00:00:00] S0: It's actually a lot smaller. [00:00:00] S1: It's not very many. It was kind of cool looking back on this and thinking about it is like, we knew this was going to be the most important product that... The whole executive team, everyone knew that this was probably going to be one of the most important products that Meta would ship this year. But we had the restraint organizationally, I think, to make sure that it was a really small, focused team working on it, and that it didn't balloon to this like, "Oh, you have huge swaths of the company working on it," because we knew it was a single work of art. Like, it required strong point of view and strong focus to pull that through into one beautiful experience. [00:00:00] S0: Say more about this. Why did you just call it a single piece of art and that you need a single perspective or point of view?
· David Senra, Alexandr Wang
[00:00:00] S0: Say more about this. Why did you just call it a single piece of art and that you need a single perspective or point of view? [00:00:00] S1: I think one criticism of products that come from larger organizations, and I think this is even now starting to plague a lot of the AI labs, is that they become kind of this Frankenstein, almost like Cronenberg amalgamation of a bunch of different people's points of view and visions and beliefs. And it's sort of like PM hell in some sense, where every PM has a thing that they're trying to get into the product and jam into the product, and then all of that kind of blends together into this smoothie almost. People feel it as users and consumers. You can feel when a product is just like, it feels like lots of different people worked on it and there were clearly different people who were goaled differently, so they just kind of jammed it all together. Muse had to be the exact opposite, which is like, there's one clear point of view. There is one cohesive experience that we're trying to deliver to the world.