Entrepreneurial Thought Leaders (ETL)
00:07:25 · Michael Bernstein
Percy and I, who probably a lot of you have taken a class from one of the two of us if you're if you're in the CS program. We brought in Lainie Allen, who's our fourth cofounder, chief operating officer, and leads a lot of our our go to market side and commercial side. All four of these individuals... Well, I won't I won't speak for myself, but the other three of these individuals are incredible people, and I think just underscoring cofounders matter a lot. So what Simile does is it gives you access to essentially an on demand audience of any sort. So you can ask them any question. So, you know, if we approved your loan in sixty seconds, would you skip the dealer and get insights kind of immediately? Not just a quantitative estimate of whether that would be the case or not, but why it might be the case. We can we can reach into first party data collection. I'm I'm running data collection with and tens of thousands of people every week, understanding why they make the decisions they do, drawing and behavioral data, self report data, whatever we need to be able to give people access to the kinds of questions that they want. So we're pursuing things like, can we make this model as accurate as possible? How do we pull in more and more people to teach the model about the sort of the dark matter of human behavior? That's just not available on the open web. You can't just go with your pre training dataset and get it, and you'll pull in behavioral data and make it accessible to any idea that you have on a napkin sketch in the middle of the night. I literally have had senior senior executives using this thing while waiting in the line at Disneyland with their kids. So we work with customers and draw out all sorts of different methodologies, things that people are interested in asking, anything from these very strategic questions to, in some cases actually, a CEO asked us to simulate the investors who show up on their quarterly earnings calls. We nailed seven of the nine questions that they got asked on that earnings call. And they're across many different domains from entertainment to financial services to consumer packaged goods, tech companies, all over the place we are seeing this happen. You can see some public press around us, CVS being one customer, New York Times deal book talking about us.
00:10:02 · Michael Bernstein
This is again June, who, by the way, if you're interested, was a painter for a while before he switched into CS. So bring those other skills in. What people tell us about this is that stuff that they used to spend like ten weeks and hired a consulting company for becomes an afternoon, and it becomes iterative because they can not just get the answer, but then say, why did this happen, or what if I had done it this way instead? There's like... They're really amplifying the powers of, you know, their ability to to try things and and ask these what ifs. And so the use cases really cover a large, large, large variety ranging from very sort of prosaic, you know, on the ground questions around like messaging or pricing to really big strategic questions like, should we turn the entirety of Stanford into, you know, should we dramatically re envision the undergraduate experience, or should we change our product or become a drive through? That one happens. You know, like this is it's all really interesting. We are also trying to build out the future of simulation, moving from this notion that I can simulate a single person or a group to something that's eventually, can we simulate 8,000,000,000 people, an entire world of not just the people, but the environments around them, like understanding the sort of helping us understand the evolving complex emergent nature of, you know, a market. If I do this, how's my competitor going to respond? How is my... How is the customer going to respond to that? All these emergent interactional outcomes that are just so hard to make sense of in any other way. If we have an accurate model of human behavior, maybe we can. So that is a bit about Simile. I'm obviously happy to nerd out at any length about it, but they encouraged me to focus on like stories and reflections. As some of you may know, I was not at Stanford last year. I was not teaching. I was advising my my PhD students, that was about it. I was, you know, full time at Simile. Let me reflect on a few things that I've learned along the way. The first set of questions has to do with what I think AI is breaking about human centered design. It's not everything, but it's interesting. One thing I saw was that no matter what you're doing, people are going to have anxiety about whether this tool is going to displace their jobs.
00:12:40 · Michael Bernstein
If they are using it, if they are the user, they are at some level also wondering if this is eventually going to replace them. Even though your tool has no intent to do that and probably will not, your tool may be the first time that these stories that they've heard in the news about, you know, executives bringing in AI and then cutting teams, it might be the first time that anything AI ish is coming anywhere close to them, and so they will project their concerns about that onto you, and you need to be prepared to help them understand why your tool is going to be a superpower for them and not something that is just going to displace them. Even again, if you've done the best design in the world to ensure that this is in fact a superpower for them, it is not in some sense fully your fault that you are going to be fighting this battle, but you will. And I think you have to come to it with a sense of empathy and understanding that this may be a knee jerk reaction that some people have. Like they see the opportunity, they're excited, and then they also may have concerns. How do you convince people that this is something that will give them superpowers rather than something that they need to feel threatened by? I think there's no secret sauce here, but I think that one thing we've seen is, you know, our deployments team at Simile is so good. Like hiring these folks who who I get to work with, I have the privilege of working with, are working with these customers, talking to them, building trust over long periods of time. You do not just parachute in technology. Like they are doing the hard work of really building the trust, building the relationships, being there, and in many cases you know we've been told that that those folks on our team at Simile are like really differentiating. Like they are they are one of the reasons why people want to use our tool. This surprised me. I honestly had thought that if as long as it was clear that this is going to augment you, we're probably fine. It won't trigger the concerns. I no longer think this is the case because everyone has the first AI tool that comes somewhere near their job. Another reflection. At Stanford, I had been teaching for I don't know how many years, this notion of intelligence augmentation. If you've taken something like CS one forty seven, James Landay will drill into your head this like...
00:15:21 · Michael Bernstein
This gentleman named Doug Engelbart who won a Turing Award. He has this big idea called like intelligence augmentation. Take AI, replace the letters like swap them, IA, intelligence augmentation. It says stop trying to replace human labor with AI. Instead ask what... How AI can be like a strap on cortex. How can it make us smarter? How can it make us faster, more strategic? Right? It's like an amplifier. How can it amplify us? Like I said, how do you give us superpowers? Doug Engelbart, when he did that, like he was inventing interactive computing. He he actually did invent the mouse. That's why he won a... In part won a Turing Award. But this framing is still, I think, just as true today. When people think about AI, they often think about, oh, here's a thing that I do. I can get the AI to do that instead, and then I just replace it. That's a replacement story. That tends to go poorly. The thing that actually creates the entire new industry, the new genre, that's about new superpowers, And I think when I reflect on what has really made Simile work, it's that. It's like I'm not coming in and just replacing labor. I'm like, I'm enabling you for example to ask these like rapid iterative questions, these what ifs that you really couldn't have before, at least not at any sort of speed that would have been made it possible to reason about stuff. This one I didn't think... I wasn't sure if I was wrong and I was glad that I wasn't wrong. I'll say also from a design perspective, one thing that really breaks that I found I had to think about quite hard is how is your product going to absorb the hit when the model's prediction is not what people are expecting to see. Famously, ChatGPT couldn't count the letter... The number of r's in the word strawberry for for many years. Right? If you are building a product around AI, you have to assume from day one and build the whole product around an assumption that sooner or later someone's going to see something that they don't expect. This is not how traditional design works. You like, you think through all of the tasks and the flows and and it's like a beautifully crafted experience.
00:17:46 · Michael Bernstein
You can get your beautifully crafted experience, but if you do not have a crumple zone for how people are supposed to be able to reason about what happens when your model disagrees with what they think it should be doing, you're going to lose trust. And at least I can't speak to your domains, but at least with simulation, trust is the beginning and the end of our conversation with our users. If they trust our models, they will do all sorts of things. If they do not trust our models, or if they see something that threatens their trust in our model, or our ability to remediate that, we're done. We're just absolutely done. So every AI powered product I've seen, probably within the first five minutes of me using it, has done something wrong. And I think you should expect yours will too. How are you designing that product to recapture, regain trust once they just saw it do something that they didn't expect and strikes them as, oh, this isn't as magic as I thought it was going to be. In Simile, we have all sorts of things. If you, you know, you can read our... There's a blog we have online about it, a confidence model that we train on top of our our simulation that can give you... Lets you ask any question, but then gives you a green light, yellow light, red light in terms of how accurate our model is likely to be. If you ask, what kinds of cool spaghetti dinner are people gonna eat when they go to Stanford on the moon? Like, we'll make a we'll make a prediction, but that's gonna be a red light. Like, there's not a lot of training data that could really give us confidence in that. Whereas, there's other questions you might ask where, you know, green light, this is like right in the sweet spot. So we absorb some amount of that by just calling our shot. This is one where we expect our our accuracy to be lower. Again, happy to nerd out on how you do that, but it is quite a quite a thing you need to think through. At least we found we had to think through. We're training models. Right? We're... I I... We are training our own model. Benchmarks are great to show, but everyone just sort of nods through them. I could show you today our... Even our internal benchmarks at Simile, and it's not clear to me based on everything I've experienced that that gives you the conviction to use it or to, like, spend money on it for your company. It's like the individual experience.
00:20:27 · Michael Bernstein
Once someone gets in and can ask a question that they know the answer to, that's the thing that I find gets... Gives them the conviction that like, wow. In an early case, what happened with one company was we were we were demoing to them, and they just asked us live, hey, can you estimate our NPS? They're like, okay. You know, it's very early, so there's a little bit of like fingers crossed under the table. Nailed it. Our model nailed it, and suddenly that gained way more trust than any number of benchmarks we could have shown them do. Especially in the space of simulation where everyone seems to be like making up numbers. We're 85% accurate. We're 92% accurate. We're ninety seven ninety seven watts. Like none of us understand like, and your users don't have time to figure this out. So what they're going to do is take something that they know what it should do and prove to themselves that it that it can do it. This was definitely something that we are still thinking through. How do we build that trust as effectively and as efficiently as possible? So that's on the design side. Org building. I often describe Simile as a frontier lab with a product, And I feel pretty strongly convicted that you need this. You cannot... You're welcome to go use your clods and your chat GPTs to try to like estimate and simulate stuff. They do okay if what you're asking is sort of, like, relatively gen pops facts that are on the public web. Most questions that people actually care to ask, like, imagine Stanford's president is thinking about something for, like, the freshman experience or something. There's not a lot of stuff. There's this dark matter online that's not online about how... What shapes our attitudes and our behaviors as Stanford students, and so you need a model as well. You need to both be training these models and build the product around it. It's a blank piece of paper. How do we help you build a simulation that's accurate? How do you make sense of what you're looking at? We cannot just give you an MCP because you're not going to know how to how to how to, like, recognize and make sense of what the simulation is going give you So we have to be a frontier lab with a product. This makes, as I'll talk about in a moment, all sorts of interesting and complicated and fun for how you have to do your org design because everything becomes extremely cross functional.