Entrepreneurial Thought Leaders (ETL)
00:17:46 · Michael Bernstein
You can get your beautifully crafted experience, but if you do not have a crumple zone for how people are supposed to be able to reason about what happens when your model disagrees with what they think it should be doing, you're going to lose trust. And at least I can't speak to your domains, but at least with simulation, trust is the beginning and the end of our conversation with our users. If they trust our models, they will do all sorts of things. If they do not trust our models, or if they see something that threatens their trust in our model, or our ability to remediate that, we're done. We're just absolutely done. So every AI powered product I've seen, probably within the first five minutes of me using it, has done something wrong. And I think you should expect yours will too. How are you designing that product to recapture, regain trust once they just saw it do something that they didn't expect and strikes them as, oh, this isn't as magic as I thought it was going to be. In Simile, we have all sorts of things. If you, you know, you can read our... There's a blog we have online about it, a confidence model that we train on top of our our simulation that can give you... Lets you ask any question, but then gives you a green light, yellow light, red light in terms of how accurate our model is likely to be. If you ask, what kinds of cool spaghetti dinner are people gonna eat when they go to Stanford on the moon? Like, we'll make a we'll make a prediction, but that's gonna be a red light. Like, there's not a lot of training data that could really give us confidence in that. Whereas, there's other questions you might ask where, you know, green light, this is like right in the sweet spot. So we absorb some amount of that by just calling our shot. This is one where we expect our our accuracy to be lower. Again, happy to nerd out on how you do that, but it is quite a quite a thing you need to think through. At least we found we had to think through. We're training models. Right? We're... I I... We are training our own model. Benchmarks are great to show, but everyone just sort of nods through them. I could show you today our... Even our internal benchmarks at Simile, and it's not clear to me based on everything I've experienced that that gives you the conviction to use it or to, like, spend money on it for your company. It's like the individual experience.
00:20:27 · Michael Bernstein
Once someone gets in and can ask a question that they know the answer to, that's the thing that I find gets... Gives them the conviction that like, wow. In an early case, what happened with one company was we were we were demoing to them, and they just asked us live, hey, can you estimate our NPS? They're like, okay. You know, it's very early, so there's a little bit of like fingers crossed under the table. Nailed it. Our model nailed it, and suddenly that gained way more trust than any number of benchmarks we could have shown them do. Especially in the space of simulation where everyone seems to be like making up numbers. We're 85% accurate. We're 92% accurate. We're ninety seven ninety seven watts. Like none of us understand like, and your users don't have time to figure this out. So what they're going to do is take something that they know what it should do and prove to themselves that it that it can do it. This was definitely something that we are still thinking through. How do we build that trust as effectively and as efficiently as possible? So that's on the design side. Org building. I often describe Simile as a frontier lab with a product, And I feel pretty strongly convicted that you need this. You cannot... You're welcome to go use your clods and your chat GPTs to try to like estimate and simulate stuff. They do okay if what you're asking is sort of, like, relatively gen pops facts that are on the public web. Most questions that people actually care to ask, like, imagine Stanford's president is thinking about something for, like, the freshman experience or something. There's not a lot of stuff. There's this dark matter online that's not online about how... What shapes our attitudes and our behaviors as Stanford students, and so you need a model as well. You need to both be training these models and build the product around it. It's a blank piece of paper. How do we help you build a simulation that's accurate? How do you make sense of what you're looking at? We cannot just give you an MCP because you're not going to know how to how to how to, like, recognize and make sense of what the simulation is going give you So we have to be a frontier lab with a product. This makes, as I'll talk about in a moment, all sorts of interesting and complicated and fun for how you have to do your org design because everything becomes extremely cross functional.
00:23:09 · Michael Bernstein
But it does raise the question, like, what's actually a moat these days? I'm curious how many of you have been wondering that recently. Yeah. Now to own my bias, I lead a lot of our... I I lead our data efforts at Simile. So, like, data, I feel pretty good about as a moat. But there are these questions of, like, what will the Frontier Labs sort of capture by means of just their general improvement, racing toward AGI? There's this question of what... Like, what you can do that's durable. Personally, my take is that I think this is a fascinating area that is... That because it requires bespoke data is not something that I think a frontier lab is just gonna automatically get better at. It just doesn't have... The data is not out there nor can it be purchased. But I think if all you do is have an API, then you get into these interesting challenges where there's nothing differentiating. And so I actually think that having the data, the product, these things all mutually reinforcing really becomes critical. Because as I saw, the further our company got and the more successful, the more competitors really start to show up. It feels like every week at this point, like I'm hearing about a new simulation company, and you have to be extremely highly convicted in her approach to be feeling like, you know what? At some point, even if this is so big that the eye of Mordor starts to turn over toward you and pay attention, that you've got what it's going to take to make it not worth it for them to just absorb it along the way. So my answer here, again, my bias, data really matters. Having, you know, identifying modeling techniques that make sense for my task but don't make sense for, you know, math proofs also important, and having novel product does matter. But remember that products can now be built very quickly, so it can't just be the product that you wrapped around an LLM's API. I sometimes refer to us as currently being in what I what I think of as the Myspace era for AI. I've learned that I'm old. I'm 42. At the at the dawn of Web two point o, there was MySpace. Myspace was a social network before social media took off. Myspace was fascinating because it had all of the sort of basic ingredients of what we later like saw with all the social networks that you all use today, but it never really took off.
00:25:46 · Michael Bernstein
It was like it didn't have all those ingredients in the right proportions. At some point, someone figured out the recipe and things went gangbusters. I think the last decade has been a story of massive improvements in the models, and I do think that the improvements in the models will continue. But I also think that if you stare at like the devices and the interactions that you and I use every day, they have really not transformed. We are in the MySpace era of AI. What are we using this to solve? How is it changing our like interactions with each other, with our... Like with information, with society? Like I think we have we have transformed the modeling stack, but we have not transformed the application stack. And my hot take is that I think the next decade, the bet to make is it's gonna change the app stack over the next decade. And I think that that is something that I encourage you to think about. How is it going to change? Why has it not changed yet? Unless you're coding in, you know, Claude code or something like this, Like, most of your experience with technology has not changed, but it's going to. So you better get ahead of it because that's where I think the ideas are gonna be. I also think of simulation as an example of this. It's like, again, a new superpower, a new affordance, a new thing you can do, and starting from that perspective. Okay. My last reflection until you can grill me with whatever you want to hear. Because we're a frontier lab with a product, it has been really interesting and challenging to get everything done. Right? Everything requires us to get modeling, working together with data, working together with product, working together with go to market, engineering, everything else. And it's like you're trying to climb a wall, and if any of those parts are kind of like lagging, you're not gonna you're not gonna climb the wall. In some ways, it's much easier and simpler to just like have a single... Like, there's just one thing that we gotta do. We just gotta, to, you know, hill climb just the modeling or just the product. But if you buy my thesis that you need to actually... For many of the the new transformations, you're gonna need to do all of these things.
00:28:16 · Michael Bernstein, Ravi Belani
Building out, you know, work streams, cross functional teams, like whatever it takes to keep these things in sync and climbing the hill together, that is I think the major interesting challenge in leadership management, whatever you want to call it, for startups these days. Let's say modeling has an idea. It's showing promise. Product doesn't know exactly when it... To expect that it'll work or the cases in which they can assume it will or will not work. How do they design the product around a model that's constantly changing? How does the data team need to work with the modeling to fill in those gaps, and how does product, you know, explain where these things are? Like, how does go to market explain to the customer? Like, they all have to stay in sync, which means you're constantly talking across technical to nontechnical to various different flavors of technical. All sorts of really, really smart people have to stay in all sorts of different kinds of sync. This has been really interesting and challenging, and I think in some ways the fun part of my job because I get to work with people who have so many different types of incredible intelligences, and also it is something that we are constantly working on because if we do not do this right, we will surely fail. And in fact, some of our earliest stumbles involve failures along these lines. So this is always an area for improvement. They told me to keep it brief, and so I've tried. I think we have about twenty minutes for all sorts of open ended conversation. [00:29:59] S1: Yes. That's that's awesome.
00:29:59 · Ravi Belani, Michael Bernstein
[00:29:59] S1: Yes. That's that's awesome. [00:30:01] S2: Everybody give give Michael a [00:30:02] S1: round of applause. So Michael... We're gonna we're gonna do a quick fireside chat with Michael, and then we'll tee up questions for the students. I'll kick it off, and then maybe we can get the student mic set up. Michael, that was awesome. You know, two weeks ago in this class we had Eric Ries, and Eric is the father of the Lean Startup Movement. Yes. And, you know, and he's baked in this practice in the Silicon Valley about closing the laptop, getting out of the building and talking to real human beings when you're doing customer discovery and problem discovery and customer discovery. So much of that, that's become like a mantra in how you build startups. I'm curious for your take. Are we entering into an era where you can do discovery through AI, or do you still believe in the sacredness of talking to live humans? [00:30:55] S2: I think if I were to advocate for, or anyone were to advocate for, like, completely cutting out talking to actual humans, like, I would lose my HCI card. Right? Like, this is a tool, and any tool has strengths and weaknesses. I think the strengths of this is that it gives you sort of instantaneous access to perspectives that might be hard to reach, the ability to iterate, but ultimately, you are trying to build a deep understanding of what the problem really is. And I think at some point, that has to involve talking to to real people.