00:06:51 · Joon Sung Park
[00:06:51] S1: So our ability to test actually has shifted quite a bit also alongside the technological progress. On day one, when we created Smallville, what we actually did was we interviewed them. We interviewed the agents. What we've done is we pulled individual agents out of this game town, set them in this hypothetical chair on a command line, and we would basically ask them questions. And these questions would probe certain qualities of these agents that's similar to how people would reason about different tasks in their lives. So simple question, tell me your name. And then we would ask, tell us your life story. We would then ask questions like, who's the one person you want to spend time with? And then we would then ask, if you had to get that person a present, what would you get them? We would ask these questions because these are questions that when you think about human brain and human mind, we can only answer these questions because we can remember, because we can plan ahead, but also because we can reflect. Then what we ended up doing was we generated all these responses, and then we actually had human evaluators come in and take a look at these responses and said, given the experience that the agents had in their game town experience, are these experiences or insights actually human like? So we would actually basically have human evaluators compare between human authors' generation and these agents' responses to see which one's more realistic. And we've actually done quite well that these agents were actually quite realistic. And the way to sometimes tell is, imagine you're asking this agent, who do you want to hang out with the most? It's a little bit strange if the agent says whoever they happen to see the most on the street. The agent should likely say the person they actually care about. Who do you have special relationship with? What kind of conversations did you have with people? What actually stood out? And that agent should really reflect on that and pick those people. So those are some of the experiments that we ran, and those are the signals that we were able to get. But then over time, we wanted to go beyond that. So here, really the initial evaluation really was about believability that we start to see what we call the illusion of life in these agents.
00:09:08 · Joon Sung Park
But as we mature as a field, what we started to do was to run evaluation that is now about more of a faithfulness and accuracy. So the experiment that we ended up running following that was we ran an experiment where we then took this architecture to create simulations of thousands of real people. So that started with us actually going out, recruiting a large number of real people that sample representatively from The US. So it's a thousand representative sample of The US population across gender, race, ethnicity, and so forth. And then we actually collected data from these people. So we would actually do this time human interview. So we actually built an AI agent that can talk to voice to voice with people, have those agents talk to people voice to voice. We would ask questions that's a little bit like the questions we ask agents in Smallville. Tell me the story of your life. Where did you grow up? What were some of the hard decisions you had to make? Who matters to you? We would use that data to then train AI agents that can basically be the twins of these people. And then we would send people away for two weeks, bring them back to a lab. And then this time we would actually ask people to complete a battery of surveys, behavioral studies, respond to stimuli, and had their agents predict what they would say. And at the end of 2024, for the first time really in the field, we could actually show that we can predict people's behavior with 85% plus accuracy with these agents. That's ultimately what gave the company foundation for today's technology. Of course, the technology itself has shifted quite a bit, but that was the initial building block that we could see that, ah, models and these agents can now actually predict human behavior, and this is an amazing building blocks. But at the same time, it's large data operation that we do not simply create simulations out of vacuum. We believe strongly that the simulation has to be grounded, and it has to be evaluated alongside people. So it really is both.
00:11:09 · Molly O'Shea, Joon Sung Park
[00:11:09] S0: I have so many questions. Do you sometimes inject like anomalies or things that could create some factors that are not predictable? And then my second question, which I wanted to be my first question, is human behavior predictable? [00:11:28] S1: In the training data, we actually do inject a lot of behavioral knowledge. It's not just about what they say, but it's also really about what they do. And there's a lot of irrational behaviors in human tendencies. So a lot of things that we study in economic behavior game, randomized control trials, they all try to cover those. At the same time, there are certain things in people that's really not known to change. Your risk tolerance, as I briefly mentioned, it's not really known to change in a person. So once we are able to learn that, then this actually becomes a trait that becomes very predictable. So it really is about understanding the gap between what people think they are and what they actually do in their life. And really the modeling challenge here actually is really learning that translational layer. How do you take observed behaviors or attitudes into their fundamental tendencies? [00:12:27] S0: And what is in that gap? Is it time? Is it events around you? Is it like, how do you determine that gap that ultimately creates the outcome?
00:12:27 · Molly O'Shea, Joon Sung Park
[00:12:27] S0: And what is in that gap? Is it time? Is it events around you? Is it like, how do you determine that gap that ultimately creates the outcome? [00:12:37] S1: So the part that is actually very consistent in our experimentation is these two functions are actually governed by different function or a different sort of shape of the curve. What people say in survey actually often just follows a normal distribution. So when I say, you know, am I likely to go to the gym this week? I'll likely say with some normal distribution curve, you know, yes, maybe, you know, a seven point scale, I'll likely say five to six every week. Now, do I actually go to the gym that week? The activation energy for me to actually go to the gym is actually way higher than what I would expect. So that looks much more like an exponential curve where it has a much clearer drop off in terms of what kind of activation we need to actually get to that state of, oh, I'm actually going to exercise this week. So those are the kinds of different functions that we need to learn, and that's where often the gap is. [00:13:40] S0: Again, I have so many questions about this. What is the number one or the most common indicator of human behavior? Like what are those traits or things that you've noticed create the reactions or events? What causes people to do things? Why would someone make a decision versus not? Like, what are those? How do you determine if someone will continue a pattern or not?
00:14:07 · Joon Sung Park, Molly O'Shea
[00:14:07] S1: So it turns out, what this actually has to do sometimes actually is the existence of optionality. So let's actually briefly take a look at how psychologists would usually study human behavior in a survey. We have an abstract question, and we have basically a certain set of options that were presented in multiple choice questions or on a likely scale of one to seven. That's a very confined space, but when people behave, they don't usually behave in vacuum or in abstract conditions. They often behave in the real world, right? And what that basically means is even simple questions like, where do we want to go eat? In abstract, if somebody said, hey, let's go eat X, the odds are if I'm really hungry, in fact, I'm very hungry and if there's no food available, I'd almost certainly say, yes, let's go do that. But what often ends up happening is we do not have access to certain options, but we have access to other options. This is where people have this optionality. So now they have to think. They have to think about the opportunity cost of taking that option. So that's actually what really governs people's behavior. That yes, can go exercise this week, but if I don't, I can sleep for one more hour or I can take one more meeting. It's these kind of things. Optionality and the type of optionality often really governs people's behaviors. [00:15:37] S0: Do you pull a lot of data from dating apps? [00:15:41] S1: Not dating apps in particular, no. [00:15:43] S0: Okay, so taking all of this into consideration, how does this then become the business of SIMILE?
00:15:53 · Joon Sung Park
[00:15:53] S1: So SIMILE is a company that is creating foundational order of human behavior that basically can take as input a description of a person or a group of people and then simulate what these people might do in different contexts. So today, simple use cases would actually go into things like surveys, experiments. Many of the insights team have a lot of studies they were meant to run. Now what Simulink can do is actually run them, but in a much faster pace, in a month with much less resources. So that's obvious use cases. So we do concept testing, and in more complex cases, we do conjoint analysis, max dip analysis. The traditional analysis that researchers had to run today that would have taken them sometimes hundreds of thousands of dollars, sometimes millions of dollars or many years, we can run them in simulation basically almost instantly. They are just a query away. So that's the use cases today. But at the same time, the users then come to us and ask, Okay, it's great that we can now do these studies. Can we then do something that our studies couldn't do? What will people actually do? How will they actually react to different stimuli? So this is where simulation really is about also creating model of the environment, not just the people. So imagine that we have customers. Just a dumb example here. Imagine you're trying to sell water. It's actually quite interesting. If this as a behavior question is actually quite interesting, imagine there's only one source of water, then that particular person can likely charge us a million dollar to just have access to water. But the reason why that cannot happen is because of the optionality, because there is a context around the water source where it is everywhere, right? It's about packaging, it's about, of course, the right stream, at the end of the day, this is widely available, that there are other optionalities in this space. So we to model that. We want to model that environment. What are the other options? What are the other alternatives? Given that as the context, can we then expose our agents to this decision making? At what price would you actually buy this water, given all these environmental factors that goes into this? So it really is about modeling people and their behavior, but it's also about modeling the environment and everything that surrounds them.