00:27:49 · Joon Sung Park
[00:27:49] S1: So for us, the no gos are actually participants who cannot give informed consent. So we take informed consent very seriously, that every member of panel who joined our study and our data collection process, they know why the data is getting collected, at least broadly, and they consent to that process and they also have the opportunity to withdraw their consent. So that's something that we take seriously. So for instance, underage population cannot participate. So that's actually one blind spot that we own, that we do not collect data from underage population, and this would also generalize to other populations that cannot give informed consent. Now, that said, aside from that population, where similarly is different is we are trying to create a model that can truly be a representative snapshot of who we are as a society. So there's actually no population that we should avoid. We actually want to know every single population that's out there. And this is also where, for instance, we're also quite different from simply creating a model like language model or other modeling companies. Other modeling companies, what they are trying to create is amazing reasoning machine that's great at coding, mathematics, and fundamentally trying to be good chatbot, good assistant. For us, we are trying to create models that can actually model the diverse viewpoints, the values, preferences, and tastes of people. A part of it is also then learning the good behaviors but also bad behaviors of people. Simile needs to be able to capture those.
00:29:29 · Molly O'Shea, Joon Sung Park
[00:29:29] S0: For part of the prep work for this, we had a simile task, what do you call it, project, pilot, for sorcery, and I thought this was really interesting because I'm like, who are these people? Who are you interviewing? But there were two different tests, and so one was on the topic appeal, and one was on titles, and so the number one title that it guessed, or I guess, I don't know, what are the terms that you use that it predicted? [00:30:01] S1: Simulated. [00:30:02] S0: Simulate, of course. The number one title that it simulated was the $2,000,000,000 company building AI simulations of real people. That's the strongest title overall with my audience, which is mostly investors and like tech executives and that kind of thing, which is interesting. Then the topics really went into digital twins. The number one one was digital twins, simulated customer testing, and privacy. There was no clear winner among them. I kind of agree with, because right now the biggest concern with all this AI stuff is cybersecurity, and privacy risks. I obviously ask a lot about, like, how simulated customer testing works. The digital twins thing is quite interesting. Don't know how does this work for other people? So, like, what is the process at which this outputs this information?
00:31:00 · Joon Sung Park
[00:31:00] S1: What our users have today is they have access to a online, let's say we basically serve a SaaS platform to our users. So they get access to this web interface where they can basically come in and define two things. They can say who they want to talk to and what questions they want to ask. And really, they can ask anything. And right now, they can ask any number of questions. There is no limit to the number of questions they can ask and will generate responses. That's what they get today. Now, one of the things that is the most important though is this is a new way of making decisions. We haven't quite seen simulation that is so generalizable in social context that can inform all human decisions. So one of the things that we do and we work very closely with our users is the change management, helping users take these results and then helping them calibrate when to use it, how to use it, where do we need more data, how do we validate it. One of the core functions attached to these reports in relation to what I'm just saying is this idea of motor of confidence or predicted accuracy. So this actually motor is actually quite important to us. Similarly, we'll be generating predictions, simulations of future. Alongside the core foundation model that we are training to make the simulation though, we are jointly training a predictive model of accuracy where its job is one thing. Its job is to output zero to 100 range probability on how confident we are in this particular response on the basis of all the data that we use to train our model and the question that we got. And this is sort of like in inferential statistics. If you're a social scientist, it's a little bit like having p value. And in science, when p value is less than 0.05, that's sort of your canonical way of saying, we're ready to make a decision. We provide that. So if confidence is higher than 90% or usually 85%, our customers have found it to be robust evidence that can actually be used to make very high stake decisions. So it is actually the whole package. So it is about delivering the insight and then also delivering evidence that is undeniable that, well, if this is the evidence, then this is a totally reasonable outcome that we have to prepare for, and then providing this model that basically is giving us a degree of confidence calibration.
00:33:34 · Molly O'Shea, Joon Sung Park
[00:33:34] S0: It's really interesting. One of the questions I had was what is the most expensive mistake similarly helped a customer avoid? [00:33:43] S1: Yeah. [00:33:45] S0: What is it?
00:33:46 · Joon Sung Park
[00:33:46] S1: So, there's actually an interesting study that we actually ran. This technically was before simile, but I still mentioned it because I think as an example, I think this is quite interesting. We worked with a few automobile companies, and one of them, and in fact, of them were interested in electric vehicles. Should we do it? When should when should we do it? How should we do it? That was the top of their mind for every single electric vehicle company that we've signed... We've met. And these were some of the largest players in the field. And one of the simulations that we ran actually had to do with can we predict or can we simulate how people would react to different messaging for new... The entry of new electric vehicle in this particular market, and how can we optimize that messaging so we convey our values to our customers. So there was a strategy that said, okay, this is a clear winner. Fantastic. We're ready to go, and this is good. What was interesting, though, is this actually had downstream implication. This particular messaging, this particular approach towards the market actually shifted people's overall perception of nonelectric vehicle and globally actually lower the sales within that entire market. Geez. And obviously, was run-in studies, and this ended up becoming an input, but this was, again, this wasn't a part of a simile, back then. But it's these kind of findings that I actually find to be quite interesting. These are the findings that are not just narrowly focused on one particular question, but it's more generalizable, where you're looking at the entire field holistically. Simulation allows you to explore that, and when it's at its best, it actually should surface insights that's counterintuitive, that's surprising. So that's really the goal and that's really the value.
00:35:52 · Molly O'Shea
[00:35:52] S0: How good is AI at real professional work? Merkor's AI Productivity Index, or APEX, tracks it with every major Frontier model release. Bill with partners like Harvey and Cognition. See where the Frontier actually stands at merkor.com/apex. Today's episode is sponsored by VCX by Fundrise, the public ticker for private tech, allowing investors of all sizes to invest in venture capital. Learn more at getvcx.com. Public is a multi asset investing platform where you can automate your investing with AI agents. Create your own custom agent with a simple prompt. Agents can monitor markets, execute trading strategies, and manage your cash across stocks, bonds, options, crypto, and more. Want to buy at market close and sell the next days open? Just tell your agent. Automate your investing at public.com. Founders scale faster on deal. Set up payroll for any country in minutes. Hire anyone anywhere, get visas handled fast, and get back to building. Visit deal.com/sorcery. That's deel.com/sorcery. Do you have any dream scenarios or dream simulations that you want to answer or uncover? What's the wildest? You probably have some wild things in your head. What is the wildest kind of thing that you wish someone would want you to solve or you could solve on your own?